diff --git a/docs/runbook-emergency-pause.md b/docs/runbook-emergency-pause.md new file mode 100644 index 0000000..76e64f5 --- /dev/null +++ b/docs/runbook-emergency-pause.md @@ -0,0 +1,244 @@ +# Emergency Contract Pause And Recovery Runbook + +This runbook gives operators a cold-start procedure for pausing the escrow contract, communicating with users, identifying affected matches, recovering funds, and deciding whether to unpause, roll back, or upgrade after an incident. + +Use it for testnet drills and production incidents. Replace placeholder contract IDs, account names, and entrypoint names with the values from the current deployment before running commands. + +## Incident Roles + +- Incident commander: owns the decision to pause, unpause, upgrade, or recover funds. +- Contract operator: runs Stellar CLI commands from the approved admin account. +- Backend operator: disables unsafe API paths, captures logs, and exports affected match IDs. +- Communications owner: posts user-facing updates and support instructions. +- Reviewer: confirms each high-risk command before it is submitted. + +## Prerequisites + +- Stellar CLI installed and authenticated with an approved operator key. +- Current network, RPC URL, passphrase, contract ID, and admin public key. +- Access to backend logs, database read replicas, and monitoring dashboards. +- A tested backup of the current contract WASM and deployment metadata. +- A private incident channel for command review and audit notes. + +Recommended environment: + + export STELLAR_NETWORK=testnet + export STELLAR_RPC_URL=https://soroban-testnet.stellar.org + export STELLAR_PASSPHRASE="Test SDF Network ; September 2015" + export CHECKMATE_CONTRACT_ID= + export CHECKMATE_ADMIN=admin + +For mainnet, use the mainnet RPC endpoint and passphrase from the release runbook. Never paste mainnet secret keys into shared chat or issue comments. + +## Decision Tree + +Pause the contract immediately when any of these are true: + +- Funds can move to an unintended account. +- Match settlement can be manipulated. +- A privileged key is suspected to be compromised. +- The backend is submitting malformed or duplicated settlement requests. +- An exploit is active or reproducible. + +Prefer a contract upgrade instead of a pause when all of these are true: + +- Funds are not at immediate risk. +- The bug is fully understood. +- The fixed WASM is already reviewed and ready. +- Users can keep using unaffected flows safely. + +Prefer monitoring only when the issue is cosmetic, off-chain only, or already blocked by existing validation. + +## 1. Start The Incident + +1. Open an incident record with the time, reporter, affected environment, and suspected impact. +2. Assign the roles above. +3. Freeze non-essential deployments. +4. Save current contract metadata: + + stellar contract inspect --id "$CHECKMATE_CONTRACT_ID" --network "$STELLAR_NETWORK" + +5. Capture current ledger height, backend version, frontend version, and indexer cursor. + +## 2. Pause The Contract + +Have a second operator review the command before submission. + + stellar contract invoke \ + --id "$CHECKMATE_CONTRACT_ID" \ + --source "$CHECKMATE_ADMIN" \ + --network "$STELLAR_NETWORK" \ + -- \ + pause + +If the deployed contract uses a different emergency entrypoint, use the project-specific name, for example set_paused --paused true. + +Confirm paused state: + + stellar contract invoke \ + --id "$CHECKMATE_CONTRACT_ID" \ + --source "$CHECKMATE_ADMIN" \ + --network "$STELLAR_NETWORK" \ + -- \ + is_paused + +Expected result: paused state is true, and settlement or match-mutating calls fail with the documented paused error. + +## 3. Notify Users + +Within 15 minutes of a production pause, publish a short status update. + +Initial template: + + We have temporarily paused Checkmate Escrow while we investigate an issue affecting contract operations. Existing funds remain under review and we are not asking users to take action right now. We will post the next update by