incident-management: tighten IR template structure and pipeline runbook

Author: frameworks-volunteer

docs/pages/incident-management/incident-response-template/incident-response-policy.mdx

@@ -171,7 +171,7 @@ See [Runbooks](./runbooks/overview) for step-by-step guides for specific inciden
 **Goal:** Confirm the fix actually worked.
 
 - Verify immediately after deployment
-- Monitor for at least a week
+- Monitor based on residual risk, blast radius, and incident type
 - Consider adding new alerts or test cases
 - Document what monitoring is now in place
 

docs/pages/incident-management/incident-response-template/overview.mdx

@@ -85,6 +85,17 @@ That's it to start. Add complexity only as you need it.
 
 Review and adapt these pages for your own internal incident response documentation.
 
+This section is different from the broader [Incident Management](/incident-management/overview) guidance:
+
+- **[Incident Management](/incident-management/overview)** explain concepts and practices
+- **[Incident Response Templates](/incident-management/incident-response-template/overview)** are meant to be copied, customized, and used internally
+
+Within this template section:
+
+- **[Policy](/incident-management/incident-response-template/incident-response-policy) / [roles and staffing](/incident-management/incident-response-template/roles-and-staffing) / [communications](/incident-management/incident-response-template/communications) / [contacts](/incident-management/incident-response-template/contacts)** define your operating model
+- **[Templates](/incident-management/incident-response-template/templates/overview)** are blank working documents to fill out during or after incidents
+- **[Runbooks](/incident-management/incident-response-template/runbooks/overview)** are scenario-specific response procedures
+
 ### What's Included
 
 | Document | Purpose |

docs/pages/incident-management/incident-response-template/roles-and-staffing.mdx

@@ -180,8 +180,7 @@ Regardless of team size, define who can make high-stakes decisions during P1 inc
 | | | |
 | | | |
 
-These people should be reachable 24/7 for critical incidents. Consider:
-
+There should be a 24/7 escalation path to these people for critical incidents. Consider:
 - Founders / C-level
 - Security Lead
 - Engineering Lead

docs/pages/incident-management/incident-response-template/runbooks/build-pipeline-compromise.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Runbook: Build Pipeline Compromise | Security Alliance"
-description: "Stub runbook. Customize with your CI/CD platform and procedures."
+description: "Example runbook for CI/CD compromise. Review and customize for your platform, release process, and trust boundaries before use."
 tags:
   - Security Specialist
   - Operations & Strategy
@@ -21,7 +21,7 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr
 <TagList tags={frontmatter.tags} />
 <AttributionList contributors={frontmatter.contributors} />
 
-> **Stub runbook.** Customize with your CI/CD platform and procedures.
+> **This is an example runbook.** Review and customize for your CI/CD platform, artifact flow, deployment model, and approval process before use.
 
 ## Quick Reference
 
@@ -36,39 +36,149 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr
 
 ### Symptoms
 
-- [ ] Unexpected code in deployed artifacts
-- [ ] CI/CD configuration changed without approval
-- [ ] Secrets accessed or exfiltrated
-- [ ] Unauthorized workflow runs
+- [ ] Unexpected workflow runs or releases
+- [ ] CI/CD configuration changed without expected approval
+- [ ] Secrets accessed, exported, or rotated unexpectedly
+- [ ] Build artifacts differ from expected source or prior reproducible output
+- [ ] Deployments reference an unexpected commit, artifact, or builder identity
 
-### Confirm Compromise
+### Likely Scope Questions
+
+- Is this limited to CI configuration, or were artifacts actually produced from a compromised pipeline?
+- Did the pipeline have deploy permissions, signing authority, or production credentials?
+- Were any releases, containers, frontend bundles, or packages published during the exposure window?
+
+### Differentiation
+
+- Unauthorized code merged without pipeline abuse may be a repository compromise first
+- Malicious package updates without CI tampering may be a dependency incident first
+- A bad deployment from a legitimate commit may be an operational failure rather than compromise
 
-- Review CI/CD audit logs
-- Compare build artifacts to source
-- Check for config changes in CI/CD platform
 
 ## Immediate Actions
 
-1. [ ] Disable compromised pipelines
-2. [ ] Rotate all secrets and tokens
-3. [ ] Take down potentially compromised deployments
-4. [ ] Audit recent builds and deployments
+### Step 1: Freeze the pipeline
+
+**Why:** Stop additional malicious builds, releases, or secret access.
+
+- [ ] Disable affected workflows/pipelines
+- [ ] Revoke or pause auto-deploy jobs
+- [ ] Block manual approvals until scope is understood
+
+### Step 2: Preserve evidence
+
+**Why:** CI audit logs, workflow definitions, artifact metadata, and deployment history are easy to overwrite.
+
+- [ ] Export CI audit logs
+- [ ] Save workflow/job history for the exposure window
+- [ ] Record affected commits, workflow files, artifact digests, release IDs, and deployment targets
+- [ ] Preserve runner details if self-hosted runners were involved
+
+### Step 3: Rotate credentials by blast radius
+
+**Why:** Pipeline compromise often becomes credential compromise.
+
+Prioritize rotation of:
+- [ ] CI platform tokens
+- [ ] cloud deploy credentials
+- [ ] package registry tokens
+- [ ] artifact signing keys or release credentials
+- [ ] secrets available to self-hosted runners
+
+### Step 4: Stop trust in recent outputs
+
+**Why:** Do not assume recent artifacts or deployments are clean.
+
+- [ ] Identify all artifacts built during the exposure window
+- [ ] Identify all deployments and releases from those artifacts
+- [ ] Quarantine or withdraw suspicious outputs where possible
+
+
+## Investigation
+
+### Key Questions
+
+- [ ] What was the initial access path: CI platform, repository permissions, runner compromise, or stolen token?
+- [ ] What permissions did the compromised pipeline actually have?
+- [ ] Were secrets exposed only to logs/runtime, or used to publish or deploy?
+- [ ] Which environments were reachable: build only, staging, production?
+- [ ] Which outputs must now be treated as untrusted?
+
+### Information to Gather
+
+| Data | Source |
+|------|--------|
+| CI audit logs | CI/CD platform |
+| workflow/config diffs | repository history |
+| release/deployment history | CI/CD platform, cloud provider, registry |
+| artifact digests / provenance | registry, signing system, artifact store |
+| runner access and execution logs | runner host / CI platform |
+
+
+## Containment and Recovery
+
+### Option A: Rebuild from a known-good commit using a clean pipeline
+
+**When:** You can identify a trusted commit and re-establish a trusted build path.
+**Impact:** Release cadence slows, but trust is restored more safely.
+
+1. Stand up a clean pipeline or isolated builder
+2. Re-verify repository state and workflow definitions
+3. Rebuild from a known-good commit
+4. Compare output metadata against expected source and release intent
+5. Redeploy only from the rebuilt trusted output
+
+### Option B: Roll back to last known-good release
+
+**When:** A trusted prior release exists and rollback is operationally safe.
+**Impact:** Feature loss or temporary service degradation may occur.
+
+1. Identify the last trusted artifact and deployment
+2. Roll back affected services
+3. Verify rollback success in production
+4. Continue investigation before resuming normal release flow
+
+### Option C: Keep service paused until trust is re-established
+
+**When:** You cannot distinguish clean from compromised outputs.
+**Impact:** Operational disruption, but lower risk of serving malicious artifacts.
+
+1. Pause releases/deployments
+2. Communicate impact internally and externally as needed
+3. Rebuild trust in source, pipeline, credentials, and artifacts before resuming
+
+
+## Verification Before Resuming
+
+Do not resume normal delivery until you can answer these clearly:
+
+- [ ] The initial access path is understood well enough to prevent immediate recurrence
+- [ ] Compromised credentials have been rotated or invalidated
+- [ ] Untrusted artifacts and releases have been identified and handled
+- [ ] Build and deploy permissions are re-scoped appropriately
+- [ ] A known-good artifact has been rebuilt or a known-good release has been restored
+
+
+## Hardening After the Incident
+
+- [ ] Separate build permissions from deploy permissions
+- [ ] Require stronger approval controls for workflow and release changes
+- [ ] Use short-lived credentials where possible
+- [ ] Reduce secret exposure to only the jobs that need them
+- [ ] Restrict or harden self-hosted runners if used
+- [ ] Improve artifact provenance, signing, and release verification
+
 
-## Mitigation
+## Escalation
 
-1. [ ] Audit CI/CD configuration for unauthorized changes
-2. [ ] Rebuild from trusted commit using clean pipeline
-3. [ ] Implement additional approval requirements
-4. [ ] Review and restrict pipeline permissions
+Escalate immediately if:
+- [ ] production deployments may have been modified
+- [ ] signing keys or release credentials may be exposed
+- [ ] user-facing artifacts may have been maliciously published
+- [ ] the pipeline had access to broader cloud or infrastructure credentials
 
-## Prevention
+See [Contacts](../contacts) and [Incident Response Policy](../incident-response-policy).
 
-- [ ] Require approval for CI/CD config changes
-- [ ] Use short-lived credentials
-- [ ] Implement branch protection
-- [ ] Audit pipeline access regularly
-- [ ] Use signed commits
-- [ ] Separate build and deploy permissions
 
 ## Related
 

docs/pages/incident-management/incident-response-template/runbooks/dependency-attack.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Runbook: Dependency Attack | Security Alliance"
-description: "Stub runbook. Customize with your package management and build procedures."
+description: "Example runbook for dependency compromise. Review and customize for your package manager, build flow, and release process before use."
 tags:
   - Security Specialist
   - Operations & Strategy
@@ -21,7 +21,7 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr
 <TagList tags={frontmatter.tags} />
 <AttributionList contributors={frontmatter.contributors} />
 
-> **Stub runbook.** Customize with your package management and build procedures.
+> **This is an example runbook.** Review and customize for your package manager, dependency policy, and build procedures before use.
 
 ## Quick Reference
 
@@ -38,41 +38,128 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr
 
 - [ ] Unexpected behavior after dependency update
 - [ ] Security advisory for a package you use
-- [ ] Malicious code found in node_modules or similar
-- [ ] Lockfile changes you didn't make
+- [ ] Malicious code found in installed dependencies or build output
+- [ ] Lockfile changes you did not expect
+- [ ] Frontend bundle or released artifact changed more than the source diff would explain
 
-### Confirm Dependency Attack
+### Scope Questions
 
-```
-npm audit
-# or
-yarn audit
-```
+- Did the malicious dependency reach production, or is this still limited to source/build environments?
+- Was the dependency executed only during build time, or also in user-facing runtime code?
+- Did the dependency have access to CI secrets, registry tokens, or signing material?
+- Were any artifacts published or deployed during the exposure window?
 
-Check for recent lockfile changes in git history.
+### Differentiation
+
+- If the compromised package was only introduced through your CI/build path, also review [Build Pipeline Compromise](./build-pipeline-compromise)
+- If the issue is malicious code being served to users, also review [Frontend Compromise](./frontend-compromise)
+- If this is only an advisory on an unused code path, severity may differ from an actively weaponized package
 
 ## Immediate Actions
 
-1. [ ] Take down site to stop serving malicious code
-2. [ ] Identify the malicious package
-3. [ ] Pin dependencies to last known good version
-4. [ ] Rebuild from clean environment
+### Step 1: Freeze releases that rely on the affected dependency
+
+**Why:** Prevent additional malicious artifacts from being built or published.
+
+- [ ] Pause affected builds and deployments
+- [ ] Stop package publishing if your pipeline republishes downstream artifacts
+- [ ] Block dependency auto-update jobs until scope is understood
+
+### Step 2: Identify the exact bad package/version path
+
+**Why:** You need a precise package, version, and introduction path before you can cleanly contain it.
+
+- [ ] Record package name and version
+- [ ] Identify whether it is direct or transitive
+- [ ] Identify which commits or lockfile changes introduced it
+- [ ] Determine whether multiple repos or services consume it
+
+### Step 3: Stop trusting recent outputs
+
+**Why:** If malicious code ran during build or packaging, recent artifacts may be untrusted even if source looks clean.
+
+- [ ] Identify builds created during the exposure window
+- [ ] Identify deployed versions and published packages that include the dependency
+- [ ] Quarantine or withdraw suspicious outputs where possible
+
+### Step 4: Preserve evidence
+
+**Why:** Registry state, lockfiles, and CI evidence can change quickly.
+
+- [ ] Save lockfile and manifest versions from affected builds
+- [ ] Save CI logs and artifact metadata
+- [ ] Record package tarball hashes, integrity values, and registry metadata where available
+
+## Investigation
+
+### Key Questions
+
+- [ ] Was the package actually malicious, or just vulnerable?
+- [ ] Was the malicious code executed in your environment?
+- [ ] Did it affect build-time systems, runtime users, or both?
+- [ ] Were secrets, signing credentials, or deploy tokens exposed?
+- [ ] Which artifacts or releases must now be treated as untrusted?
+
+### Information to Gather
+
+| Data | Source |
+|------|--------|
+| package/version metadata | registry, lockfile, manifest |
+| introduction point | git history, dependency bot PRs |
+| build logs | CI/CD platform |
+| deployed artifacts using bad dependency | release history, artifact store |
+| advisories / upstream incident details | package registry, maintainer advisories |
+
+
+## Containment and Recovery
+
+### Option A: Revert to the last known-good dependency state
+
+**When:** A trusted prior lockfile/package set exists.
+**Impact:** Fastest path in many cases.
+
+1. Revert the dependency or lockfile to a known-good state
+2. Rebuild from a clean environment
+3. Verify the bad package is no longer present in the output
+4. Redeploy only after validating the resulting artifact
+
+### Option B: Replace or remove the affected package
+
+**When:** A clean replacement exists, or the package is non-essential.
+**Impact:** May require code changes or degraded functionality.
+
+1. Remove or replace the affected package
+2. Regenerate lockfile carefully
+3. Rebuild from a clean environment
+4. Validate both functionality and resulting dependency tree
+
+### Option C: Keep service paused until trust is re-established
+
+**When:** You cannot clearly determine which artifacts are clean.
+**Impact:** Operational disruption, but lower chance of serving malicious code.
+
+1. Pause affected deployments
+2. Reconstruct a trusted dependency set
+3. Rebuild from a clean environment
+4. Resume only after verifying outputs and rotation of any exposed credentials
+
 
-## Mitigation
+## Verification Before Resuming
 
-1. [ ] Remove or replace malicious package
-2. [ ] Update lockfile with known good versions
-3. [ ] Rebuild using `npm ci` or `yarn --frozen-lockfile`
-4. [ ] Redeploy verified build
+- [ ] The malicious or suspect dependency is removed from the build
+- [ ] A clean rebuild has been produced from a trusted environment
+- [ ] Any exposed credentials have been rotated
+- [ ] Suspicious builds, releases, or packages have been identified and handled
+- [ ] The resulting artifact has been checked against expected changes
 
 ## Prevention
 
 - [ ] Use lockfiles and commit them
-- [ ] Use `npm ci` / `yarn --frozen-lockfile` in CI
-- [ ] Regularly audit dependencies
-- [ ] Consider using a private registry
-- [ ] Pin exact versions for critical packages
-- [ ] Review dependency changes in PRs
+- [ ] Use deterministic install commands in CI (`npm ci`, `pnpm install`, `yarn install --immutable`, etc.)
+- [ ] Review dependency and lockfile changes in PRs
+- [ ] Restrict who can approve dependency update automation
+- [ ] Audit critical dependencies and minimize unnecessary package surface area
+- [ ] Treat build-time dependencies as part of the production attack surface
 
 ## Related