Skip to content

Commit aeeee92

Browse files
JunirezzJunirezz
authored
Merge pull request #876 from esthertitilayo-dev/fix/769-general-add-incident-postmortem-template-and-publication-workflow
[#769] General: Add incident postmortem template and publication workflow
2 parents f5a67b4 + 216437f commit aeeee92

15 files changed

Lines changed: 602 additions & 3 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
2020
- Vault performance dynamic date filter
2121

2222
### Documentation
23+
- Add incident postmortem templates, publication playbook, and CI validation workflow (#769)
2324
- Add release notes playbook and changelog curation guidelines (#618)
2425
- Add API versioning and deprecation policy with sunset windows, migration guide, and breaking-change classification (#610)
2526

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,16 @@ YieldVault has comprehensive disaster recovery procedures to ensure system resil
163163
- [Disaster Recovery Runbooks Overview](./docs/runbooks/README.md)
164164
- [Replay and State Recovery Procedures](./docs/runbooks/REPLAY_PROCEDURES.md)
165165

166+
## Incident Postmortems
167+
168+
YieldVault documents significant incidents with blameless postmortems and tracked action items:
169+
170+
- **Templates:** [Post-mortem](./docs/runbooks/templates/post-mortem.md), [Incident Report](./docs/runbooks/templates/incident-report.md)
171+
- **Publication workflow:** [Postmortem Playbook](./docs/postmortem-playbook.md)
172+
- **Published reports:** [docs/incidents/](./docs/incidents/README.md)
173+
174+
Postmortem drafts are due within 48 hours of incident resolution; publication within 5 business days.
175+
166176
## Roadmap (Phases)
167177

168178
- **Phase 1**: Planning, Documentation, and Frontend UI Baseline (Completed)

‎docs/ci/postmortem-docs.workflow.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Postmortem Docs CI Workflow
2+
3+
Install this file at `.github/workflows/postmortem-docs.yml` to enable PR validation
4+
for published postmortem reports.
5+
6+
```yaml
7+
name: Validate Postmortem Docs
8+
9+
on:
10+
pull_request:
11+
paths:
12+
- 'docs/incidents/**'
13+
- 'docs/runbooks/templates/**'
14+
- 'docs/postmortem-playbook.md'
15+
- 'scripts/validate-postmortem.sh'
16+
17+
jobs:
18+
validate:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@v4
22+
- name: Validate postmortem structure
23+
run: chmod +x scripts/validate-postmortem.sh && ./scripts/validate-postmortem.sh
24+
```
25+
26+
See [ISSUE_769_IMPLEMENTATION_SUMMARY.md](../runbooks/ISSUE_769_IMPLEMENTATION_SUMMARY.md).

‎docs/incident_response_runbook.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,9 @@ This runbook documents the operational procedures for handling **RPC degradation
7777

7878
---
7979
## 7. Post‑mortem & Continuous Improvement
80-
- Complete the **Post‑mortem Template** (`docs/POSTMORTEM_TEMPLATE.md`).
80+
- Complete the **Post‑mortem Template** ([`docs/runbooks/templates/post-mortem.md`](./runbooks/templates/post-mortem.md)).
81+
- Follow the **Publication Workflow** in [`docs/postmortem-playbook.md`](./postmortem-playbook.md).
82+
- Publish finalized reports to [`docs/incidents/`](./incidents/README.md).
8183
- Update runbook if new failure modes were discovered.
8284
- Review alert thresholds and adjust if false‑positives occurred.
8385
- Schedule a **runbook drill** quarterly.

‎docs/incidents/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Published Incident Postmortems
2+
3+
This directory contains finalized, published postmortem reports for YieldVault incidents and significant DR exercises.
4+
5+
## Index
6+
7+
| Date | Incident ID | Title | Severity | Postmortem |
8+
|------|-------------|-------|----------|------------|
9+
| — | — | *No published postmortems yet* | — | — |
10+
11+
## Creating a New Postmortem
12+
13+
1. Copy [`docs/runbooks/templates/post-mortem.md`](../runbooks/templates/post-mortem.md)
14+
2. Draft in `docs/incidents/drafts/` during review (optional)
15+
3. Follow the publication workflow in [`docs/postmortem-playbook.md`](../postmortem-playbook.md)
16+
4. Publish via PR using filename: `YYYY-MM-DD-INCIDENT-XXX-short-slug.md`
17+
5. Update this index table
18+
19+
## Related Resources
20+
21+
- [Postmortem Playbook](../postmortem-playbook.md)
22+
- [Incident Response Runbooks](../runbooks/README.md)
23+
- [Incident Report Template](../runbooks/templates/incident-report.md)
24+
25+
**Last Updated:** June 26, 2026
26+
**Maintained By:** DevOps Team

‎docs/incidents/drafts/.gitkeep‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎docs/postmortem-playbook.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Incident Postmortem Playbook
2+
3+
This document describes when YieldVault writes postmortems, how action items are
4+
tracked, and the publication workflow for finalized reports.
5+
6+
---
7+
8+
## 1. When to write a postmortem
9+
10+
Write a postmortem for any of the following:
11+
12+
| Trigger | Examples |
13+
|---------|----------|
14+
| **Severity 1–2 incidents** | Full outage, data loss risk, contract pause |
15+
| **DR events** | Database restore, RPC failover, backend redeploy under pressure |
16+
| **Security incidents** | Key compromise, unauthorized access, exploit attempt |
17+
| **Contract upgrades with issues** | Failed upgrade, rollback, unexpected state |
18+
19+
Lower-severity incidents may use a shortened report at the Incident Commander's
20+
discretion, but must still capture root cause and action items.
21+
22+
---
23+
24+
## 2. Timeline
25+
26+
| Phase | Deadline | Deliverable |
27+
|-------|----------|-------------|
28+
| During incident | Real-time | [Incident Report Template](./runbooks/templates/incident-report.md) |
29+
| Post-incident | Within 48 hours | Postmortem draft |
30+
| Publication | Within 5 business days | Published report in `docs/incidents/` |
31+
32+
These deadlines align with the [Quick Reference](./runbooks/QUICK_REFERENCE.md)
33+
post-mortem checklist and [Incident Response Runbooks](./runbooks/README.md).
34+
35+
---
36+
37+
## 3. Roles
38+
39+
| Role | Responsibility |
40+
|------|----------------|
41+
| **Incident Commander** | Owns timeline accuracy and severity classification |
42+
| **Author** | Drafts postmortem from incident report and logs |
43+
| **Reviewer** | DevOps or Security lead validates technical accuracy |
44+
| **Release engineer** | Ensures security-sensitive details follow disclosure rules |
45+
46+
---
47+
48+
## 4. Creation flow
49+
50+
1. **Start from template** — Copy
51+
[`docs/runbooks/templates/post-mortem.md`](./runbooks/templates/post-mortem.md).
52+
2. **Optional draft location** — Save work-in-progress to
53+
`docs/incidents/drafts/INCIDENT-XXX-slug.md` (not indexed until published).
54+
3. **Gather inputs**:
55+
- Live [incident report](./runbooks/templates/incident-report.md)
56+
- Grafana / PagerDuty timelines
57+
- Backend diagnostics bundle (`/api/diagnostics/bundle`)
58+
- Relevant runbook steps exercised
59+
4. **Complete all sections** — Summary, impact metrics, timeline, root cause,
60+
action items table, lessons learned.
61+
62+
---
63+
64+
## 5. Action-item tracking
65+
66+
Every postmortem must include an **Action Items** table with:
67+
68+
| Column | Required |
69+
|--------|----------|
70+
| ID | Yes (`AI-001`, `AI-002`, …) |
71+
| Action | Yes |
72+
| Owner | Yes |
73+
| Priority | Yes (P0/P1/P2) |
74+
| Due Date | Yes |
75+
| Tracking Issue | Yes — link to GitHub issue |
76+
| Status | Yes (Open / In Progress / Done) |
77+
78+
**Workflow:**
79+
80+
1. File each action item as a GitHub issue referencing the incident ID.
81+
2. Link the issue number in the postmortem table.
82+
3. Review open action items in the quarterly runbook review
83+
([runbooks README](./runbooks/README.md) §Continuous Improvement).
84+
85+
---
86+
87+
## 6. Review and redaction
88+
89+
Before publication:
90+
91+
- [ ] Incident Commander and Reviewer sign off on timeline and severity
92+
- [ ] Remove credentials, PII, and unreleased vulnerability details
93+
- [ ] For **security incidents**, follow the 48-hour minimum disclosure window
94+
described in [Release Notes Playbook](./release-notes-playbook.md) §8
95+
- [ ] Confirm customer-facing language is approved if published externally
96+
97+
---
98+
99+
## 7. Publication flow
100+
101+
1. **Open a PR** adding the finalized report to `docs/incidents/` using the
102+
naming convention: `YYYY-MM-DD-INCIDENT-XXX-short-slug.md`
103+
2. **Set `Status: Published`** in the report header (drafts must not remain in
104+
`docs/incidents/` root)
105+
3. **Update the index** in [`docs/incidents/README.md`](./incidents/README.md)
106+
4. **Link action items** — Ensure every `AI-xxx` row has a merged or open GitHub
107+
issue
108+
5. **Update runbooks** if new failure modes were discovered
109+
6. **Announce** in `#yieldvault-incidents`; update status page if user-facing
110+
7. **Merge PR** after reviewer approval
111+
112+
CI validates postmortem structure via `scripts/validate-postmortem.sh`. Install the
113+
workflow from [`docs/ci/postmortem-docs.workflow.yml`](./ci/postmortem-docs.workflow.yml)
114+
into `.github/workflows/` to enable automated PR checks.
115+
116+
---
117+
118+
## 8. DR test reports
119+
120+
Disaster recovery exercises that surface runbook gaps should file a
121+
[DR Test Report](./runbooks/templates/dr-test-report.md). Significant findings
122+
warrant a full postmortem using the same publication flow.
123+
124+
---
125+
126+
## 9. Runbook feedback loop
127+
128+
After each published postmortem:
129+
130+
1. Identify runbook sections that were unclear or missing
131+
2. Open a follow-up PR updating the relevant runbook under `docs/runbooks/`
132+
3. Record the change in the postmortem's **Runbook Updates Required** section
133+
134+
---
135+
136+
**Last Updated:** June 26, 2026
137+
**Maintained By:** DevOps Team
138+
**Issue:** [#769](https://github.com/Junirezz/YieldVault-RWA/issues/769)

‎docs/runbooks/ISSUE_769_IMPLEMENTATION_SUMMARY.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Issue #769 Implementation Summary: Incident Postmortem Template and Publication Workflow
2+
3+
**Issue:** General: Add incident postmortem template and publication workflow
4+
**Status:** ✅ COMPLETED
5+
**Date:** June 26, 2026
6+
7+
---
8+
9+
## Goal
10+
11+
Create a standard postmortem template with action-item tracking and a publication
12+
workflow so the team can consistently document and learn from incidents.
13+
14+
---
15+
16+
## Scope Delivered
17+
18+
### 1. Postmortem and Incident Templates ✅
19+
20+
**Directory:** [docs/runbooks/templates/](./templates/)
21+
22+
| File | Purpose |
23+
|------|---------|
24+
| [post-mortem.md](./templates/post-mortem.md) | Blameless postmortem with action-item table and publication checklist |
25+
| [incident-report.md](./templates/incident-report.md) | Live incident log during active response |
26+
| [dr-test-report.md](./templates/dr-test-report.md) | DR exercise report with RTO/RPO tracking |
27+
28+
Fixes previously broken links in [runbooks README](./README.md) Appendix C.
29+
30+
### 2. Publication Workflow Playbook ✅
31+
32+
**File:** [docs/postmortem-playbook.md](../postmortem-playbook.md)
33+
34+
- When to write postmortems (severity, DR, security, contract events)
35+
- 48-hour draft / 5-day publication timeline
36+
- Roles, review/redaction, and security disclosure alignment
37+
- PR-based publication flow into `docs/incidents/`
38+
- Action-item → GitHub issue tracking requirements
39+
40+
### 3. Published Postmortem Archive ✅
41+
42+
**File:** [docs/incidents/README.md](../incidents/README.md)
43+
44+
- Index table for published reports
45+
- Naming convention: `YYYY-MM-DD-INCIDENT-XXX-slug.md`
46+
- Optional drafts under `docs/incidents/drafts/`
47+
48+
### 4. Automation ✅
49+
50+
| File | Purpose |
51+
|------|---------|
52+
| [scripts/new-postmortem.sh](../../scripts/new-postmortem.sh) | Scaffold draft from template |
53+
| [scripts/validate-postmortem.sh](../../scripts/validate-postmortem.sh) | CI validation for published reports |
54+
| [docs/ci/postmortem-docs.workflow.yml](../ci/postmortem-docs.workflow.yml) | Workflow definition for maintainers to install under `.github/workflows/` |
55+
56+
### 5. Cross-Link Updates ✅
57+
58+
- [docs/incident_response_runbook.md](../incident_response_runbook.md) — fixed broken template link
59+
- [docs/runbooks/README.md](./README.md) — quick links to playbook and incidents index
60+
- [docs/runbooks/QUICK_REFERENCE.md](./QUICK_REFERENCE.md) — postmortem step links
61+
- [README.md](../../README.md) — incident postmortems section
62+
- [CHANGELOG.md](../../CHANGELOG.md) — unreleased documentation entry
63+
64+
---
65+
66+
## Acceptance Checklist
67+
68+
- [x] Standard postmortem template with action-item tracking
69+
- [x] Incident report template for live incidents
70+
- [x] DR test report template (unblocks broken README link)
71+
- [x] Publication workflow playbook
72+
- [x] Published postmortem archive index
73+
- [x] Scaffold and validation scripts
74+
- [x] CI workflow for postmortem doc validation
75+
- [x] Broken documentation links fixed
76+
77+
---
78+
79+
## Related Files
80+
81+
- Issue: [#769](https://github.com/Junirezz/YieldVault-RWA/issues/769)
82+
- Pattern reference: [ISSUE_392_IMPLEMENTATION_SUMMARY.md](./ISSUE_392_IMPLEMENTATION_SUMMARY.md)
83+
- Release disclosure pattern: [release-notes-playbook.md](../release-notes-playbook.md) §8
84+
85+
**Last Updated:** June 26, 2026
86+
**Maintained By:** DevOps Team

‎docs/runbooks/QUICK_REFERENCE.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -146,8 +146,8 @@ All runbooks: `docs/runbooks/`
146146
3. **Notify** - Alert team via PagerDuty/Slack
147147
4. **Respond** - Follow appropriate runbook
148148
5. **Verify** - Confirm system restored
149-
6. **Document** - Create incident report
150-
7. **Review** - Post-mortem within 48 hours
149+
6. **Document** - Create [incident report](./templates/incident-report.md)
150+
7. **Review** - [Post-mortem](./templates/post-mortem.md) within 48 hours per [playbook](../postmortem-playbook.md)
151151

152152
---
153153

‎docs/runbooks/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ This directory contains operational runbooks for disaster recovery and incident
1515
| [RPC Failover](./RPC_FAILOVER.md) | 5 min | N/A | Stellar RPC node failure |
1616
| [Full DR Procedure](./FULL_DR_PROCEDURE.md) | 4 hours | 15 min | Complete infrastructure failure |
1717
| [Replay & State Recovery](./REPLAY_PROCEDURES.md) | N/A | N/A | Recovering/syncing ledger events or email queue |
18+
| [Postmortem Playbook](../postmortem-playbook.md) | N/A | N/A | Publishing incident postmortems |
19+
| [Published Postmortems](../incidents/README.md) | N/A | N/A | Archive of finalized incident reports |
1820

1921
---
2022

0 commit comments

Comments
 (0)