Publish post-mortem #2951
Open
Publish post-mortem #2951
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thib <[email protected]>
Deploying matrix-website with
|
| Latest commit: |
54d3b35
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e1c9ad4e.matrix-website.pages.dev |
| Branch Preview URL: | https://publish-post-mortem.matrix-website.pages.dev |
Signed-off-by: Thib <[email protected]>
HarHarLinks
reviewed
Oct 10, 2025
|
|
||
|  | ||
|
|
||
| Confusingly, at the time of the incident, the primary database server is called `db-02`, and the secondary database server is called `db-01`. The deployment runs on bare metal servers at Mythic Beasts and the Postgres database servers both use their own logical RAID 10 array with `mdraid`. |
There was a problem hiding this comment.
presumably
Suggested change
| Confusingly, at the time of the incident, the primary database server is called `db-02`, and the secondary database server is called `db-01`. The deployment runs on bare metal servers at Mythic Beasts and the Postgres database servers both use their own logical RAID 10 array with `mdraid`. | |
| Confusingly, at the time of the incident, the primary database server is called `db-02`, and the secondary database server is called `db-01`. The deployment runs on bare metal servers at Mythic Beasts and the Postgres database servers both use their own logical RAID 10 array with [`mdraid`](https://docs.kernel.org/admin-guide/md.html). |
There was a problem hiding this comment.
Unsure, I'd rather not link to something I'm not certain about.
There was a problem hiding this comment.
Makes sense, perhaps if we had the responsible team here @neilisfragile we could at least amend it. The name mdraid is unfamiliar to me as a private enthusiast previous mdadm user so clarity would be beneficial to overall readability
There was a problem hiding this comment.
Yes, that link is correct
There was a problem hiding this comment.
(unresolving given the change was not yet applied)
|
|
||
| ### Timeline | ||
|
|
||
| At 11:03 UTC on Sept 2nd 2025, Mythic Beasts’ teams added 2 NVMe drives to both `db-02` and `db-01`, respectively the primary and secondary database servers. |
There was a problem hiding this comment.
reading top to bottom: is this supposed to imply the start of a mdadm --grow operation?
|
|
||
| But we also needed the team to get some rest. Given how slow it was to replay WALs, we reconfigured our backups to happen against the primary database rather than against the (missing) replica. We let the European team go to bed, while our American SRE kept tabs on everything. At 03:26 UTC a new incremental backup completed. | ||
|
|
||
| At 09:21 UTC we added the two NVMe disks to the RAID array and to the LVM volumes group of `db-01`. We rebooted to ensure the disks were properly detected and mounted \- but the server didn’t come back. We opened the lights-out console Mythic Beasts provides us, and saw that the RAID array was not in the functional state. We had rebooted `db-01` at a critical moment of the array reshaping. |
There was a problem hiding this comment.
why was it reshaping at this point? wouldn't it have been ~empty?
Co-authored-by: Kim Brose <[email protected]>
Signed-off-by: HarHarLinks <[email protected]>
Signed-off-by: HarHarLinks <[email protected]>
HarHarLinks
approved these changes
Oct 10, 2025
There was a problem hiding this comment.
I am not hard blocking a merge at this point, but I strongly recommend to clarify the open points. At least to me they raise questions when they should be answering them, and I think I am a rather big enthusiast who is able to interpret some stuff compared to the average already technical the blog audience.
HarHarLinks
reviewed
Oct 10, 2025
| @@ -0,0 +1,182 @@ | |||
| +++ | |||
| date = "2025-10-10T10:00:00Z" | |||
HarHarLinks
requested changes
Oct 21, 2025
|
|
||
| The SRE team would like to thank our hosting provider Mythic Beasts. They reached out quickly and proactively when adding new disks, reporting the errors they were seeing. They have been much more than just a pair of remote hands. They also reached out with an offer of support during the incident. | ||
|
|
||
| Finally, we’d like to sincerely apologise again to everyone impacted by the outage. We hope you found the post-mortem informative and you’d like to talk about it more, several of us will be at the [Matrix Conference 2025](https://conference.matrix.org) in Strasbourg. In addition to a flurry of great talks, there will be workshops about how to set up a Matrix homeserver and tune the clients to your liking\! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✔️ Checklist
<>to linkify them (learn more).[label](@/target.md)syntax./blogpage, especially for multiple posts on the same day. Prefer UTC format, e.g.2025-12-01T14:00:00Zfor Dec 1st, 2025, 2pm UTC.🧢 Website & WG cap