Postmortems
Other people's worst days
Close readings of published outage reports from companies operating at a scale most of us only read about. Not my incidents — theirs, sourced and attributed. The reasoning is the point.
- Roblox
73 hours down, and the monitoring depended on the thing that broke
A Consul feature flag enabled a week earlier met a latent BoltDB free-page bug. Two unrelated faults, one of which only appeared under the other's load.
- Duration
- 73 hours
- Blast radius
- 50M users · 18,000 servers · 170,000 containers
- Cloudflare
One regex took down 80% of the internet's traffic for 27 minutes
A WAF rule with catastrophic backtracking pinned CPU to 100% across the entire edge network. It passed tests, because the tests measured correctness, not cost.
- Duration
- 27 minutes
- Blast radius
- Global — every Cloudflare domain served 502
Every figure on these pages comes from the primary source linked at the bottom of each write-up. Where I've drawn a parallel to my own work it is labelled as such.