Cloudflare

One regex took down 80% of the internet's traffic for 27 minutes

A WAF rule with catastrophic backtracking pinned CPU to 100% across the entire edge network. It passed tests, because the tests measured correctness, not cost.

Duration: 27 minutes
Blast radius: Global — every Cloudflare domain served 502
Organisation: Cloudflare

The part that stings

The deploy pipeline had four staged environments — DOG, PIG, Canary, then global. WAF rules skipped all of them by design, because security rules need to ship in seconds. The safety mechanism existed and this change was explicitly exempt from it.

Timeline

13:31
Engineer merges a PR adding the rule.
13:37
CI builds and tests it. Tests pass — they check matching, not cost.
13:42
Automatic global deploy begins, in seconds, via the Quicksilver KV store.
13:45
First page: WAF synthetic test failing. CPU near 100% network-wide.
14:00
WAF identified as the cause from live CPU data and strace.
14:02
Decision to kill the WAF globally — accept losing protection to restore traffic.
14:07
Global WAF kill switch executed.
14:09
Traffic and CPU normal. 27 minutes total.
14:52
WAF re-enabled after verification.

The expression

The rule ended with .*(?:.*=.*) — three greedy quantifiers where two would do. The middle group can match the same text the outer one already consumed, so for any failing input the engine must try every possible split before giving up.

Why it explodes

Steps grow exponentially with input length, not linearly. Matching x=x took 23 steps; x= followed by twenty x's took 555. Extend that to a real request body and a single HTTP request occupies a core indefinitely. PCRE backtracks with no runtime ceiling, and CPU guards that would have caught it had been removed in an earlier refactor.

Why tests didn't catch it

The suite asserted that rules matched what they should and didn't match what they shouldn't. It had no notion of how long a match was permitted to take, so an expression that was correct but unbounded passed cleanly.

Commands that mattered

Reproduce the blowup before shipping
```
time perl -e '$s="x="."x"x30; $s =~ /.*(?:.*=.*)/ and print "match\n"'
```
Add one x at a time. Linear growth is fine; doubling per character is a production incident waiting for traffic.
What they used to find it live
```
strace -c -p $(pgrep -n nginx) -f 2>&1 | head -20
```
Confirmed the CPU was burning in userspace regex evaluation rather than syscalls or I/O.
Bound it in CI
```
rg -n '\.\*.*\.\*' rules/ | head
```
Two .* in one expression is the smell. Not always fatal, always worth a second reader.
The structural fix
```
# swap PCRE for RE2 — linear time, no backtracking, by construction
```
RE2 and the Rust regex crate refuse to compile patterns they can't run in linear time. The guarantee moves from discipline to the type system.

What it teaches

01A test suite that checks correctness but not cost will pass the change that takes you down.
02Every staged-rollout exemption is a place where an outage becomes global instantly. Cloudflare kept emergency global deploy but made staged the default afterwards.
03Kill switches earn their keep in minutes. Turning the WAF off globally was the right call and took five minutes to decide — because the switch already existed.
04Protections removed during a refactor don't announce themselves. The CPU guard that would have contained this had been taken out earlier for unrelated reasons.
05Prefer engines with runtime guarantees over reviewer discipline. RE2 makes the failure mode unrepresentable.

This rhymes with something I hit

Pipeline step aborts with exit 2 and no error message ↗

Both are a one-line change whose failure mode lives in the tool's semantics rather than the code under review. Nobody reads a regex or a grep exit code as a hazard until it costs them.

Next postmortem

Roblox — 73 hours ↗

Sources

Cloudflare — Details of the outage on July 2, 2019 (primary) ↗

This is an analysis of a published incident at Cloudflare. It is not my own work, and every figure above is taken from the source linked here.