shared infra · SEV-1
Trace ingestion stopped for every tenant at once
How to confirm it
How deep is the backlog?
redis-cli -h HOST LLEN bull:ingestion-queue:wait && redis-cli -h HOST LLEN bull:ingestion-queue:activeA growing wait list with a static active count means workers have stopped consuming, not that they are slow.
What is the store actually doing?
redis-cli -h HOST --statWatch blocked_clients and ops/sec together. Blocked clients climbing while ops flatline is contention, not load.
Find the noisy producer
redis-cli -h HOST CLIENT LIST | awk '{print $2, $NF}' | sort | uniq -c | sort -rn | headIdentifies which client is issuing the most commands — usually one tenant is responsible for the burst.
Drain the poisoned queue
redis-cli -h HOST --scan --pattern 'bull:ingestion-queue*' | xargs -n 100 redis-cli -h HOST DELDestructive: this discards queued work. Only after you have accepted the data loss and captured what was in flight.
Read the source
Cause
Jobs wedged in a shared queue backed by a Redis-compatible store running with atomicity relaxed. One noisy producer stalled every consumer.
Fix
Drained the poisoned queues, restarted the workers, restored atomicity, and wrote the runbook. Shared infrastructure needs per-tenant blast-radius limits before it needs more throughput.
Part of this work
Observability Across 50+ Microservices ↗
Distributed tracing and real-time metrics across 50+ services — and 30% less alert noise.
The same shape, at scale
Roblox — 73 hours ↗
A Consul feature flag enabled a week earlier met a latent BoltDB free-page bug. Two unrelated faults, one of which only appeared under the other's load.
Next incident
401 from a private package registry, but only inside CI ↗