ALL NOTES

shared infra · SEV-1

Trace ingestion stopped for every tenant at once

scroll to render

How to confirm it

  • How deep is the backlog?

    redis-cli -h HOST LLEN bull:ingestion-queue:wait && redis-cli -h HOST LLEN bull:ingestion-queue:active

    A growing wait list with a static active count means workers have stopped consuming, not that they are slow.

  • What is the store actually doing?

    redis-cli -h HOST --stat

    Watch blocked_clients and ops/sec together. Blocked clients climbing while ops flatline is contention, not load.

  • Find the noisy producer

    redis-cli -h HOST CLIENT LIST | awk '{print $2, $NF}' | sort | uniq -c | sort -rn | head

    Identifies which client is issuing the most commands — usually one tenant is responsible for the burst.

  • Drain the poisoned queue

    redis-cli -h HOST --scan --pattern 'bull:ingestion-queue*' | xargs -n 100 redis-cli -h HOST DEL

    Destructive: this discards queued work. Only after you have accepted the data loss and captured what was in flight.

Read the source

Cause

Jobs wedged in a shared queue backed by a Redis-compatible store running with atomicity relaxed. One noisy producer stalled every consumer.

Fix

Drained the poisoned queues, restarted the workers, restored atomicity, and wrote the runbook. Shared infrastructure needs per-tenant blast-radius limits before it needs more throughput.