Architecture
Self-Healing Loop
A recurring incident with a known fix gets wired to the signal instead of to a human. Roughly 80% resolve before anyone is paged.
What you're looking at
Workloads emit telemetry; monitoring evaluates it; an alert publishes an event to Pub/Sub rather than paging a person. A Cloud Function subscribes and runs the remediation that a human would have run — the runbook, expressed as code. Most incidents close there. The dashed red edge is the escalation path for anything the function cannot handle, and the loop back to workloads is the restored state.
Why the event bus is in the middle
Wiring monitoring straight to a remediation function couples them, and the coupling becomes the failure. Pub/Sub between the two means remediations can be added, replaced or disabled without touching alerting, several subscribers can react to one event, and a failed remediation can be retried or dead-lettered rather than lost.
What goes wrong here
Automated remediation acting on a bad signal is worse than no automation — a flapping health check plus an auto-restart is an outage generator. Every action needs to be idempotent, rate-limited, and give up after N attempts rather than looping. The most common real failure is a remediation that masks a degrading dependency for weeks until it fails in a way restarting cannot fix.
Inspect it yourself
Is anything stuck in the dead-letter queue?
gcloud pubsub subscriptions pull DLQ_SUB --limit=10 --auto-ack --format='value(message.data)' | base64 -dThe dead-letter queue is the list of incidents automation could not fix. Read it weekly; it is the backlog of what to automate next.
Are remediations looping?
gcloud functions logs read self-heal --limit=200 --format='value(textPayload)' | grep -oE 'resource=[^ ]+' | sort | uniq -c | sort -rn | headThe same resource remediated repeatedly is a masked problem, not a solved one.
Read the source
Components
- WORKLOADS— GKE
- MONITORING— metrics + logs
- PUB/SUB— event bus
- CLOUD FUNCTION— runbook-as-code
- REMEDIATED— most incidents
- HUMAN— the remainder
Flows
- wl→montelemetry
- mon→busalert
- bus→fntrigger
- fn→fixauto-fix
- fn⇢humanescalate
- fix→wlrestored