Most pages are the same page
Look at a quarter of incidents and the distribution is brutal: a small set of causes, repeating, each with a known fix already written in a runbook. A human is woken to read the runbook and type the commands. That is not judgement work — it is a person being used as a scheduler with worse latency and a worse mood.
Wire the fix to the signal
If the remediation is deterministic, the alert should trigger it rather than a human. Monitoring publishes to Pub/Sub, a Cloud Function subscribes, and the runbook runs as code. The human is notified of what happened, not asked to perform it. Roughly 80% of recurring incidents resolve before anyone opens a laptop.
gcloud pubsub subscriptions pull ops-events --auto-ack --limit=5 \
--format='table(message.attributes.alert,message.publishTime)'Start by watching the event stream for a week without acting on it. The repeats are obvious and they are your automation backlog, in priority order.
Idempotency is not optional
Pub/Sub guarantees at-least-once delivery, so your handler will be invoked twice for the same event eventually. A remediation that restarts a deployment is safe to repeat; one that scales up by a fixed increment is not, and will happily scale you to the quota ceiling at 3am. Every handler needs a deduplication key and a guard.
# dedupe on the event id before acting
if cache.add(f"seen:{event_id}", ttl=3600) is False:
return "duplicate, skipping"The failure this prevents is subtle: nothing errors, the action just happens N times. Without a guard you find out from the bill or the quota.
Always leave a way to stop it
Automation that fights a human is worse than no automation. If someone is mid-incident and the self-healer keeps reverting their change, you have built an adversary. Every handler checks a maintenance flag first, and every action it takes is announced in the incident channel with what it did and how to disable it.
kubectl annotate deploy/$SVC ops.selfheal/paused=true --overwriteOne annotation, honoured by every handler, checked before acting. The off switch has to be faster to reach than the thing it stops.
What goes wrong
Flapping is the main hazard — an alert that clears and re-fires drives a restart loop that looks like the automation causing the outage, so handlers need cool-down windows and a circuit breaker after N attempts. Masking is the subtler one: a self-healer that quietly restarts a leaking service every four hours removes the symptom and the incentive to fix the leak. Track remediation counts as a metric, and treat a rising trend as a bug rather than a success.