Coverage you believe is not coverage you have
Every org I have worked in believed it was instrumented. The dashboards existed, the agent was in the base image, the onboarding doc had a checklist. An actual audit — deployment by deployment, checking labels, injection and whether data was arriving — put real coverage at 40%. The other 60% looked fine because a service with no telemetry produces no alerts, and no alerts reads as healthy.
kubectl get deploy -A -o json | jq -r '.items[] |
[.metadata.namespace, .metadata.name,
(.spec.template.metadata.labels."app-name" // "MISSING"),
(.spec.template.metadata.annotations."instrumentation.opentelemetry.io/inject-sdk" // "NONE")]
| @tsv' | column -tLabels and annotations are necessary but not sufficient. Cross-reference against what the backend has actually received in the last hour — that gap is the real number.
The four states nobody models
Onboarding is usually treated as a boolean. It isn't. A deployment can be: correctly instrumented and reporting; injected but silent; labelled but never injected; or untouched entirely. Each needs a different fix, and lumping them together as 'not done' is why the remediation stalls. Silent-but-injected is the nastiest, because it looks correct from the cluster side.
Compiled languages break the pattern
Agent injection via init container works for interpreted runtimes. For Go and other compiled languages there is no agent to inject at runtime — instrumentation is compile-time. A rollout that assumes one mechanism for every service produces a set of pods stuck pulling an image that was never published. The lesson generalises: verify the mechanism exists for each runtime before rolling it out to hundreds of deployments.
kubectl get pods -A --field-selector=status.phase!=Running \
-o custom-columns=NS:.metadata.namespace,POD:.metadata.name,REASON:.status.containerStatuses[*].state.waiting.reason \
| grep -i 'ImagePullBackOff\|ErrImagePull'If this returns init containers across several services at once, the cause is a rollout assumption rather than a registry problem.
Alerting is where trust is won or lost
Inherited alert policies accumulate. Nobody deletes an alert, because deleting one feels like accepting risk. So the pager fires for things nobody acts on, people learn to dismiss it, and the one that mattered gets dismissed with the rest. Cutting 30% of the noise made the remainder credible — which is the actual goal. An alert nobody trusts is worse than no alert, because it costs attention and buys nothing.
Metrics say slow. Profiles say why.
Dashboards are excellent at telling you a system is degraded and nearly useless at telling you which line is responsible. Roblox spent two days on hardware theories before a flame graph pointed at contention inside a code path. Keep continuous profiling available before you need it — at 3am is a bad time to discover you cannot capture one.
perf record -F 99 -a -g -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svgThirty seconds of samples usually settles an argument that metrics alone will run for hours.
What goes wrong
Cardinality is the quiet killer — one label carrying a user ID or a request path multiplies your series count and your bill until retention gets cut to compensate, at which point you have less history exactly where you needed more. Circular dependencies are the other: telemetry that runs on the cluster it observes disappears with it. And a CRD migration silently dropping labels leaves stale entities in the backend that still render green.
curl -s http://prometheus:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'Run this monthly. The metric at the top of that list is the one about to force a retention decision.