ALL NOTES

observability · SEV-3

Dashboards green, services listed as onboarded, no data arriving

scroll to render

How to confirm it

  • Injected, or just annotated?

    kubectl get pods -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) containers=\(.spec.containers|length)"' | grep 'containers=1'

    An annotation without an extra container means the operator never acted. The manifest looks right and nothing was injected.

  • Is data actually arriving?

    # entity exists != entity reporting — check last-seen, not presence
    curl -s -H "Api-Key: $NR_KEY" 'https://api.newrelic.com/graphql' -d '{"query":"{actor{entitySearch(query:\"reporting=false\"){results{entities{name}}}}}"}'

    Stale entities keep appearing in dashboards long after they stop reporting. Filter on reporting=false to find them.

  • What did the CRD migration drop?

    kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.template.metadata.annotations != null and .spec.template.metadata.labels["app-language"] == null) | .metadata.name'

    Annotation present, label missing — the exact signature of the regression.

Read the source

Cause

A CRD migration dropped the labels while leaving the annotation in place. Entities stayed in the backend as stale objects, so nothing looked broken.

Fix

Audited every deployment end to end — labels, injection, and actual data arrival — rather than trusting the onboarding doc. True coverage was 40%.