The answer existed. Nobody could find it.
Post-incident reviews get written, filed, and never read again. The next engineer to hit the same failure searches the wiki with different words, finds nothing, and rediscovers the cause from first principles at 2am. The knowledge is not missing — it is unreachable, because search matches words and incidents are remembered as symptoms.
Symptoms are the query, causes are the content
People search with what they see: 'connection reset after deploy'. The document that answers it is titled 'PeerAuthentication STRICT rollout'. Keyword search cannot bridge that; embeddings can, because the symptom and the cause are semantically close even with no shared vocabulary. That gap is the entire reason this works better than the wiki's own search.
Retrieval quality is a data problem
Most RAG disappointment is bad chunking blamed on the model. RCAs have structure — symptom, timeline, root cause, remediation — and chunking on those boundaries with the document title carried into each chunk's metadata changes the results far more than swapping models does.
# measure before tuning
for q, expected_doc in golden_set:
hits = store.search(q, limit=5)
recall_at_5.append(expected_doc in [h.id for h in hits])
print(sum(recall_at_5) / len(recall_at_5))Thirty real questions with known-correct documents. Without this number you are tuning on vibes, and every change feels like an improvement.
Cite, or it is worse than the wiki
For operational knowledge a confident wrong answer is actively dangerous — someone will run the command. Every response carries the source document and its date, and the model is instructed to refuse rather than infer when retrieval comes back weak. Refusal is a feature here, not a limitation.
What goes wrong
Stale content is the big one: a runbook from three reorganisations ago answers confidently and sends someone to a service that no longer exists, so age has to be surfaced in the answer and heavily weighted in ranking. Access control is the other — an index built across all documentation will happily surface HR content to an engineer unless the filter is applied at retrieval, not after generation. And embedding drift: re-embedding with a new model without reindexing everything gives you a silently broken index that still returns results.
curl -s localhost:6333/collections/runbooks | jq '.result | {points_count, config: .config.params.vectors}'Pin and record the embedding model alongside the collection. A mismatch between query-time and index-time models degrades silently — no error, just quietly worse answers.