A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.

On this page

Real-World RAG Incidents: Lessons from a Production Rollout

When we first rolled out a RAG-based assistant for our internal SRE team, nothing in the vendor docs really prepared us for the messy parts.

Incident: Cached Wrong Answers #

The first painful incident happened on a Monday morning. A runbook query returned an outdated PostgreSQL failover procedure because:

We cached answers aggressively to save tokens.
The underlying runbook in Git had been updated over the weekend.
Our invalidation logic only watched the vector store, not the source repo.

How We Fixed It #

We changed our cache key to include the document commit hash.
We added a background job that compares Git commits against stored vectors.
We updated the runbook template to show the last updated date in the answer.

Incident: Embeddings Going Silent #

Two weeks later, we saw a spike in “no relevant context found” errors during incident calls. The vector DB was healthy; the problem turned out to be:

A new data source with HTML-heavy content.
We were chunking purely by character count.
The relevant text was split across three different chunks.

Changes We Made #

Switched to semantic + heading based chunking with overlap.
Added a metric for “chunks per query” and “distance of top-1 match”.
Logged a sample of low-quality retrievals for manual review.

Checklist for RAG in Production #

Track cache hit rate, LLM error rate, and no-context rate.
Store the retrieved chunk IDs alongside each answer.
Regularly sample answers and review them with the owning team.

The marketing pages sold RAG as magic. In reality it behaves more like a database: if you don’t design for drift, invalidation, and observability, it will betray you at the worst moment.

Real-World RAG Incidents: Lessons from a Production Rollout

Real-World RAG Incidents: Lessons from a Production Rollout

Incident: Cached Wrong Answers #

How We Fixed It #

Incident: Embeddings Going Silent #

Changes We Made #

Checklist for RAG in Production #

Stay Updated

How We Stopped Terraform Drift from Surprising On-Call

GitHub Actions Monorepo CI: How We Cut Build Times Without Breaking Main

More from AI

Model Fallback Policies for Customer-Facing AI: The Routing Rules That Kept SLA Intact

Embedding Model Upgrades Without Search Chaos: A Safer RAG Rollout Pattern

Prompt Versioning and Regression Testing: How Teams Avoid Silent AI Regressions

Model Fallback Policies for Customer-Facing AI: The Routing Rules That Kept SLA Intact

Embedding Model Upgrades Without Search Chaos: A Safer RAG Rollout Pattern

Prompt Versioning and Regression Testing: How Teams Avoid Silent AI Regressions

RAG Retrieval Quality Evaluation: The Checks We Added After Bad Answers Reached Production

Real-World RAG Incidents: Lessons from a Production Rollout

AI Best Practices in 2026: Shipping Reliable Systems, Not Demo Magic

About Kiril urbonas

Real-World RAG Incidents: Lessons from a Production Rollout

Incident: Cached Wrong Answers#

How We Fixed It#

Incident: Embeddings Going Silent#

Changes We Made#

Checklist for RAG in Production#

Stay Updated

How We Stopped Terraform Drift from Surprising On-Call

GitHub Actions Monorepo CI: How We Cut Build Times Without Breaking Main

More from AI

Model Fallback Policies for Customer-Facing AI: The Routing Rules That Kept SLA Intact

Embedding Model Upgrades Without Search Chaos: A Safer RAG Rollout Pattern

Prompt Versioning and Regression Testing: How Teams Avoid Silent AI Regressions

About Kiril urbonas

Incident: Cached Wrong Answers #

How We Fixed It #

Incident: Embeddings Going Silent #

Changes We Made #

Checklist for RAG in Production #