A RAG chatbot without an eval suite is a demo, not a product. It works in the meeting because you asked it the three questions you already knew it could answer. Then it ships. Then a real user asks question four. Now it confidently invents a refund policy that does not exist.
You did not build a liar on purpose. You built a system with no way to know when it lies. Those are the same thing in production.
The demo gap is a measurement gap
Demos test the happy path. Production tests everything else.
The chatbot that nailed your demo has never seen a typo, a multi-part question, or a query that spans two documents. It has never been asked something the docs simply do not cover. Each of those is a distinct failure mode. None of them show up until a stranger triggers them.
The fix is not a better model. The fix is knowing your numbers before users find them.
What actually breaks in RAG
Retrieval breaks first, and it breaks quietly. The model only sees the chunks you hand it. Hand it the wrong chunks and even a strong model answers wrong with total confidence.
Here are the failures worth naming.
Bad retrieval. The right answer is in your docs, but the retriever pulls the wrong chunks. The model never had a chance.
Chunking damage. You split a table or a procedure across two chunks. Now half the answer is gone and the model fills the gap.
Stale index. Your docs changed. Your vector store did not. The bot answers from last quarter's policy.
Hallucination under pressure. The retrieved context does not contain the answer. A good system says so. A demo guesses.
No citations. The bot gives an answer with no source. Now no one can tell if it is right, including you.
Every one of these is measurable. That is the whole point.
An eval suite is just tests for non-deterministic code
You already write tests for code that returns the same value twice. RAG returns different text every time. So you test the properties that must hold, not the exact string.
A working suite has four parts.
A golden set. Fifty to a few hundred real questions with known-good answers. Pull them from actual support tickets, not your imagination. This is the single highest-leverage thing you can build.
Retrieval metrics. For each question, did the right chunk make it into the context? Measure recall at k. If retrieval misses, nothing downstream matters.
Answer grading. Does the response match the known-good answer? Use an LLM-as-judge for fuzzy matching, but spot-check the judge against humans. A judge you never audit is just another untested component.
Refusal checks. Ask questions the docs cannot answer. The correct response is "I don't know" with no invented facts. Score how often the bot makes things up. This number predicts your first angry customer.
Run the suite on every change. New embedding model, new chunk size, new prompt, new docs. If the score drops, you caught it before a user did.
What this looks like in practice
We built a support agent that answers in 12 minutes, down from 4 hours. That number is not luck. It holds because we measure retrieval and refusals on a golden set, and we re-run it every time something changes.
The eval suite is the difference between "it worked when I tried it" and "it works." One is a story. The other is a number you can defend in front of a customer.
In 7 months our 2-person team shipped 10 production products. We move fast because we test, not despite it. The harness is what lets you change the model on Friday and sleep on Saturday.
The honest scope
A RAG chatbot is a real project with a fixed price. Ours runs in the $5K to $15K band, eval suite included, because shipping it without one is shipping a demo. You own the code. We build it, hand it over, and exit. See pricing for the bands.
If you already have a bot and no idea how often it lies, that is the first thing to fix. Build the golden set. Measure retrieval and refusals. The number will tell you whether you have a product or a liability.
We are ex-AWS, 12x AWS Certified, and we have shipped enough of these to know where they break. If you want a second set of eyes on a RAG system before it embarrasses you, book a free 30-minute call and we will walk through your failure modes.