A RAG chatbot that makes things up is worse than no chatbot. It answers with confidence and it is wrong. Your users trust the wrong answer because the bot sounds sure. So before we hand a RAG system to a client, we measure how often it lies. Here is the method, with the numbers we actually look at.
What hallucination means in RAG
Hallucination is when the model states something the retrieved context does not support. There are two ways this happens. The retriever pulls the wrong chunks, or the right chunks are there and the model ignores them. These are different failures. You fix them in different places. So we measure them separately.
The first failure is a retrieval problem. The second is a generation problem. Lumping them together hides which half is broken.
Build a labeled eval set first
You cannot measure what you have not labeled. So we start by writing a question set tied to the corpus. For a support agent on a product manual, that is real customer questions. We write 80 to 150 of them. Each one gets a known-correct answer and the source passage that proves it.
We include hard cases on purpose. Questions the docs do not answer. Questions with answers that changed between versions. Questions that look similar but mean different things. A clean eval set with only easy questions tells you nothing.
This set is the contract. We test against it before handover, and you keep it to test against later.
Measure retrieval on its own
Retrieval gets scored before the model writes a word. For each question, we check whether the correct passage shows up in the top-k chunks. We report recall at k and mean reciprocal rank. Plain version: did we fetch the right source, and how high did it rank.
If recall at k is low, the model never had a chance. No prompt fix saves a retriever that does not surface the answer. So we tune chunk size, overlap, and embeddings until retrieval clears our bar. Only then do we look at generation.
Measure faithfulness in generation
Faithfulness asks one question. Is every claim in the answer supported by the retrieved context? We grade this two ways.
A model-graded check runs every answer through a separate judge model. The judge sees the answer and the source chunks, and flags any sentence the chunks do not support. We also hand-grade a random sample to keep the judge honest. The judge can be wrong too, so we audit it.
We track the share of answers that are fully grounded. We track unsupported-claim rate per answer. A bot that is right 90 percent of the time but invents a fact in the other 10 is not ready.
Test refusal, not just answers
A good RAG system knows when to say it does not know. So we feed it the unanswerable questions from the eval set. The correct behavior is a refusal or a handoff, not a guess. We measure how often it guesses anyway.
This number matters more than people expect. The support agent we built answers in 12 minutes, down from 4 hours. That only helps if the fast answers are right. A fast wrong answer at scale is a liability, not a win. Refusal rate on unanswerable questions is the guardrail.
What we hand over
You get the eval set, the scores, and the harness to re-run it. When you change the corpus or swap a model, you run the same suite and see if quality moved. The metrics do not live in our heads. They live in your repo.
That is the point of the handover. Fixed price every time, you own the code, we exit. The eval harness is part of the code you own. No retainer to keep the lights on.
Where this fits
This is how we deliver a RAG Chatbot, scoped fixed-price and shipped with its own test suite. The same discipline runs through everything we build. We are a two-person team, ex-AWS, with the certs and the production work to back the method. Ten production products in seven months, each one measured before it shipped.
If you have a RAG project and want it to ship with a number instead of a vibe, we should talk. Book a free 30-minute call and we will walk through what measuring your system would look like.