Most bad RAG answers are not a model problem. They are a retrieval problem. Your vector search pulled the wrong chunks, and the LLM did exactly what you asked with bad input.
The usual reaction is to reach for a bigger model. That is the expensive fix. The cheap fix is a reranker, and most pipelines skip it.
Why your top-k is lying to you
Vector search ranks chunks by embedding similarity. That is a rough signal. Two passages can sit close in embedding space and answer completely different questions.
So you ask for the top 5 chunks. The actual answer is sitting at rank 11. The model never sees it. It answers from whatever made the cut, confidently, and wrong.
This is the failure mode people misread as hallucination. The model is not making things up. It is summarizing the wrong source. No prompt tweak fixes that, because the right context never reached the context window.
What a reranker actually does
A reranker is a second pass. You over-retrieve, then you re-score.
Pull a wide net first. Ask the vector store for the top 30 or 50 chunks instead of 5. Recall matters more than precision here. You want the right answer somewhere in that pool.
Then a cross-encoder scores each chunk against the query directly. Unlike embeddings, it reads the query and the chunk together. It catches relevance that cosine similarity misses. You keep the top 5 by that score and pass those to the LLM.
The model now sees better context. Same model, same prompt, better answer.
The cost is small and predictable
This is why it is the cheap win. You are not retraining anything. You are not swapping your foundation model.
Reranking adds latency, usually tens to low hundreds of milliseconds, depending on pool size and whether you call a hosted reranker or run one yourself. For most chatbots and internal tools, that is invisible. The user waited on the LLM anyway.
Cost-wise you are scoring 30 to 50 short passages. That is cheap next to the generation call. Hosted reranking APIs price it per thousand documents and it rounds to nothing at typical volume.
A support agent we built answers in 12 minutes, down from 4 hours. Retrieval quality is a big part of why that number holds. If the bot pulls the wrong policy doc, it answers fast and useless, and you have just automated a wrong answer at scale.
How to know if you need it
Do not add a reranker on faith. Measure first.
Pull a set of real questions your system gets. For each, check whether the correct chunk is in your top 5. Then check whether it is in your top 30. If the answer is "missing from 5, present in 30," that gap is exactly what reranking closes.
If the right chunk is not even in your top 30, reranking will not save you. That is a chunking or embedding problem, and you fix it upstream. Reranking reorders what retrieval found. It cannot find what retrieval missed.
This order matters. Teams skip the measurement, bolt on a reranker, and get a smaller win than expected. The reranker was fine. The retrieval pool was already broken.
Where it fits in the build
We put a reranker in by default on retrieval-heavy work because the math is obvious. A small latency cost buys a real jump in answer quality. Skipping it to "keep things simple" usually means shipping a bot that is confidently wrong a quarter of the time.
It also lowers cost in a way people miss. Better retrieval means you can often use a smaller, cheaper generation model and still hit your quality bar. The reranker does the heavy lifting on relevance, so the LLM has an easier job.
Our RAG Chatbot work bakes this in. Fixed price, scoped up front, and you own the code when we exit. No retainer, no lock-in to a pipeline only we understand.
The short version
Reranking is over-retrieve, re-score, then keep the best. It fixes the most common RAG failure, which is the right answer never reaching the model. The cost is a little latency and a cheap scoring call.
If your RAG bot is confidently wrong and you are eyeing a bigger model, check your retrieval first. The fix is usually one component, not a rebuild.
Want a second set of eyes on your pipeline? Book a free 30-minute call and we will tell you straight whether reranking is your win or whether the problem is somewhere else.