Your RAG chatbot is wrong, and you blame the model. That is usually the wrong call.
We have shipped 17 production products, 10 of them in our first 7 months. A few of them retrieve over documents. The pattern is consistent. When retrieval gives bad answers, the corpus is almost always the cause, not the LLM. Swapping models is the expensive way to not fix the problem.
Here is the failure mode in plain terms. The model only sees what retrieval hands it. If the right passage never makes the top-k, no model recovers. A smarter model writes a more confident wrong answer over the same bad context. You pay more per token to be misled with better grammar.
What "the corpus is the problem" actually looks like
Start with the obvious one: stale and duplicate documents. Your corpus has the 2023 refund policy and the 2025 one. Both embed near the query. Retrieval pulls the old one half the time. The model is not hallucinating. It is reading the wrong true document.
Next: chunking that splits answers in half. A policy lives across two paragraphs. Your splitter cuts between them at 512 tokens. Now the chunk with the question has no answer, and the chunk with the answer has no question. Cosine similarity ranks both low. The fix is structure-aware chunking, not a new embedding model.
Then there is the metadata you never added. No source, no date, no product line, no access scope. So you cannot filter, you cannot rank by recency, and you cannot tell the model which document wins when two disagree. Retrieval becomes a popularity contest among near-duplicates.
Last: documents that were never written for retrieval. A 60-page PDF with a table of contents, headers, and footers on every page. The boilerplate embeds as strong signal. The actual content gets buried. Garbage in, confident garbage out.
How to tell the corpus is at fault
Run the test before you touch the model. Take 30 real questions. For each, check whether the correct passage appears anywhere in the top-k retrieved chunks.
If the right chunk is not in the top-k, the model was never going to get it right. That is a retrieval problem, full stop. If the right chunk is in the top-k and the answer is still wrong, now you have a generation problem worth a model conversation. Most teams skip this test and buy a bigger model. They are debugging the wrong layer.
A second check: read the chunks, not the answers. Pull the raw retrieved text for ten bad responses. You will see the duplicates, the half-policies, the header noise. The bug is visible in the context window before the model says a word.
What we actually do
The work is unglamorous and it is where the wins are. We dedupe and date the corpus, so the current document beats the old one. We chunk along document structure, so answers stay whole. We add metadata for filtering and recency, so retrieval can prefer the right source. We strip boilerplate before it ever embeds.
Only after that do we tune top-k, rerankers, and the prompt. The model is the last knob, not the first. One support agent we built and run in production works because of clean retrieval over a corpus people trust, not because of a heroic model choice.
A note on cost while we are here. A bigger model is a recurring per-token bill for a one-time corpus fix. Most of our clients are already overpaying somewhere; most AWS bills hide savings nobody is tracking. Spending more on inference to mask a data problem is the same mistake in a different column.
Where this fits
This is the RAG cluster, and it maps to one SKU. Our RAG Chatbot build is fixed-price, and the corpus work is the build, not an upsell. We scope it, we price it once, and you own the code when we exit. No retainer, no surprise invoice when the model bill creeps up. See pricing for the band.
We are a 2-person team, ex-AWS Federal SA, 18x cloud certifications and 13x AWS Certified. We have run this exact diagnosis enough times to know where to look first. It is almost never the model.
If your RAG answers are wrong and you are not sure which layer to blame, that is a 30-minute conversation. Book a call and we will tell you what the top-k test would show.