Most RAG projects die at the chunking step, not the model step. The model is fine. The retrieval is feeding it garbage. If your chatbot answers confidently and wrong, the chunk it pulled probably never contained the answer in the first place.
Chunking is how you split documents before you embed them. It sounds like plumbing. It decides what your retriever can even find.
Fixed-size splitting is where it goes wrong
The default everyone reaches for is fixed-size chunks. Split every 1,000 characters, add some overlap, embed, done. It is fast to write and it loses information at every boundary.
A 1,000-character cut lands mid-sentence. It lands mid-table. It separates a heading from the paragraph that explains it. Now the embedding for that chunk describes half a thought. Retrieval scores it against the query and gets a weak match, so it ranks below a chunk that is confidently off-topic.
The failure mode is specific. The answer exists in your docs. The retriever returns four neighbors that don't contain it. The model hallucinates a plausible-sounding gap-filler. You ship it, a customer reads it, and you find out from a support ticket.
Split on structure, not on character count
The first real fix is to chunk on document structure. Markdown headings, HTML sections, PDF layout, function boundaries in code. Split where the author already split.
This keeps a heading attached to its body. It keeps a list together. Each chunk is now one coherent unit, so its embedding means one thing. Retrieval against a single clear topic beats retrieval against a smeared average of two.
For most internal-doc and knowledge-base use, structural chunking is the highest-return change you can make. It costs an afternoon. It usually moves answer quality more than upgrading the embedding model.
Add overlap, but small and deliberate
Overlap repeats a little text between adjacent chunks. The point is to avoid orphaning a sentence whose meaning depends on the one before it.
Keep it small. Ten to fifteen percent is usually enough. Large overlap inflates your index, slows retrieval, and returns near-duplicate chunks that crowd out genuinely different context. You want the model to see varied evidence, not the same paragraph three times.
Match chunk size to the question shape
Chunk size is a tradeoff, and the right size depends on what people ask. Short factual lookups ("what is the refund window") want small, precise chunks. Reasoning questions ("how does our pricing compare to theirs") want larger chunks that hold more context.
There is no universal number. We pick a size, build an eval set of real questions, and measure retrieval hit rate before we touch the model. If the right chunk is not in the top results, no model fixes that.
Keep the metadata, it does the quiet work
Strip your chunks of their source and you have thrown away free signal. Title, section, date, document type, customer tier. Attach all of it.
Metadata lets you filter before you rank. "Only chunks from current pricing docs." "Only this product line." Many retrieval failures are not bad embeddings. They are the right answer from last year ranking above the right answer from this quarter. Filtering on a date field fixes that cleanly.
Test retrieval on its own first
The discipline that separates working RAG from a demo: evaluate retrieval as its own layer. Write thirty real questions. For each, mark which chunk should come back. Then measure how often it lands in the top results.
If retrieval hit rate is low, stop. Do not tune prompts. Do not swap models. Fix the chunking, because everything downstream inherits this number. A support agent we built and run in production only works because the retrieval reliably surfaces the right passage first.
What this looks like as a project
We treat chunking as the first engineering decision in a RAG build, not a config default. Structural splitting, small overlap, sized to your real questions, metadata kept, retrieval measured before the model gets blamed.
Our RAG Chatbot is fixed-price. You own the code, you keep the eval set, and we exit. No retainer, no lock-in, no surprise on the bill. We have shipped 17 production products, 10 of them in our first 7 months with a 2-person team, and the retrieval work is where most of the quality actually comes from.
If your chatbot answers confidently and wrong, the chunking is the first place to look. Want a second set of eyes on it? Book a free 30-minute call and we'll talk through your docs and where retrieval is leaking.