Pure vector search loses exact matches. That is the failure mode most RAG demos hide.
Your embedding model maps text to meaning. Ask it for "error code SQL-1042" and it returns chunks about database errors in general. The exact code is sitting in your docs. The retriever just ranked three vaguely-related paragraphs above it. The model never sees the right chunk, so it guesses. The user gets a confident wrong answer.
This is not a prompt problem. It is a retrieval problem. And it is where most production RAG quietly breaks.
Why vectors miss
Embeddings are good at semantics and bad at literals. They cluster "how do I cancel" near "stop my subscription." That is the win. But they smear over part numbers, SKUs, function names, account IDs, and rare tokens. A dense vector has no special respect for the string `SQL-1042`. It just sees another token in a soup of meaning.
Your users do not type like that. They paste error codes. They search for the exact name of a policy. They quote a field from an invoice. Semantic similarity is the wrong tool for an exact-string query.
What BM25 does
BM25 is keyword search done properly. It is decades old, boring, and very good at its job. It ranks documents by term frequency, weighted so rare terms count more and common words count less. When a user searches `SQL-1042`, BM25 finds the one chunk that literally contains that string and puts it first.
BM25 has the opposite weakness. It cannot match "cancel" to "stop my subscription." No keyword overlap, no result. So neither method is complete on its own.
Run both, then fuse
Hybrid retrieval runs BM25 and vector search in parallel, then merges the results. You query both indexes, get two ranked lists, and combine them into one. The semantic side catches paraphrases. The keyword side catches exact terms. The fused list covers both.
The merge step matters. The two systems produce scores on different scales, so you cannot just add them. Reciprocal Rank Fusion is the standard answer. RRF ignores raw scores and uses rank position instead. A document that ranks high in either list floats to the top of the combined list. It is simple, has one tunable constant, and works without per-query calibration.
For most teams that is enough. You can add a reranker later if you need more precision. A cross-encoder rescores the top 20 fused results and reorders them with full attention to the query. It is slower and more expensive, so reserve it for the final shortlist, not the whole corpus.
What this looks like on AWS
You do not need exotic infrastructure. OpenSearch supports BM25 and k-NN vector search in the same index, so you can run both queries against one cluster and fuse the results in your application layer. Postgres with pgvector plus its built-in full-text search is another clean option for smaller corpora. Pick the one that matches the data you already have.
The expensive mistakes happen upstream of the query. Bad chunking splits a sentence in half and ruins both retrievers. Stale indexes serve last month's docs. No evaluation set means you ship changes blind and find out from an angry user. Build a small set of real questions with known-correct chunks, and measure retrieval before you touch the prompt.
The order of operations
Fix retrieval first. Most RAG projects we see reach for a bigger model when the real bug is that the right chunk never showed up. A larger model cannot answer from context it never received. The cheapest, highest-leverage work is in the retrieval layer, not the generation layer.
We built a support agent that answers in 12 minutes, down from 4 hours. The win was not a clever prompt. It was getting the right document in front of the model every time. Hybrid retrieval is a large part of how you do that.
Where this fits
This is the core of our RAG Chatbot work. We scope it fixed-price, build the retrieval layer to handle both semantic and exact-match queries, stand up an evaluation set so you can trust it, and hand you code you own. Same on every engagement: fixed price, you own the code, we exit. See pricing for the bands.
We are a 2-person team, ex-AWS, 12x AWS Certified, with 10 production products shipped in 7 months. RAG that works in production is one of the things we do.
If your chatbot is confidently wrong on the questions that matter most, the retriever is the first place to look. Want a second pair of eyes? Book a free 30-minute call and we will talk through where it is breaking.