Cosine similarity is lying to you. A user asks “How do I reset my password?” Your vector database returns chunks about password security, MFA setup, and creating strong passwords. All semantically close. All factually useless. The correct three-step procedure sits in chunk #47 — invisible to top-3 retrieval.
Naive top-k leaves 15-30% recall behind. This guide walks through four retrieval layers — hybrid search, query expansion, and cross-encoder reranking — that recover every missing point. Each layer adds measurable improvement. Each comes with working Python code.
If you’re setting up a RAG system from scratch, start with our guide to integrating LLM APIs before optimizing the retrieval layer.
Why Top-K Cosine Similarity Isn’t Enough
Vector search finds chunks that are semantically nearby. That’s not the same as finding chunks that answer the question. Three specific failure modes:
Vocabulary mismatch. Your docs say “purchase refund procedure.” Users search “how do I get my money back.” Zero lexical overlap. Pure vector search handles this decently. But when your docs say “RFC 6749 compliance” and users search “OAuth setup,” vector search often fails — technical terms with distinct vector representations but identical real-world meanings.
Exact match requirements. Error codes. Product SKUs. API endpoint names. “ERR-429-TOKEN” must match “ERR-429-TOKEN” — not “rate limit error” which is semantically close but factually different. Vector search fuzzes out exact matches. Keyword search (BM25) catches them perfectly.
The “first result” trap. Vector search returns the closest chunks. The most relevant chunk is often the 5th or 12th closest — because relevance depends on whether the chunk answers the question, not on how close it sits in embedding space.
Layer 1→2: Hybrid Search with Reciprocal Rank Fusion
Add BM25 keyword search alongside vector search. Merge results with Reciprocal Rank Fusion (RRF). This single addition typically recovers 10-15 points of retrieval recall.
def reciprocal_rank_fusion(dense_results, sparse_results, k=60, alpha=0.7):
"""Merge dense (vector) and sparse (BM25) results with RRF.
alpha=0.7 means dense results contribute 70% of the final score.
Higher alpha → vector-dominant. Lower alpha → keyword-dominant.
"""
scores = {}
for rank, doc in enumerate(dense_results):
scores[doc.id] = scores.get(doc.id, 0) + alpha / (rank + k)
for rank, doc in enumerate(sparse_results):
scores[doc.id] = scores.get(doc.id, 0) + (1 - alpha) / (rank + k)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)The 70:30 dense-to-sparse weighting is a strong default. Tune it based on your data: document-heavy knowledge bases with lots of technical terminology benefit from higher sparse weight (0.5). Narrative content with natural-language queries benefits from higher dense weight (0.8).
BM25 catches what vector search misses: product codes, error numbers, API endpoints, exact phrases. Vector search catches what BM25 misses: paraphrased questions, conceptual relationships, cross-lingual queries. Together, they’re substantially more robust than either alone. For reducing the per-query cost of running both retrieval passes on every search, our cost optimization guide covers caching strategies that lower your LLM spend across the pipeline.
Layer 2→3: Query Expansion with HyDE & Multi-Query
HyDE (Hypothetical Document Embeddings). Instead of embedding the user’s query, ask an LLM to generate a hypothetical document that would answer it. Embed that document. Search for chunks similar to the hypothetical answer.
Counterintuitively, a generated document often sits closer in vector space to relevant real chunks than the original query does. “How do I reset my password?” generates “To reset your password, navigate to Settings > Security > Password Reset. Enter your current password, then type your new password twice. Click Save. You’ll receive a confirmation email.” This hypothetical answer is rich in the vocabulary and structure of actual documentation — and its embedding lands near the real password reset instructions.
Multi-Query. For ambiguous queries — “what’s the policy on that thing from last week?” — generate 3-5 query variations. “What policy changed last week?” “Recent policy updates.” “New company policies announced.” Retrieve against all variations. Merge and deduplicate results. Typical recall gain on ambiguous queries: 5-10 points.
The cost: one extra LLM call per user query. For GPT-4o Mini, that’s roughly $0.0005. The recall improvement is worth roughly 100× that in reduced hallucination-driven support tickets.
Layer 3→4: Cross-Encoder Reranking
This is the highest-ROI single change you can make to a RAG retrieval pipeline. The Cohere Rerank API is the leading managed service — $1 per 1,000 queries, zero infrastructure, production-ready in minutes.
For teams that need self-hosted reranking, bge-reranker-large provides comparable cross-encoder scoring quality with no per-query cost, at the expense of GPU provisioning.
A bi-encoder (what your embedding model is) encodes the query and each document independently, then compares them with cosine similarity. Fast but shallow — the query and document never “see” each other during encoding.
A cross-encoder reads the query and document together, as a pair, and outputs a relevance score. It understands that “password reset” in the query combined with “password security best practices” in the document is a partial match — not the answer the user needs. Much slower per pair, but astronomically more accurate.
The production pattern: use fast bi-encoder search to retrieve 20-50 candidates. Feed each (query, candidate) pair through a cross-encoder. Keep the top 3-5 for LLM generation.
import cohere
co = cohere.Client()
results = co.rerank(
query="How do I reset my password?",
documents=[chunk.text for chunk in top_50_candidates],
top_n=5,
model="rerank-english-v3"
)
top_chunks = [top_50_candidates[r.index] for r in results.results]Retrieval is one stage in a larger pipeline. The embedding model you choose determines which chunks surface; the vector database you pick determines how fast — both choices shape what hybrid search can retrieve and how quickly it returns.
Cohere Rerank vs. bge-reranker-large:
| Cohere Rerank | bge-reranker-large (self-hosted) | |
|---|---|---|
| Cost | ~$1/1,000 queries | GPU cost only (~$0.84/hr for T4) |
| Precision gain | 10-20 points over vector-only top-k | 8-18 points |
| Latency per query | ~150ms | ~80ms (on GPU) |
| Setup | One API call | Docker container + GPU |
Cohere Rerank is easier. bge-reranker-large is cheaper at volume and keeps data in your VPC. Both deliver massive precision improvements. The choice comes down to whether $1 per 1,000 queries is material to your budget.
These retrieval layers feed directly into the generation stage — where the retrieved chunks become the context window for your LLM’s response.
The cumulative effect: Naive RAG recall ~63%. Add hybrid search → ~76%. Add HyDE → ~82%. Add reranking → ~92%. Cost: roughly $0.003 per query for the reranker call. Twenty-seven points of recall for 0.3 cents per query. If you make one change to your RAG pipeline this month, make it reranking. For optimizing the cost of the LLM calls behind HyDE and Multi-Query expansion, our prompt caching guide can eliminate up to 90% of repeated input token costs.
Retrieval Mistakes That Survive Code Review
Your RAG pipeline passes code review. The abstraction is clean. The tests pass. Then it ships — and the things that break have nothing to do with code quality.
The chunk-size cargo cult. Your team defaults to 512-token chunks because “that’s what the LangChain tutorial uses.” Your legal documents have paragraphs that run 800 tokens. Your FAQ entries average 120 tokens. Every chunk boundary that splits a coherent unit of meaning creates a retrieval failure that no reranker can fix. One team at a contract management startup spent three weeks tuning their reranker before realizing their chunk size was splitting every clause in half — the right answer was literally never in a single chunk. Fix: measure the distribution of semantic units in your documents. Chunk to boundaries that match your actual data, not a tutorial’s defaults.
The embedding-model mismatch. You built your vector index with text-embedding-ada-002 in 2024. In 2026 you upgraded to text-embedding-3-large and re-indexed. But your retrieval recall dropped 12 points. Why? Ada-002’s 1536-dimensional vectors have different nearest-neighbor geometry than 3-large’s 3072 dimensions. Your old similarity threshold of 0.78 is now meaningless. Your hard-coded top_k=5 retrieves a different set of documents. One e-commerce team discovered this when their product search went from 89% relevance to 73% after a “trivial” embedding model upgrade. Fix: never change embedding models without re-benchmarking your retrieval metrics end-to-end. The model name changed. The geometry changed. Your thresholds must change too.
The metadata blindness. You index chunks with no metadata. A user searches “refund policy for EU orders.” Your hybrid search retrieves chunks about refunds. But three of the top five chunks are from the US policy page, one is from the outdated 2023 policy, and only one is from the current EU policy. Without metadata filtering — region=EU, version=current — your retrieval is returning factually correct chunks from the wrong jurisdiction. One SaaS company shipped a GDPR compliance bot that was citing California privacy law chunks for EU customer queries. The chunks were topically relevant. The metadata was missing. The legal exposure was real.
The silent reranker failure. Your Cohere Rerank API call has a 0.5% timeout rate. You didn’t notice because 0.5% is below your alerting threshold. But the fallback path — using raw vector results when reranking fails — means 1 in 200 queries silently returns un-reranked chunks. For a system handling 50,000 queries a day, that’s 250 users a day getting degraded answers. One finance team caught this only because a quant noticed their Q3 earnings summary bot was occasionally citing the wrong quarter’s data. The root cause: timed-out reranker calls falling back to cosine-similarity-only results that ranked Q2 chunks above Q3 chunks.
FAQ
Is hybrid search’s extra latency worth it?
BM25 search adds 5-15ms to query latency. RRF merging is sub-millisecond. The total overhead of hybrid over pure vector is negligible compared to the 200ms-2s LLM generation time. And the 10-15 point recall gain means fewer queries need retries or human escalation — net latency to resolution usually decreases with hybrid search.
Cohere Rerank vs. open-source bge-reranker — which should I use?
Cohere: use when you want one API call and zero infrastructure. Best for teams under 100K queries/month. bge-reranker-large: use when you have GPU infrastructure, need data to stay in your VPC, or exceed 500K queries/month where self-hosting becomes meaningfully cheaper. The precision difference is 2-3 points in Cohere’s favor on most benchmarks — measurable but not transformative.
What if the HyDE-generated document contains errors?
The hypothetical document doesn’t need to be factually correct. It needs to be stylistically and structurally similar to the documents that contain the correct answer. A HyDE document that says “To reset your password, click the giant red button” will still embed near real password reset documentation — because the surrounding vocabulary and structure are authentic. HyDE retrieves relevant chunks. The LLM generates the actual answer from those chunks. The hypothetical document is a retrieval tool, not an answer.
When is top-k retrieval good enough — no reranking needed?
When your queries and documents share high vocabulary overlap (FAQ matching FAQ, support ticket matching knowledge base article), when latency is so constrained that 150ms for a reranker API call is unacceptable, or when you’ve measured your retrieval recall at 90%+ without reranking. For everyone else: add reranking. The improvement is large, consistent, and cheap.
How do I know if my chunking strategy is wrong?
Measure retrieval recall at the semantic-unit level, not the chunk level. If a single coherent answer spans three chunks in your index, your chunk boundaries are cutting through meaning. Test: take 50 real user queries, manually identify which document sections contain the correct answer, then check whether those sections appear intact within single chunks in your index. If more than 15% of answers span chunk boundaries, your chunk size or overlap is wrong. The fix is rarely “make chunks bigger” — it’s “chunk at natural boundaries” (paragraph breaks, section headers, FAQ boundaries) with overlap generous enough to capture context without splitting atomic facts. For running systematic evaluation of your chunking strategy against a held-out query set, see our CI/CD testing guide which covers automated retrieval benchmarking.
Naive retrieval is the biggest silent quality leak in production RAG. Each layer fixes one failure mode: hybrid search plugs the exact-match gap, HyDE catches the ambiguous queries, reranking separates “close” from “correct.” None of them require infrastructure changes. All of them pay for themselves within the first thousand queries. The 62% pipeline and the 92% pipeline are separated by about 200 lines of Python and $3 per thousand queries.
Hybrid search, HyDE, and reranking — all powered through one endpoint. Start building on TokSpan — embeddings, chat, and Cohere Rerank on the same bill.