In this article
Problem: Pure vector search misses exact keyword matches; pure keyword search misses semantic meaning. Neither alone is sufficient for production RAG.
Solution: Implement a multi-stage retrieval pipeline: broad candidate retrieval (keyword + semantic), fusion/dedup, then a re-ranking model to surface the most relevant chunks.
Implementation:
- Stage 1 — Broad retrieval: Run BM25 keyword search AND vector similarity search in parallel, retrieve top-50 from each
- Stage 2 — Fusion: Merge results using Reciprocal Rank Fusion (RRF) or weighted combination
- Stage 3 — Re-rank: Apply a cross-encoder re-ranker (Cohere Rerank, BGE-reranker, or ColBERT) to score the top-100 candidates
- Stage 4 — Select: Take top-5 re-ranked chunks for the LLM context
Trade-Offs:
- ✔ Pro: 15-30% improvement in retrieval accuracy over single-method search
- ✔ Pro: Handles both exact-match (product codes, names) and conceptual queries
- ✖ Con: Higher latency (50-200ms added for re-ranking)
- ✖ Con: More infrastructure: need both keyword index and vector store
When To Use: Production RAG systems where retrieval quality directly impacts user experience. When to avoid: Simple Q&A over small document sets where basic vector search suffices.
Related Articles
designing rag systems that actually scale