Problem
Pure vector search misses exact keyword matches; pure keyword search misses semantic meaning. Neither alone is sufficient for production RAG.
Solution
Implement a multi-stage retrieval pipeline: broad candidate retrieval (keyword + semantic), fusion/dedup, then a re-ranking model to surface the most relevant chunks.
Implementation
- Stage 1 — Broad retrieval: Run BM25 keyword search AND vector similarity search in parallel, retrieve top-50 from each
- Stage 2 — Fusion: Merge results using Reciprocal Rank Fusion (RRF) or weighted combination
- Stage 3 — Re-rank: Apply a re-ranker (a cross-encoder such as Cohere Rerank or BGE-reranker, or a late-interaction model such as ColBERT) to score the top-100 candidates
- Stage 4 — Select: Take top-5 re-ranked chunks for the LLM context
Trade-offs
- Pro: Usually better retrieval than either method alone; measure the gain on your own eval set
- Pro: Handles both exact-match (product codes, names) and conceptual queries
- Con: Higher latency, because re-ranking runs a model over every candidate
- Con: More infrastructure: need both keyword index and vector store
When to use
Production RAG systems where retrieval quality directly impacts user experience.
When to avoid
Simple Q&A over small document sets where basic vector search suffices.