Problem

Pure vector search misses exact keyword matches; pure keyword search misses semantic meaning. Neither alone is sufficient for production RAG.

Solution

Implement a multi-stage retrieval pipeline: broad candidate retrieval (keyword + semantic), fusion/dedup, then a re-ranking model to surface the most relevant chunks.

Implementation

  1. Stage 1 — Broad retrieval: Run BM25 keyword search AND vector similarity search in parallel, retrieve top-50 from each
  2. Stage 2 — Fusion: Merge results using Reciprocal Rank Fusion (RRF) or weighted combination
  3. Stage 3 — Re-rank: Apply a re-ranker (a cross-encoder such as Cohere Rerank or BGE-reranker, or a late-interaction model such as ColBERT) to score the top-100 candidates
  4. Stage 4 — Select: Take top-5 re-ranked chunks for the LLM context

Trade-offs

  • Pro: Usually better retrieval than either method alone; measure the gain on your own eval set
  • Pro: Handles both exact-match (product codes, names) and conceptual queries
  • Con: Higher latency, because re-ranking runs a model over every candidate
  • Con: More infrastructure: need both keyword index and vector store

When to use

Production RAG systems where retrieval quality directly impacts user experience.

When to avoid

Simple Q&A over small document sets where basic vector search suffices.