Retrieval-Augmented Generation injects fresh, authoritative context into the prompt at inference time. The pipeline chunks documents, embeds them, stores vectors, retrieves the top-k semantically relevant chunks for a user query, then generates an answer that cites sources — eliminating hallucinations on private data.
CI/CD pipeline for an evolving RAG knowledge base — re-embed on doc change, evaluate retrieval quality.
Too small (<200 tok) loses context. Too large (>1000 tok) dilutes retrieval signal. 400-600 tokens with 50-100 overlap is the sweet spot for most prose.
A cross-encoder reranker on top-20 candidates typically lifts top-3 precision by 20-40 points. The latency cost (~200ms) is almost always worth it.
Combine BM25 (keyword) + dense vector + reciprocal rank fusion. Pure semantic search misses exact-match queries (SKUs, error codes, product names).
Switching embedding models requires re-indexing the entire corpus. Pin model version in metadata; never mix vectors from different models in the same index.
Instruct the LLM: “only answer if context is sufficient, otherwise reply 'I don't know'”. Add citation IDs in the prompt template; reject answers without citations.
Build a golden set of (question, expected_chunks, expected_answer). Track recall@k, MRR, faithfulness, and answer relevance on every index change.
Sign in to share your feedback and join the discussion.