From high-level architecture to runtime sequence — flip through the lenses to see how a RAG system actually moves data from a document to an answer.
High-level system architecture showing the two stages of a RAG system: offline ingestion and online serving — with latency budgets, vector dimensions, and the bridging vector store.
Small chunks (128t) → high precision, low recall. Large chunks (2048t) → high recall, low precision. 512t with 50t overlap is the production sweet spot.
Vector search is fast but imprecise (~71% P@5). Cross-encoder reranking is slow but precise (~89% P@5). Over-fetch with vector search, rerank for quality.
Query and document vectors MUST use the same embedding model. Mixing models produces random results — they live in incompatible geometric spaces.
Chunking + embedding happens once, not per-query. Index once, query forever. Re-index only on document updates — not for every user question.
Budget: system prompt (500t) + context (5×512=2560t) + query (100t) + response (800t) = ~3960t. Stay under your model's context limit.
Embed query: 21ms. HNSW search: 23ms. Cross-encoder rerank: 360ms. LLM generation: 1200ms. Total E2E: ~1.6s P50. Cache for P95 < 50ms.
Sign in to share your feedback and join the discussion.
