RAG demos always work. Production doesn't.

Before any text can be embedded and indexed, long documents must be split into smaller chunks. Fixed-size chunking splits text every N tokens regardless of sentence boundaries, which is simple but can cut sentences awkwardly. Sentence-boundary chunking keeps each chunk grammatically complete but produces uneven chunk sizes. Paragraph chunking preserves the author's intended structure and often aligns best with semantic units. Overlapping chunks by 10 to 20 percent helps prevent important context from being split across a boundary, at the cost of extra storage and duplicate embeddings. Choosing a chunking strategy is one of the highest-leverage decisions in a retrieval pipeline, because retrieval quality can never exceed the quality of the chunks being searched.

A vector database stores high-dimensional embeddings and finds the nearest neighbors to a query vector using approximate nearest neighbor search. HNSW builds a multi-layer navigable graph that trades a small amount of recall for very fast lookup, while IVF partitions vectors into clusters and only searches the closest clusters. Popular vector databases include Qdrant, Pinecone, Weaviate, and pgvector as a Postgres extension. Index build time, memory footprint, and query latency all depend heavily on which ANN algorithm and distance metric these systems use under the hood.

An embedding model converts text into a dense numeric vector that captures semantic meaning, so that texts with similar meaning end up close together in vector space. Common choices include OpenAI text-embedding-3-small at 1536 dimensions, and open-weight alternatives like BGE or E5. Higher dimensionality can capture more nuance but increases storage and search cost. Most embedding models are trained with a contrastive objective so that cosine similarity between related passages is high. Picking the same embedding model for indexing and querying is essential, since vectors from different models are not comparable.

Semantic caching stores previous query embeddings alongside their generated responses. When a new query arrives, its embedding is compared against cached queries, and if cosine similarity exceeds a threshold the cached response is returned instantly instead of calling the language model again. This can cut LLM API costs by forty to seventy percent in applications where users frequently ask semantically similar questions. The main trade-off is staleness: cached answers can go out of date, so production systems attach a time-to-live to each cache entry.

Retrieval quality is measured with the same metrics used in classic information retrieval. Precision at K is the fraction of the top K retrieved chunks that are actually relevant, while recall at K is the fraction of all relevant chunks that were successfully retrieved. Mean reciprocal rank rewards systems that place the first correct result near the top. A labeled golden test set, with queries mapped to known relevant documents, lets teams measure these metrics automatically every time the pipeline changes.

Zero-shot prompting asks the model to complete a task with instructions alone and no examples. Few-shot prompting adds two or three worked examples directly in the prompt so the model can infer the desired format and tone. Chain-of-thought prompting explicitly asks the model to reason step by step before producing a final answer, which tends to improve accuracy on multi-step arithmetic and logic problems. System prompts set persistent behavior for an entire conversation.

An agent extends a language model with the ability to call external tools such as search, calculators, or internal APIs. The ReAct pattern interleaves reasoning traces with actions: the model thinks about what to do, calls a tool, observes the result, and repeats until it has enough information to answer. Multi-step agents must track intermediate state and decide when to stop, since an unbounded loop can burn through tokens or get stuck retrying a failing tool call.

Fine-tuning updates a model's weights on a custom dataset so the model internalizes new behavior, tone, or domain knowledge directly. Retrieval augmented generation instead keeps the base model frozen and supplies fresh, verifiable facts at query time from an external knowledge base. Fine-tuning is a better fit for teaching a consistent style or output format, while retrieval is a better fit for knowledge that changes often. Many production systems combine both approaches.

Hallucination happens when a language model states something confidently that is not actually true or not supported by any source. Grounding the model in retrieved passages and instructing it to answer only from the provided context substantially reduces ungrounded claims. Asking the model to cite which chunk supports each statement makes hallucinations easier to catch during review. Lowering the sampling temperature is a simple, effective mitigation.

Every language model has a maximum context window measured in tokens, and stuffing it with irrelevant retrieved chunks wastes budget and can degrade answer quality. Effective context management ranks candidate chunks by relevance and keeps only the top few that fit comfortably within the token budget, reserving room for the system prompt, conversation history, and the model's own response. Sliding window approaches keep only the most recent turns of a long conversation.

A first-stage vector search is fast but approximate, so many production pipelines add a second-stage reranker to improve precision. A cross-encoder reranker jointly scores the query and each candidate passage together, rather than comparing two independently computed vectors, which produces a much more accurate relevance score at the cost of extra latency. A typical pattern retrieves fifty candidates with a cheap vector search, then reranks down to the final top five before sending them to the language model.

A similarity threshold discards retrieved chunks whose score falls below a chosen cutoff, trading recall for precision. Set the threshold too low and irrelevant chunks leak into the prompt, diluting context and inviting hallucination. Set it too high and genuinely relevant chunks get discarded, starving the model of information it needs. The right value always depends on the embedding model and the corpus, and should be tuned against a labeled test set rather than guessed.