Watch RAG work, step by step

Semantic caching in AI systems stores and retrieves previous LLM responses based on query similarity rather than exact matches. When a new query arrives, its embedding is compared to cached query embeddings. If the cosine similarity exceeds a threshold (typically 0.92–0.96), the cached response is returned directly, bypassing the expensive LLM call entirely.[1]This technique can reduce LLM API costs by 40–70% in enterprise applications where users frequently ask semantically similar questions. The trade-off is freshness — a cached response may become stale if the underlying knowledge base changes. Production implementations set a TTL (time-to-live) on cache entries, typically 24–72 hours for stable factual content.[2]The embedding model used for cache lookups is usually the same as the retrieval embedding model (e.g. text-embedding-3-small) to ensure semantic alignment. Some implementations use a cheaper, faster model for cache lookups and the full-quality model only for cache misses.