In this article
Problem: LLM API calls are expensive and slow. Traditional exact-match caching misses semantically identical queries phrased differently.
Solution: Embed each prompt using a fast embedding model, store the embedding alongside the cached response, and on new requests compare the query embedding against cached embeddings using cosine similarity. If the similarity exceeds a threshold (e.g., 0.95), return the cached response.
Implementation:
- Hash the system prompt + user message
- Generate embedding via a lightweight model (e.g., text-embedding-3-small)
- Query vector store for nearest match above threshold
- Cache hit → return stored response; cache miss → call LLM, store result + embedding
Trade-Offs:
- ✔ Pro: 60-90% cost reduction for repetitive workloads (customer support, FAQ)
- ✔ Pro: Sub-50ms response for cache hits vs 1-5s for LLM calls
- ✖ Con: Stale responses if underlying data changes — need TTL or invalidation
- ✖ Con: Threshold tuning is domain-specific; too low = wrong answers, too high = low hit rate
When To Use: High-volume, repetitive query patterns — chatbots, search, customer support. When to avoid: Creative/generative tasks, real-time data queries, low-volume APIs.
Related Articles
the ai gateway pattern