Many users ask nearly identical questions. What avoids unnecessary model calls?
Why A is the answer
Semantic caching
A semantic cache returns a stored answer for a similar question, which avoids an unnecessary model call.
In detail
- Semantic caching reuses answers for similar questions.
- Set a similarity threshold, and an expiry.
Use case
An illustrative scenario, not a real incident.
Thousands ask about store hours in different words. A semantic cache returns the stored answer, with no model call.
See it in different ways
flowchart TD
A["Request"] --> B{"Answer in the cache?"}
B -->|Yes| C["Return it, no model call"]
B -->|No| D{"Simple?"}
D -->|Yes| E["Lower-cost model"]
D -->|No| F["More capable model"]mindmap
root((Cost))
Avoid calls
Semantic caching
Smaller prompts
Trim context
Prompt caching
Right model
Tiered usageflowchart TD
Q{"Is a similar question cached?"}
Q -->|"Yes"| A0["Return cached answer"]
Q -->|"No"| A1["Call the model"]Key point: Semantic caching avoids repeat calls.
Why the other options are wrong
- B. A larger output length on each request
- That makes each call cost more.
- C. A second model that repeats each answer
- That doubles the calls.
- D. Fine-tuning a new model for each question
- That is far too costly.