Your long reference documents are resent on every call. How do you cut cost and latency?
Why A is the answer
Put them early in the prompt with a cache breakpoint after them
Caching needs an identical prefix, so stable content goes first. Cache reads cost a fraction of the base input price.
In detail
- Prompt caching needs an identical prefix, so stable content must come first.
- Place a cache breakpoint after that content.
- Later calls read it from the cache at a fraction of the base input price.
- Keep anything that changes, such as the user's question, after the breakpoint.
Use case
An illustrative scenario, not a real incident.
A legal assistant sends the same long reference pack on every call. Placing it early, with a cache breakpoint after it, means later calls read it from the cache at a fraction of the input price.
See it in different ways
flowchart LR A["Tools"] --> B["System prompt"] B --> C["Long reference docs"] C --> D["Cache breakpoint"] D --> E["New user question"]
flowchart LR A["Tools"] --> B["System"] B --> C["Messages"] C --> D["Newest turn, not cached"]
flowchart LR S0["Tools"] --> S1["System and documents"] --> S2["Cache breakpoint"] --> S3["Changing messages"]
Key point: Stable content first, then the breakpoint.
Why the other options are wrong
- B. Send them only every fifth turn and rely on the model to remember them
- The model has no memory between calls. It only sees what you send.
- C. Summarize them again on each call with a separate model request first
- Summarizing adds a model call each time and loses detail.
- D. Move them into the newest user message so that the model sees them last
- Content in the newest message comes after whatever changes, so it will not stay in a stable prefix.