Costs doubled last month, and traffic is flat. What do you look at first?
Why C is the answer
Token use per request, by step, including the cache hit rate
Measure before changing anything. Per-step tokens and cache hits show whether prompts grew, loops lengthened or caching broke.
In detail
- Measure before you change anything.
- Per-step tokens and cache hits show whether prompts grew, loops lengthened or caching broke.
- Alert on token use per request, not only on the monthly bill.
Use case
An illustrative scenario, not a real incident.
Costs double with flat traffic. Per-step tracking shows a prompt edit broke caching. Restoring the prefix brings the bill back down.
See it in different ways
stateDiagram-v2 [*] --> Retrieval Retrieval --> FixRetrieval: passage missing Retrieval --> Prompt: passage present Prompt --> Model: prompt is clear Model --> Labels: still wrong Labels --> [*]: check the test itself
flowchart TD
Q{"What changed?"}
Q -->|"Cache hit rate fell"| A0["Prefix changed"]
Q -->|"Tokens per step rose"| A1["Prompts grew"]
Q -->|"Steps per task rose"| A2["Loops got longer"]Key point: Measure per step before you change anything.
Why the other options are wrong
- A. Whether fewer users could be given access to the feature
- Traffic is flat, so cutting users hides the cause.
- B. The model release notes for any change in published pricing
- Your bill is the evidence, and token use shows the cause directly.
- D. A cheaper provider for the whole system, starting next month
- Switching provider is a large move, made before the cause is known.