A latency target is hard to hit, and answers are long. What do you try first?
Why B is the answer
Stream the output, and check whether a faster tier passes your evals
Streaming lowers perceived latency, and a faster tier may lower the real figure. Your evals show whether quality holds.
In detail
- Streaming lowers perceived latency.
- A faster tier may lower the real figure.
- Your evals show whether quality holds.
- Measure time to first token as well as total time.
Use case
An illustrative scenario, not a real incident.
A chat product misses its latency target. Streaming fixes how it feels, and a faster tier passes the evals, which also brings the real figure down.
See it in different ways
sequenceDiagram
participant U as Request
participant F as Fast model
participant V as Validation
participant S as Stronger model
U->>F: Handle first
F->>V: Check the output
alt Fails validation
V->>S: Escalate
else Passes
V-->>U: Return the answer
endflowchart LR S0["Stream the reply"] --> S1["Test a faster tier"] --> S2["Check evals"] --> S3["Ship if quality holds"]
Key point: Stream first, then test a faster tier.
Why the other options are wrong
- A. Ask users whether they would accept the current wait for a few weeks
- Asking users to accept it does not meet the target.
- C. Add a second model that reviews every answer before it is shown
- A reviewing model adds more waiting before users see anything.
- D. Increase the size of each retrieved chunk so that fewer searches run
- Larger chunks add input, which can slow generation.