Users of a chat feature wait 20 seconds and then see the whole answer at once. Total generation time cannot change. What improves the experience most?
Why B is the answer
Stream the response and render tokens as they arrive
Streaming fixes perceived latency without changing generation time. Lowering max_tokens cuts answers off, retries add load, and batch work is for jobs that can wait.
In detail
- The wait is a problem of perceived speed. Total generation time stays the same.
- Streaming sends text as it is produced, so users see progress at once.
- Show a stop button during streaming, so people can cancel a reply they do not need.
Use case
An illustrative scenario, not a real incident.
A legal assistant takes 20 seconds to draft a clause, and users think it has frozen. With streaming, the first words appear quickly and the same draft reads as it is written.
See it in different ways
flowchart LR A["message_start"] --> B["content_block_start"] B --> C["content_block_delta"] C --> C C --> D["content_block_stop"] D -->|"Next block"| B D --> E["message_delta"] E --> F["message_stop"]
sequenceDiagram participant A as API participant C as Your client A-->>C: message_start A-->>C: content_block_start A-->>C: content_block_delta, many times A-->>C: content_block_stop A-->>C: message_delta A-->>C: message_stop
sequenceDiagram participant P0 as User participant P1 as Your app participant P2 as Claude P0->>P1: Ask a question P1->>P2: Request with streaming on P2-->>P1: Text deltas P1-->>P0: Render each delta
Key point: Streaming improves how fast a reply feels, not how fast it finishes.
Why the other options are wrong
- A. Lower max_tokens so replies are shorter
- This cuts answers off rather than making the same answer feel faster.
- C. Retry the request if it takes over 10 seconds
- A retry restarts the work and adds load, so the wait gets longer.
- D. Move the request to the Message Batches API
- Batch work is for jobs that can wait hours, which a chat cannot.