Users say a chatbot feels slow, although total response time is acceptable. What helps?
Why D is the answer
Stream tokens as they are generated
Streaming shows progress at once.
In detail
- Streaming shows progress at once.
- Time to first token matters here.
Use case
An illustrative scenario, not a real incident.
A reply takes ten seconds either way, but streaming shows words at once, so it feels quick.
See it in different ways
flowchart LR A["Users"] --> B["API"] B --> C["Batched GPU inference"] C --> D["Streamed tokens"] B --> E["Latency and error monitoring"]
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMflowchart TD
Q{"How is the reply sent?"}
Q -->|"All at the end"| A0["Feels slow"]
Q -->|"As it is generated"| A1["Feels quick"]Key point: Streaming improves perceived speed.
Why the other options are wrong
- A. Add more detail to the system prompt for each request
- A longer prompt slows it down.
- B. Switch to a larger model with a longer context window
- A larger model is often slower.
- C. Wait for the full answer before showing anything
- That is what feels slow.