Streaming sends tokens to the client as the model generates them, dropping perceived latency from "model speed" to "first-token speed" (~100ms). Server-Sent Events (SSE) is the standard transport; client-side, partial JSON parsing and incremental rendering complete the experience.
Streaming request lifecycle including reconnection and metrics.
Users perceive first-token-time, not total. Stream + skeleton loaders make 5s feel like 300ms.
Nginx, ALB, Cloudflare all buffer by default. Set X-Accel-Buffering: no, disable_response_buffering, or chunked encoding.
Use Last-Event-ID for SSE resume. Critical on mobile networks where connections drop mid-stream.
For structured outputs, use partial-json or json-streaming libs. Lets UI render objects as they form.
Two key SLOs: TTFT (P95 < 500ms) and tokens/sec (varies by model). Alert on either degrading.
On UI abort, cancel the upstream LLM call (AbortController). Otherwise tokens (and $) keep flowing.
Sign in to share your feedback and join the discussion.