LLM Streaming

Streaming is the difference between a ChatGPT-like experience and a spinner that blocks the user for 10 seconds. LLMs generate token-by-token — streaming sends each token as it is produced over SSE or WebSockets. The engineering challenge is backpressure, partial JSON parsing, abort handling, and reconnection logic at scale.