Problem
Waiting for a full LLM response before displaying anything creates a poor user experience — users watch a spinner for the whole generation before any output appears.
Solution
Use Server-Sent Events (SSE) or WebSockets to stream tokens from the LLM through your backend to the client as they are generated.
Implementation
- Provider: Call LLM with stream=True/streaming endpoint
- Backend: Use a streaming HTTP response (Next.js Route Handler with ReadableStream, Vercel AI SDK)
- Client: Consume the stream with the AI SDK useChat/useCompletion hooks, or fetch plus a stream reader (EventSource only makes GET requests, so it cannot call a POST chat endpoint)
- Safety gate: Buffer a sliding window of ~200 tokens; run a lightweight content safety check and terminate the stream if violations are detected mid-output
Vercel AI SDK pattern
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
const result = streamText({ model: openai('gpt-6.1-sol'), prompt });
return result.toUIMessageStreamResponse();
Trade-offs
- Pro: Users see the first words as soon as the model emits them, not after the full response
- Pro: Users can abort early if the response is going in the wrong direction
- Con: Mid-stream cancellation leaves partial content
- Con: Streaming safety checks have incomplete context — harder than full-response validation
When to use
All user-facing chat and content generation interfaces.
When to avoid
Background batch processing where latency doesn't matter.