Problem

Waiting for a full LLM response before displaying anything creates a poor user experience — users watch a spinner for the whole generation before any output appears.

Solution

Use Server-Sent Events (SSE) or WebSockets to stream tokens from the LLM through your backend to the client as they are generated.

Implementation

  1. Provider: Call LLM with stream=True/streaming endpoint
  2. Backend: Use a streaming HTTP response (Next.js Route Handler with ReadableStream, Vercel AI SDK)
  3. Client: Consume the stream with the AI SDK useChat/useCompletion hooks, or fetch plus a stream reader (EventSource only makes GET requests, so it cannot call a POST chat endpoint)
  4. Safety gate: Buffer a sliding window of ~200 tokens; run a lightweight content safety check and terminate the stream if violations are detected mid-output

Vercel AI SDK pattern

import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

const result = streamText({ model: openai('gpt-6.1-sol'), prompt });
return result.toUIMessageStreamResponse();

Trade-offs

  • Pro: Users see the first words as soon as the model emits them, not after the full response
  • Pro: Users can abort early if the response is going in the wrong direction
  • Con: Mid-stream cancellation leaves partial content
  • Con: Streaming safety checks have incomplete context — harder than full-response validation

When to use

All user-facing chat and content generation interfaces.

When to avoid

Background batch processing where latency doesn't matter.