On this page
How it works
flowchart TD
L[LLM token stream] --> B[Backend stream]
B --> W[Sliding safety window]
W --> C{Safety check}
C -->|Pass| U[Progressive client output]
C -->|Violation| S[Terminate stream]The problem
Our AI chat and content generation features need to stream LLM responses to users rather than waiting for full completion. We need to choose between Server-Sent Events (SSE), WebSockets, and polling for delivering streaming tokens from the LLM to the browser.
What we weighed
Server-Sent Events (SSE) via Vercel AI SDK
Pros
- +Native browser support — no library needed
- +Vercel AI SDK provides toUIMessageStreamResponse() for SSE streaming (toDataStreamResponse() before AI SDK 5)
- +HTTP/1.1 compatible — works through proxies and firewalls
- +The AI SDK stream protocol is SSE-based, with keep-alive pings and reconnect support
- +One-directional (server → client) is all we need for token streaming
Cons
- −One-directional only — cannot send messages back to server mid-stream
- −Limited to 6 concurrent connections per browser and domain over HTTP/1.1 (HTTP/2 negotiates a much higher stream limit, 100 by default)
WebSocket
Pros
- +Bidirectional — useful if user needs to send corrections mid-stream
- +Lower overhead per message than HTTP for long-lived connections
Cons
- −Overkill for unidirectional token streaming
- −More complex to manage connection lifecycle, reconnection, and error handling
- −Vercel Edge Functions have limited WebSocket support
- −Not needed in our current use cases
Long-polling
Pros
- +Simplest implementation — standard HTTP
- +Works everywhere
Cons
- −High server load — each poll is a new HTTP request
- −Noticeable latency between token batches
- −Not suitable for <100ms token delivery
What we chose
We chose SSE via the Vercel AI SDK. The toUIMessageStreamResponse() pattern handles all the streaming mechanics, and the useChat/useCompletion hooks on the client consume the stream cleanly. Serving over HTTP/2 lifts the 6-connection limit. SSE meets all our current requirements at the lowest complexity.
What it committed us to
- •All LLM route handlers use streamText + toUIMessageStreamResponse()
- •Client components use useChat() or useCompletion() hooks from @ai-sdk/react
- •Implement stream abort support so users can stop generation early
- •Add streaming safety check: buffer 200-token sliding window for mid-stream content safety