LLM latency has unique characteristics: Time To First Token (TTFT) determines perceived responsiveness for streaming, token generation rate determines reading speed, total latency scales with output length. Bottlenecks can be network, model inference, or pipeline overhead (retrieval, reranking). Streaming responses dramatically improve UX for long outputs. Understanding the latency breakdown enables targeted optimization.
How streaming improves perceived latency.
For responses longer than 50 tokens, always stream. Users experience the response as starting at TTFT (0.8s) rather than total latency (4s+). Never make users stare at a blank screen for 4 seconds.
Cross-encoder reranking (BAAI/bge-reranker) processes query × document pairs sequentially. 20 candidates × 30ms each = 600ms. For latency-sensitive paths, use a faster bi-encoder reranker, reduce candidate count, or skip reranking for simple queries.
Many users ask similar questions. A semantic cache (embedding + cosine similarity match > 0.92) returns pre-computed answers in < 100ms instead of running the full pipeline. Cache recent popular queries in Redis.
Sign in to share your feedback and join the discussion.