LLM Latency: Measurement and Optimization

Problem Context

"The LLM is slow" isn't a metric. There are at least four latencies that matter for an LLM-backed feature, and they degrade for different reasons. Confusing them — or aggregating them into a single "p95 latency" — hides every interesting failure mode.

The Four Latencies

• TTFT (time-to-first-token) — from request sent to first token received. Dominates perceived responsiveness.
• TPS / TPOT (tokens-per-second / time-per-output-token) — generation throughput once streaming starts.
• Total generation time — TTFT + (output_tokens × TPOT). The number your billing-ops team cares about.
• End-to-end (E2E) — what the user sees: includes auth, retrieval, guardrails, post-processing, network back to client.

flowchart LR
    U[User clicks] -->|t0| API[API gateway]
    API -->|auth| RAG[Retrieval]
    RAG -->|t1| LLM[LLM provider]
    LLM -->|t2: first token| STR[Stream to client]
    STR -->|t3: last token| POST[Guardrails + format]
    POST -->|t4| U
      

TTFT = t2 - t1. TPOT = (t3 - t2) / output_tokens. E2E = t4 - t0.

What Drives Each

• TTFT — input length (KV cache prefill), provider load, region, model size, queueing.
• TPOT — model size, output length tier (some providers throttle long outputs), provider capacity.
• Total generation — output length × TPOT; cut output length, cut total.
• E2E — everything above + your own retrieval/guardrails/post-processing overhead.

The Right Percentiles

Always look at p50, p95, p99per metric. P50 tells you the "feels like", p95 catches creeping regressions, p99 catches the long-tail nightmares (cold model loads, regional failover). Mean is a lie for LLM latency distributions — they are heavy-tailed.

Streaming Changes Everything

With streaming, the user starts reading as the tokens arrive. The metric you optimise is no longer total generation; it is TTFT plus sustained TPOT > ~30 tokens/sec (the comfortable reading speed). A model that finishes in 4 s total but starts in 1.8 s feels worse than one that finishes in 6 s but starts in 0.4 s.

The Levers

• Cut input tokens — shorter context = faster prefill = lower TTFT. Often the cheapest win.
• Prompt caching — cached prefixes prefill far faster; TTFT can drop 40–60%.
• Smaller / distilled model — gpt-4o-mini vs gpt-4o: 2–4× faster TPOT. Use task-specific routing.
• Speculative decoding / draft models — provider-side; available on some hosted models. Free TPOT win when enabled.
• Provisioned throughput (Azure PTU, Anthropic dedicated) — eliminates queueing variance for steady workloads.
• Region pinning — same continent as your app server cuts 50–200 ms of network alone.

Measuring TTFT in Streaming Code

const t1 = performance.now();
const stream = await client.chat.completions.create({ model, messages, stream: true });

let ttft = 0, tokens = 0, t2 = 0;
for await (const chunk of stream) {
  if (!t2 && chunk.choices[0]?.delta?.content) {
    t2 = performance.now();
    ttft = t2 - t1;
  }
  tokens += 1;
}
const t3 = performance.now();
metrics.histogram('llm.ttft.ms', ttft, { model });
metrics.histogram('llm.tpot.ms', (t3 - t2) / tokens, { model });

Failure Modes

• Conflating E2E with model latency — your "LLM is slow" alert fires because retrieval got slow.
• Aggregating across models — gpt-4o and gpt-4o-mini in one chart hides everything useful.
• No TTFT metric at all — streaming UX optimisations are invisible.
• Sampling out the tail — if you only capture 1% you miss p99. Use tail-based sampling that prioritises slow requests.

Production Checklist

• Four metrics, not one: TTFT, TPOT, total generation, E2E.
• p50/p95/p99 per metric per model per endpoint.
• SLOs phrased in TTFT and TPOT, not just total — they map to UX.
• Tail-biased sampling so the slow requests are over-represented in your traces.

Key Takeaways

• TTFT is the perceived-latency king for streaming apps; TPOT keeps the perception going.
• The right intervention depends on which of the four latencies is the bottleneck — diagnose before you optimise.
• Prompt caching + smaller-model routing + region pinning are the three highest-ROI levers for most teams.