vLLM PagedAttention Diagram
Azure OpenAI + AI Search + embeddings — real-world architecture for production AI systems, including legacy data, orchestration, hybrid retrieval, cost control, and failure modes.
A served LLM needs continuous batching, KV-cache management, paged attention, and quantization to hit production QPS. Inference servers like vLLM and TGI handle this — exposing OpenAI-compatible APIs while squeezing 5-10x throughput from each GPU.
The pipeline
Each stage in order — click any step to read what it does.
Interactive Pipeline Flow
Quantize the model (AWQ, GPTQ, FP8) to fit larger context or batch onto the same GPU.
Architecture diagrams
vLLM-style server: API layer, scheduler, paged KV cache, GPU model.
Key design decisions
The trade-offs worth knowing before you build this.
vs static batching, gives 3-10x throughput because slow requests don't block the batch. Standard in vLLM, TGI.
Treats KV cache like virtual memory — fixed pages, no fragmentation. Cuts GPU memory waste from ~60% to <5%.
AWQ-INT4 or FP8 gives ~2x throughput with <1% quality loss on most benchmarks. Always benchmark on your goldens.
For 70B+ models, split across 2-8 GPUs (TP). Communication overhead is real — TP=4 is usually the sweet spot.
Long-prompt + short-output and short-prompt + long-output behave differently. Bench with your actual traffic mix.
Prefix sharing across requests (same system prompt) reuses KV pages. Order requests by prefix to maximize sharing.
“What’s our parental leave policy in Germany?”
The same request as in Read, with the values it has at each stage. Step through it, or click any stage. Values are illustrative for this scenario.
Gateway
Auth, then a per-user rate limit and daily budget check. This user has spent $0.40 of $2.
Cache
A semantic cache lookup finds “parental leave Germany” asked 31 times this week, similarity 0.97 against a threshold of 0.95
Freshness
The HR policy index changed yesterday, so the cached answer is stale and is skipped
Route
A cheap classifier rates this a simple factual lookup, so it goes to the small model with retrieval rather than the large one
Call
Claude Haiku 5.5 answers from 3 retrieved chunks in 1.1 s. A 20 s timeout and a fallback to a second provider are configured.
Record
Tokens, cost, latency, route and cache result go to the trace. The answer is cached against today’s policy version.
The same request, failing at “Call”
The primary provider returns 529 overloaded errors for twenty minutes. Without a fallback, every question fails. With one, the gateway retries once, then sends traffic to a second provider’s comparable model, and the answers carry on with a small quality difference nobody notices.
flowchart LR
U["User"] --> G["Gateway: auth, limits, budget"]
G --> C{"Fresh cache hit?"}
C -->|"yes"| A["Cached answer"]
C -->|"no"| R{"Route by difficulty"}
R -->|"simple"| S["Small model"]
R -->|"hard"| L["Large model"]
S --> F{"Provider OK?"}
L --> F
F -->|"no"| B["Fallback provider"]
F -->|"yes"| O["Answer, then trace and cache"]
B --> OContinue reading
All AI Engineering articles →Guardrails: Input/Output Safety in Production
Input, output, schema, tool, and cost guardrails — the layered defence system that catches LLM failures before they reach a user or downstream system.
Embeddings: From Theory to Production Choice
Convert text, images, and code into dense vectors that capture semantic meaning — the foundational primitive of modern AI systems, and the choice that dominates RAG quality.
Token Economics: Understanding and Optimizing LLM Costs
A practical guide to understanding token pricing, measuring real costs, and implementing optimization strategies — caching, prompt compression, model routing.
Discussion
Sign in to share your feedback and join the discussion.