vLLM, TGI, Triton: turning model checkpoints into low-latency APIs.

Model Serving — From Weights to QPS

A served LLM needs continuous batching, KV-cache management, paged attention, and quantization to hit production QPS. Inference servers like vLLM and TGI handle this — exposing OpenAI-compatible APIs while squeezing 5-10x throughput from each GPU.

Interactive Pipeline Flow

IngestProcessServe

🗜️ Quantize — AWQ / GPTQ

Quantize the model (AWQ, GPTQ, FP8) to fit larger context or batch onto the same GPU.

Architecture Diagrams

Build → deploy pipeline for a quantized model on Kubernetes.

Loading diagram...

Key Design Decisions

🚀Continuous Batching

vs static batching, gives 3-10x throughput because slow requests don't block the batch. Standard in vLLM, TGI.

📦PagedAttention

Treats KV cache like virtual memory — fixed pages, no fragmentation. Cuts GPU memory waste from ~60% to <5%.

🗜️Quantize Aggressively

AWQ-INT4 or FP8 gives ~2x throughput with <1% quality loss on most benchmarks. Always benchmark on your goldens.

🔀Tensor Parallel

For 70B+ models, split across 2-8 GPUs (TP). Communication overhead is real — TP=4 is usually the sweet spot.

⚖️Bench Realistic Loads

Long-prompt + short-output and short-prompt + long-output behave differently. Bench with your actual traffic mix.

🧹Watch KV Hit Rate

Prefix sharing across requests (same system prompt) reuses KV pages. Order requests by prefix to maximize sharing.

Discussion