Inference Server Architecture
Break-even maths, GPU sizing from weights and KV cache, choosing between vLLM, SGLang and llama.cpp, and what to scale and alert on once open-weight models carry production traffic.
Self-hosting LLMs eliminates API costs and provides data privacy. Key frameworks: Ollama (simplest, good for dev), vLLM (production inference server, continuous batching), llama.cpp (CPU/GPU, quantized models). Hardware requirements: GPU VRAM is the bottleneck. 7B models need 8GB VRAM, 70B models need 40GB+. Quantization (GGUF Q4/Q8) reduces VRAM at slight quality cost. vLLM's PagedAttention enables much higher throughput than naive serving.
The pipeline
Each stage in order — click any step to read what it does.
Interactive Pipeline Flow
Architecture diagrams
Framework comparison for self-hosted model serving.
Key design decisions
The trade-offs worth knowing before you build this.
Q4_K_M quantization (GGUF format) reduces model size by 75% with ~3-5% quality degradation on most tasks. For a 7B model: 14GB FP16 → 4GB Q4_K_M. Fits on consumer GPUs.
vLLM's PagedAttention manages KV cache memory like virtual memory pages, eliminating fragmentation. Result: 10x more concurrent requests per GPU vs Hugging Face naive inference.
Set OPENAI_API_BASE=http://localhost:8000/v1 and OPENAI_API_KEY=any. Your existing OpenAI SDK code runs against your self-hosted model with no changes.
“Should we self-host an open model instead of paying per token?”
The same request as in Read, with the values it has at each stage. Step through it, or click any stage. Values are illustrative for this scenario.
Why
The real drivers: a new customer contract requires data to stay in the company’s cloud account, and the team wants to fine-tune for internal jargon
Break-even
Hosted cost at current volume: under $100 a month. Self-hosted: about $4,000 a month for two GPUs, plus on-call. Cost is not the reason; residency is.
Serve
An open model served by vLLM with continuous batching, behind the internal gateway, using an OpenAI-compatible API
Capacity
Load tests show about 2,500 output tokens a second per GPU at the target latency; two GPUs cover peak with headroom
Fallback
For requests without data-residency constraints, the gateway can fall back to a hosted model during maintenance
The same request, failing at “Capacity”
The first plan sized capacity by requests per minute. Long answers took far more GPU time than short ones, and at peak the queue grew to minutes. Measure tokens per second at your latency target, with realistic prompt and output lengths.
Continue reading
All LLM Model Landscape articles →GPT: OpenAI's Model Family in Practice
The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.
Claude: Anthropic's Model Family in Practice
The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.
Llama: Open-Weight Models for Production
The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.
Discussion
Sign in to share your feedback and join the discussion.