LLM Model LandscapeAdvanced

Deploy an Inference Server with vLLM

Break-even maths, GPU sizing from weights and KV cache, choosing between vLLM, SGLang and llama.cpp, and what to scale and alert on once open-weight models carry production traffic.

By Amit UpadhyayPublished 06 Oct 20269 min

Following one scenario: “Should we self-host an open model instead of paying per token?”An internal assistant used by 3,000 employees, currently on a hosted small modelWhy it matters →

What you will build

Serve an open model with vLLM, measure its capacity in tokens per second, and produce an honest break-even against hosting.

The same scenario as Read and See. You are done when every line below is true. Tick them off as you go.

Done when

vLLM serves an OpenAI-compatible endpoint
A load test reports output tokens per second at the latency target
A break-even sheet includes GPUs and operations time
A gateway route falls back to a hosted model for allowed traffic

0 of 4 done

0/1 steps

Build Steps

Progress

0/1

◆ Before you start

Prerequisites

Python 3.11+Required
python --version
vllm or ollamaRequired
pip install vllm
GPU (optional)Optional
NVIDIA with 16GB+ VRAM for 7B models

Step 1 of 1

Deploy a self-hosted LLM with vLLM

Deploy an open-source LLM with vLLM for high-throughput serving, expose OpenAI-compatible API.

Tasks

Install vLLM and launch a server with a small model

Call the server via OpenAI Python client

Benchmark throughput (req/s) vs latency under load

Compare self-hosted vs API cost at scale

Add Prometheus metrics scraping to vLLM

🎯 What You'll Learn

▸Launch vLLM server with an open-source model
▸Configure continuous batching for high throughput
▸Call self-hosted model via OpenAI SDK

Code

Bashvllm_server.sh
1 #!/bin/bash
2 # Install vLLM
3 pip install vllm
4  
5 # Launch vLLM server (OpenAI-compatible API)
6 # Replace model with any HuggingFace model ID
7 python -m vllm.entrypoints.openai.api_server \
8     --model meta-llama/Llama-3.2-3B-Instruct \
9     --host 0.0.0.0 \
10     --port 8000 \
11     --max-model-len 8192 \
12     --dtype auto \
13     --api-key "your-secret-key"
14  
15 # For CPU-only (Ollama alternative):
16 # ollama serve  (runs on port 11434)
17

💡 Pro Tips

▸vLLM uses PagedAttention for 3-5x better GPU memory utilization vs naive serving
▸For CPU-only environments: use Ollama with Q4_K_M quantized models

Evaluate it

Prove it works before you call it done

Working once on your machine is a demo. Measure it against a fixed set of cases.

Load-test with realistic prompt and output lengths at increasing concurrency, then compute the break-even at current and projected volume.

Pass bar

Known capacity at the latency target, and a written decision with its real reason

Ship and observe

What changes when it is live

Before it ships, in order

Scale down off-peak where possible
Alert on queue time and tokens per second

Put on a dashboard

GPU utilisation
Tokens per second
Queue time

Break it on purpose

Learn the failure modes while they are cheap

Make one change at a time, predict what will happen, then run it.

ChangeWhat you should see

Size capacity by request count with short test promptsReal traffic with long answers queues at peak.

Leave GPUs running at full size overnightThe bill stays high with almost no traffic.

Remove the hosted fallbackMaintenance takes every feature down.

What this build costs

Load testing on rented GPUs costs a few dollars an hour.

Where this leads

Model serving

Self-hosted or not, the serving layer (routing, caching, budgets and fallbacks) is what makes it production-ready.

Next topic →

self-hosting vLLM inference GPU KV cache Kubernetes

← PreviousGemini: Google's Multimodal AI Next →Edge Models: Running Small LLMs on Phones, Laptops and Browsers

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape11 min

Llama: Open-Weight Models for Production

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

Discussion

Serve an open model with vLLM, measure its capacity in tokens per second, and produce an honest break-even against hosting.

The same scenario as Read and See. You are done when every line below is true. Tick them off as you go.

Done when

vLLM serves an OpenAI-compatible endpoint
A load test reports output tokens per second at the latency target
A break-even sheet includes GPUs and operations time
A gateway route falls back to a hosted model for allowed traffic

0 of 4 done

1 #!/bin/bash 2 # Install vLLM 3 pip install vllm 4 5 # Launch vLLM server (OpenAI-compatible API) 6 # Replace model with any HuggingFace model ID 7 python -m vllm.entrypoints.openai.api_server \ 8 --model meta-llama/Llama-3.2-3B-Instruct \ 9 --host 0.0.0.0 \ 10 --port 8000 \ 11 --max-model-len 8192 \ 12 --dtype auto \ 13 --api-key "your-secret-key" 14 15 # For CPU-only (Ollama alternative): 16 # ollama serve (runs on port 11434) 17

Prove it works before you call it done

Working once on your machine is a demo. Measure it against a fixed set of cases.

Load-test with realistic prompt and output lengths at increasing concurrency, then compute the break-even at current and projected volume.

Pass bar

Known capacity at the latency target, and a written decision with its real reason

Learn the failure modes while they are cheap

Make one change at a time, predict what will happen, then run it.

ChangeWhat you should see

Size capacity by request count with short test promptsReal traffic with long answers queues at peak.

Leave GPUs running at full size overnightThe bill stays high with almost no traffic.

Remove the hosted fallbackMaintenance takes every feature down.

1	#!/bin/bash
2	# Install vLLM
3	pip install vllm
4
5	# Launch vLLM server (OpenAI-compatible API)
6	# Replace model with any HuggingFace model ID
7	python -m vllm.entrypoints.openai.api_server \
8	--model meta-llama/Llama-3.2-3B-Instruct \
9	--host 0.0.0.0 \
10	--port 8000 \
11	--max-model-len 8192 \
12	--dtype auto \
13	--api-key "your-secret-key"
14
15	# For CPU-only (Ollama alternative):
16	# ollama serve (runs on port 11434)
17