LLM Model LandscapeAdvanced

Request Batching and KV-Cache

Break-even maths, GPU sizing from weights and KV cache, choosing between vLLM, SGLang and llama.cpp, and what to scale and alert on once open-weight models carry production traffic.

By Amit UpadhyayPublished 06 Oct 20269 min

Following one scenario: “Should we self-host an open model instead of paying per token?”An internal assistant used by 3,000 employees, currently on a hosted small modelWhy it matters →

Step 1 / 4

Speed

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

●self_hosted.py

vLLM ServerL1–6

1# vLLM: high-throughput LLM serving (OpenAI-compatible)

2# pip install vllm

3# python -m vllm.entrypoints.openai.api_server \

4# --model meta-llama/Llama-3.1-8B-Instruct \

5# --tensor-parallel-size 2 # split across 2 GPUs

7from openai import OpenAI

9# vLLM exposes OpenAI-compatible API — same client!

10client = OpenAI(

11 base_url="http://localhost:8000/v1",

12 api_key="none", # placeholder (vLLM doesn't authenticate by default)

13)

15response = client.chat.completions.create(

16 model="meta-llama/Llama-3.1-8B-Instruct",

17 messages=[{"role": "user", "content": "Explain Docker networking"}],

18 max_tokens=512,

19 temperature=0.7,

20)

21print(response.choices[0].message.content)

23# vLLM performance vs naive inference:

24# - Continuous batching: 5-10× higher throughput

25# - PagedAttention: 90% memory reduction for KV cache

26# - Tensor parallelism: multi-GPU scaling

28# Kubernetes deployment: vLLM on 2× A100 80GB

29# yaml snippet:

30# resources:

31# limits:

32# nvidia.com/gpu: 2

33# args:

34# - --tensor-parallel-size=2

35# - --model=meta-llama/Llama-3.1-70B-Instruct

36# Expected: 350 tok/s throughput at 50 concurrent users

36 lines

Steps · 4

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

Next: Test →

self-hosting vLLM inference GPU KV cache Kubernetes

← PreviousGemini: Google's Multimodal AI Next →Edge Models: Running Small LLMs on Phones, Laptops and Browsers

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape11 min

Llama: Open-Weight Models for Production

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

Discussion

LLM Model LandscapeAdvanced

Request Batching and KV-Cache

Break-even maths, GPU sizing from weights and KV cache, choosing between vLLM, SGLang and llama.cpp, and what to scale and alert on once open-weight models carry production traffic.

By Amit UpadhyayPublished 06 Oct 20269 min

Following one scenario: “Should we self-host an open model instead of paying per token?”An internal assistant used by 3,000 employees, currently on a hosted small modelWhy it matters →

Step 1 / 4

Speed

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

●self_hosted.py

vLLM ServerL1–6

1# vLLM: high-throughput LLM serving (OpenAI-compatible)

2# pip install vllm

3# python -m vllm.entrypoints.openai.api_server \

4# --model meta-llama/Llama-3.1-8B-Instruct \

5# --tensor-parallel-size 2 # split across 2 GPUs

7from openai import OpenAI

9# vLLM exposes OpenAI-compatible API — same client!

10client = OpenAI(

11 base_url="http://localhost:8000/v1",

12 api_key="none", # placeholder (vLLM doesn't authenticate by default)

13)

15response = client.chat.completions.create(

16 model="meta-llama/Llama-3.1-8B-Instruct",

17 messages=[{"role": "user", "content": "Explain Docker networking"}],

18 max_tokens=512,

19 temperature=0.7,

20)

21print(response.choices[0].message.content)

23# vLLM performance vs naive inference:

24# - Continuous batching: 5-10× higher throughput

25# - PagedAttention: 90% memory reduction for KV cache

26# - Tensor parallelism: multi-GPU scaling

28# Kubernetes deployment: vLLM on 2× A100 80GB

29# yaml snippet:

30# resources:

31# limits:

32# nvidia.com/gpu: 2

33# args:

34# - --tensor-parallel-size=2

35# - --model=meta-llama/Llama-3.1-70B-Instruct

36# Expected: 350 tok/s throughput at 50 concurrent users

36 lines

Steps · 4

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

Next: Test →

self-hosting vLLM inference GPU KV cache Kubernetes

← PreviousGemini: Google's Multimodal AI Next →Edge Models: Running Small LLMs on Phones, Laptops and Browsers

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min