Self-Hosted Models Animate

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Self-Hosted LLMs

📋vLLM

⚙️OpenAI API

🔄Batching

✅Economics

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

●self_hosted.py

vLLM ServerL1–6

1# vLLM: high-throughput LLM serving (OpenAI-compatible)

2# pip install vllm

3# python -m vllm.entrypoints.openai.api_server \

4# --model meta-llama/Llama-3.1-8B-Instruct \

5# --tensor-parallel-size 2 # split across 2 GPUs

7from openai import OpenAI

9# vLLM exposes OpenAI-compatible API — same client!

10client = OpenAI(

11 base_url="http://localhost:8000/v1",

12 api_key="none", # placeholder (vLLM doesn't authenticate by default)

13)

15response = client.chat.completions.create(

16 model="meta-llama/Llama-3.1-8B-Instruct",

17 messages=[{"role": "user", "content": "Explain Docker networking"}],

18 max_tokens=512,

19 temperature=0.7,

20)

21print(response.choices[0].message.content)

23# vLLM performance vs naive inference:

24# - Continuous batching: 5-10× higher throughput

25# - PagedAttention: 90% memory reduction for KV cache

26# - Tensor parallelism: multi-GPU scaling

28# Kubernetes deployment: vLLM on 2× A100 80GB

29# yaml snippet:

30# resources:

31# limits:

32# nvidia.com/gpu: 2

33# args:

34# - --tensor-parallel-size=2

35# - --model=meta-llama/Llama-3.1-70B-Instruct

36# Expected: 350 tok/s throughput at 50 concurrent users

36 lines

Steps · 4

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

Was this page helpful?

Discussion

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Self-Hosted LLMs

📋vLLM

⚙️OpenAI API

🔄Batching

✅Economics

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅

●self_hosted.py

vLLM ServerL1–6

1# vLLM: high-throughput LLM serving (OpenAI-compatible)

2# pip install vllm

3# python -m vllm.entrypoints.openai.api_server \

4# --model meta-llama/Llama-3.1-8B-Instruct \

5# --tensor-parallel-size 2 # split across 2 GPUs

7from openai import OpenAI

9# vLLM exposes OpenAI-compatible API — same client!

10client = OpenAI(

11 base_url="http://localhost:8000/v1",

12 api_key="none", # placeholder (vLLM doesn't authenticate by default)

13)

15response = client.chat.completions.create(

16 model="meta-llama/Llama-3.1-8B-Instruct",

17 messages=[{"role": "user", "content": "Explain Docker networking"}],

18 max_tokens=512,

19 temperature=0.7,

20)

21print(response.choices[0].message.content)

23# vLLM performance vs naive inference:

24# - Continuous batching: 5-10× higher throughput

25# - PagedAttention: 90% memory reduction for KV cache

26# - Tensor parallelism: multi-GPU scaling

28# Kubernetes deployment: vLLM on 2× A100 80GB

29# yaml snippet:

30# resources:

31# limits:

32# nvidia.com/gpu: 2

33# args:

34# - --tensor-parallel-size=2

35# - --model=meta-llama/Llama-3.1-70B-Instruct

36# Expected: 350 tok/s throughput at 50 concurrent users

36 lines

Steps · 4

What's happening

vLLM Server

vLLM serves LLMs with continuous batching — much higher throughput than naive

💡 Analogy

Like a ride-sharing dispatcher vs one driver — batching multiple requests dramatically improves efficiency.

vLLM is a high-performance LLM inference framework. It exposes an OpenAI-compatible REST API for any open-source model. 5-10× better throughput than naive serving.

Console Output

> vLLM server: Llama-3.1-8B-Instruct
> tensor-parallel-size: 2 (2× GPUs)
> API: http://localhost:8000/v1
> OpenAI-compatible endpoint ✅