Model Serving — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— vLLM Server
🚀Launch
📡Client
🌊Stream
⚡Batching
📊Metrics
🎯 Next: Test →
Step 1 / 5
Speed
Boot
Launch with FP8 quant, prefix cache, large gpu mem💡 Analogy
Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).
fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.
Console Output
> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB
●serve.py
BootL1–8
1# 1. Launch vLLM with quantization + tensor parallel
2# vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
3#   --quantization fp8 \
4#   --max-model-len 8192 \
5#   --gpu-memory-utilization 0.92 \
6#   --enable-prefix-caching \
7#   --tensor-parallel-size 1 \
8#   --port 8000
9 
10# 2. Client uses OpenAI-compatible API
11from openai import OpenAI
12client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
13 
14# 3. Streaming chat
15stream = client.chat.completions.create(
16    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
17    messages=[{"role":"user","content":"Explain HNSW in 3 sentences."}],
18    stream=True,
19    max_tokens=200,
20)
21 
22# 4. Continuous batching: many concurrent requests share the GPU
23import asyncio, time
24async def fire(i):
25    t0 = time.time()
26    r = client.chat.completions.create(
27        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
28        messages=[{"role":"user","content":f"Q{i}: write a haiku"}],
29        max_tokens=64,
30    )
31    print(f"req{i}: {time.time()-t0:.2f}s, {r.usage.completion_tokens} tok")
32 
33asyncio.run(asyncio.gather(*[fire(i) for i in range(32)]))
34 
35# 5. Metrics endpoint
36# curl http://localhost:8000/metrics | grep vllm_
37#   vllm:num_requests_running 12
38#   vllm:gpu_cache_usage_perc 0.78
39#   vllm:time_to_first_token_seconds_bucket{le="0.5"} 0.94
39 lines
Timeline — 5 Steps
Boot
Launch with FP8 quant, prefix cache, large gpu mem💡 Analogy
Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).
fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.
Console Output
> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

Was this page helpful?

Discussion

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB