AI EngineeringAdvanced

Continuous Batching Walkthrough

Azure OpenAI + AI Search + embeddings — real-world architecture for production AI systems, including legacy data, orchestration, hybrid retrieval, cost control, and failure modes.

By Amit UpadhyayPublished 18 FebUpdated 30 Aug 202615 min

Following one scenario: “What’s our parental leave policy in Germany?”An internal knowledge assistant, moving from a 20-person pilot to 2,000 employeesWhy it matters →

Step 1 / 5

Speed

What's happening

Boot

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

●serve.py

BootL1–8

1# 1. Launch vLLM with quantization + tensor parallel

2# vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \

3# --quantization fp8 \

4# --max-model-len 8192 \

5# --gpu-memory-utilization 0.92 \

6# --enable-prefix-caching \

7# --tensor-parallel-size 1 \

8# --port 8000

10# 2. Client uses OpenAI-compatible API

11from openai import OpenAI

12client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")

14# 3. Streaming chat

15stream = client.chat.completions.create(

16 model="meta-llama/Meta-Llama-3.1-8B-Instruct",

17 messages=[{"role":"user","content":"Explain HNSW in 3 sentences."}],

18 stream=True,

19 max_tokens=200,

20)

22# 4. Continuous batching: many concurrent requests share the GPU

23import asyncio, time

24async def fire(i):

25 t0 = time.time()

26 r = client.chat.completions.create(

27 model="meta-llama/Meta-Llama-3.1-8B-Instruct",

28 messages=[{"role":"user","content":f"Q{i}: write a haiku"}],

29 max_tokens=64,

30 )

31 print(f"req{i}: {time.time()-t0:.2f}s, {r.usage.completion_tokens} tok")

33asyncio.run(asyncio.gather(*[fire(i) for i in range(32)]))

35# 5. Metrics endpoint

36# curl http://localhost:8000/metrics | grep vllm_

37# vllm:num_requests_running 12

38# vllm:gpu_cache_usage_perc 0.78

39# vllm:time_to_first_token_seconds_bucket{le="0.5"} 0.94

39 lines

Steps · 5

What's happening

Boot

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

Next: Test →

Azure-OpenAI AI-Search production RAG architecture

← PreviousToken Economics: Understanding and Optimizing LLM Costs Next →Embeddings: From Theory to Production Choice

Continue reading

All AI Engineering articles →

AI Engineering13 min

Guardrails: Input/Output Safety in Production

Input, output, schema, tool, and cost guardrails — the layered defence system that catches LLM failures before they reach a user or downstream system.

AI Engineering11 min

Embeddings: From Theory to Production Choice

Convert text, images, and code into dense vectors that capture semantic meaning — the foundational primitive of modern AI systems, and the choice that dominates RAG quality.

AI Engineering10 min

Token Economics: Understanding and Optimizing LLM Costs

A practical guide to understanding token pricing, measuring real costs, and implementing optimization strategies — caching, prompt compression, model routing.

Discussion

AI EngineeringAdvanced

Continuous Batching Walkthrough

Azure OpenAI + AI Search + embeddings — real-world architecture for production AI systems, including legacy data, orchestration, hybrid retrieval, cost control, and failure modes.

By Amit UpadhyayPublished 18 FebUpdated 30 Aug 202615 min

Following one scenario: “What’s our parental leave policy in Germany?”An internal knowledge assistant, moving from a 20-person pilot to 2,000 employeesWhy it matters →

Step 1 / 5

Speed

What's happening

Boot

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

●serve.py

BootL1–8

1# 1. Launch vLLM with quantization + tensor parallel

2# vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \

3# --quantization fp8 \

4# --max-model-len 8192 \

5# --gpu-memory-utilization 0.92 \

6# --enable-prefix-caching \

7# --tensor-parallel-size 1 \

8# --port 8000

10# 2. Client uses OpenAI-compatible API

11from openai import OpenAI

12client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")

14# 3. Streaming chat

15stream = client.chat.completions.create(

16 model="meta-llama/Meta-Llama-3.1-8B-Instruct",

17 messages=[{"role":"user","content":"Explain HNSW in 3 sentences."}],

18 stream=True,

19 max_tokens=200,

20)

22# 4. Continuous batching: many concurrent requests share the GPU

23import asyncio, time

24async def fire(i):

25 t0 = time.time()

26 r = client.chat.completions.create(

27 model="meta-llama/Meta-Llama-3.1-8B-Instruct",

28 messages=[{"role":"user","content":f"Q{i}: write a haiku"}],

29 max_tokens=64,

30 )

31 print(f"req{i}: {time.time()-t0:.2f}s, {r.usage.completion_tokens} tok")

33asyncio.run(asyncio.gather(*[fire(i) for i in range(32)]))

35# 5. Metrics endpoint

36# curl http://localhost:8000/metrics | grep vllm_

37# vllm:num_requests_running 12

38# vllm:gpu_cache_usage_perc 0.78

39# vllm:time_to_first_token_seconds_bucket{le="0.5"} 0.94

39 lines

Steps · 5

What's happening

Boot

Launch with FP8 quant, prefix cache, large gpu mem

💡 Analogy

Like configuring a kitchen: prep stations (quant), pantry (memory), and shared workbench (prefix cache).

fp8 halves memory; prefix-caching reuses KV for repeated system prompts; gpu-memory 0.92 maximizes batch.

Console Output

> Loading FP8 weights... done in 12s
> KV cache: 24 GB / 26 GB

Next: Test →

Azure-OpenAI AI-Search production RAG architecture

← PreviousToken Economics: Understanding and Optimizing LLM Costs Next →Embeddings: From Theory to Production Choice

Continue reading

All AI Engineering articles →

AI Engineering13 min