Build Steps

Progress

0/1

◆ Before you start

Prerequisites

Python 3.11+Required
python --version
vllm or ollamaRequired
pip install vllm
GPU (optional)Optional
NVIDIA with 16GB+ VRAM for 7B models

🖥️

Step 1 of 1

Deploy a self-hosted LLM with vLLM

Deploy an open-source LLM with vLLM for high-throughput serving, expose OpenAI-compatible API.

Tasks

Install vLLM and launch a server with a small model

Call the server via OpenAI Python client

Benchmark throughput (req/s) vs latency under load

Compare self-hosted vs API cost at scale

Add Prometheus metrics scraping to vLLM

🎯 What You'll Learn

▸Launch vLLM server with an open-source model
▸Configure continuous batching for high throughput
▸Call self-hosted model via OpenAI SDK

Code

Bashvllm_server.sh
1 #!/bin/bash
2 # Install vLLM
3 pip install vllm
4  
5 # Launch vLLM server (OpenAI-compatible API)
6 # Replace model with any HuggingFace model ID
7 python -m vllm.entrypoints.openai.api_server \
8     --model meta-llama/Llama-3.2-3B-Instruct \
9     --host 0.0.0.0 \
10     --port 8000 \
11     --max-model-len 8192 \
12     --dtype auto \
13     --api-key "your-secret-key"
14  
15 # For CPU-only (Ollama alternative):
16 # ollama serve  (runs on port 11434)
17

💡 Pro Tips

▸vLLM uses PagedAttention for 3-5x better GPU memory utilization vs naive serving
▸For CPU-only environments: use Ollama with Q4_K_M quantized models

Build Steps

Progress

0/1

◆ Before you start

Prerequisites

Python 3.11+Required
python --version
vllm or ollamaRequired
pip install vllm
GPU (optional)Optional
NVIDIA with 16GB+ VRAM for 7B models

🖥️

Step 1 of 1

Deploy a self-hosted LLM with vLLM

Deploy an open-source LLM with vLLM for high-throughput serving, expose OpenAI-compatible API.

Tasks

Install vLLM and launch a server with a small model

Call the server via OpenAI Python client

Benchmark throughput (req/s) vs latency under load

Compare self-hosted vs API cost at scale

Add Prometheus metrics scraping to vLLM

🎯 What You'll Learn

▸Launch vLLM server with an open-source model
▸Configure continuous batching for high throughput
▸Call self-hosted model via OpenAI SDK

Code

Bashvllm_server.sh
1 #!/bin/bash
2 # Install vLLM
3 pip install vllm
4  
5 # Launch vLLM server (OpenAI-compatible API)
6 # Replace model with any HuggingFace model ID
7 python -m vllm.entrypoints.openai.api_server \
8     --model meta-llama/Llama-3.2-3B-Instruct \
9     --host 0.0.0.0 \
10     --port 8000 \
11     --max-model-len 8192 \
12     --dtype auto \
13     --api-key "your-secret-key"
14  
15 # For CPU-only (Ollama alternative):
16 # ollama serve  (runs on port 11434)
17

💡 Pro Tips

▸vLLM uses PagedAttention for 3-5x better GPU memory utilization vs naive serving
▸For CPU-only environments: use Ollama with Q4_K_M quantized models

Build: Deploy a self-hosted LLM with vLLM

Prerequisites

Deploy a self-hosted LLM with vLLM

Tasks

🎯 What You'll Learn

Code

💡 Pro Tips

Discussion

Build: Deploy a self-hosted LLM with vLLM

Prerequisites

Deploy a self-hosted LLM with vLLM

Tasks

🎯 What You'll Learn

Code

💡 Pro Tips

Discussion

1	#!/bin/bash
2	# Install vLLM
3	pip install vllm
4
5	# Launch vLLM server (OpenAI-compatible API)
6	# Replace model with any HuggingFace model ID
7	python -m vllm.entrypoints.openai.api_server \
8	--model meta-llama/Llama-3.2-3B-Instruct \
9	--host 0.0.0.0 \
10	--port 8000 \
11	--max-model-len 8192 \
12	--dtype auto \
13	--api-key "your-secret-key"
14
15	# For CPU-only (Ollama alternative):
16	# ollama serve (runs on port 11434)
17