Self-hosting LLMs eliminates API costs and provides data privacy. Key frameworks: Ollama (simplest, good for dev), vLLM (production inference server, continuous batching), llama.cpp (CPU/GPU, quantized models). Hardware requirements: GPU VRAM is the bottleneck. 7B models need 8GB VRAM, 70B models need 40GB+. Quantization (GGUF Q4/Q8) reduces VRAM at slight quality cost. vLLM's PagedAttention enables much higher throughput than naive serving.
Each stage in order — click any step to read what it does.
Framework comparison for self-hosted model serving.
The trade-offs worth knowing before you build this.
Q4_K_M quantization (GGUF format) reduces model size by 75% with ~3-5% quality degradation on most tasks. For a 7B model: 14GB FP16 → 4GB Q4_K_M. Fits on consumer GPUs.
vLLM's PagedAttention manages KV cache memory like virtual memory pages, eliminating fragmentation. Result: 10x more concurrent requests per GPU vs Hugging Face naive inference.
Set OPENAI_API_BASE=http://localhost:8000/v1 and OPENAI_API_KEY=any. Your existing OpenAI SDK code runs against your self-hosted model with no changes.
Sign in to share your feedback and join the discussion.
