Small Model Size vs Quality
What fits in device memory, why bandwidth sets generation speed, which runtime to use per platform, and how to ship quantised weights with a cloud fallback that keeps the privacy promise.
Edge models run entirely on-device (phone, laptop, browser) without cloud API calls. Benefits: privacy (no data leaves device), low latency (no network round-trip), offline capability, zero server cost. Small Language Models (SLMs): Phi-4-mini (3.8B), Gemma 3 (1B/4B), Llama 3.2 (1B/3B), Qwen2.5 (0.5B-7B). WebLLM runs SLMs in the browser via WebGPU. Apple Intelligence uses on-device models for privacy-preserving iPhone/Mac features.
The pipeline
Each stage in order — click any step to read what it does.
Interactive Pipeline Flow
Architecture diagrams
Deployment options for edge/on-device AI.
Key design decisions
The trade-offs worth knowing before you build this.
Even when using 'privacy-preserving' cloud APIs, data leaves the device. For genuinely sensitive data (medical records, legal documents, personal communications), on-device inference is the only option.
Microsoft Phi-4-mini (3.8B parameters) achieves scores competitive with much larger models on reasoning and coding benchmarks. Quality no longer requires a 70B+ model.
The optimal strategy is neither pure edge nor pure cloud. Classify and extract locally (fast, private), then route only de-identified complex queries to cloud models for reasoning.
“Classify this note offline: “gearbox oil temp high again, topped up, recheck Thursday”.”
The same request as in Read, with the values it has at each stage. Step through it, or click any stage. Values are illustrative for this scenario.
Model
A 3B-parameter model quantised to 4 bits: about 1.5 GB of weights, around 2 GB of RAM in use
3 billion parameters × 0.5 bytes each, plus runtime overhead.
Prompt
A short instruction plus 4 examples covering the 6 note categories and an “urgent” flag
Classify
On-device result in about 1 second: category
mechanical, urgenttrue(a repeated overheating issue)Act
The tablet raises a local alert to the site lead straight away, with no network needed
Sync
When the tablet reconnects, notes the small model was unsure about are re-checked by a cloud model, and any disagreement is logged
The same request, failing at “Model”
The first choice was a 7B model at 8 bits, about 7 GB, on tablets with 8 GB of RAM. It crashed the app under memory pressure. Size the model to the device you actually have, and test on the oldest device in the fleet.
Continue reading
All LLM Model Landscape articles →GPT: OpenAI's Model Family in Practice
The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.
Claude: Anthropic's Model Family in Practice
The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.
Llama: Open-Weight Models for Production
The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.
Discussion
Sign in to share your feedback and join the discussion.