SLMs (Phi-4, Gemma 3), WebLLM, Apple Neural Engine, privacy.

Edge Models — On-device AI with Small Language Models

Edge models run entirely on-device (phone, laptop, browser) without cloud API calls. Benefits: privacy (no data leaves device), low latency (no network round-trip), offline capability, zero server cost. Small Language Models (SLMs): Phi-4-mini (3.8B), Gemma 3 (1B/4B), Llama 3.2 (1B/3B), Qwen2.5 (0.5B-7B). WebLLM runs SLMs in the browser via WebGPU. Apple Intelligence uses on-device models for privacy-preserving iPhone/Mac features.

Interactive Pipeline Flow

UnderstandImplementOptimise

📱 SLM Select — size vs quality

SLM selection trade-off: quality vs size. Phi-4-mini (3.8B) = near GPT-3.5 quality at 2GB. Gemma 3 1B = minimal quality, very fast. Llama 3.2 3B = good balance for chat tasks.

Architecture Diagrams

Routing requests between on-device model and cloud API.

Loading diagram...

Key Design Decisions

🔒Edge models are the only true privacy solution

Even when using 'privacy-preserving' cloud APIs, data leaves the device. For genuinely sensitive data (medical records, legal documents, personal communications), on-device inference is the only option.

📱Phi-4-mini achieves near GPT-3.5 quality at edge size

Microsoft Phi-4-mini (3.8B parameters) achieves scores competitive with much larger models on reasoning and coding benchmarks. Quality no longer requires a 70B+ model.

🔀Hybrid routing: sensitive tasks on-device, complex tasks in cloud

The optimal strategy is neither pure edge nor pure cloud. Classify and extract locally (fast, private), then route only de-identified complex queries to cloud models for reasoning.

Discussion