LLM Model LandscapeIntermediate

On-Device Inference Pipeline

What fits in device memory, why bandwidth sets generation speed, which runtime to use per platform, and how to ship quantised weights with a cloud fallback that keeps the privacy promise.

By Amit UpadhyayPublished 09 Oct 20268 min

Following one scenario: “Classify this note offline: “gearbox oil temp high again, topped up, recheck Thursday”.”Field engineers’ tablets at remote wind farms, logging maintenance notesWhy it matters →

Step 1 / 4

Speed

What's happening

SLM Landscape

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-4-mini 3.8B: ~2.5GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

●edge_models.py

SLM LandscapeL1–5

1# Run Phi-4-mini (3.8B) locally with Ollama

2# ollama pull phi4-mini

4import ollama

5import json

7# Phi-4-mini: 3.8B params, runs on a laptop (Apple Silicon or CPU)

8def local_inference(prompt: str) -> str:

9 response = ollama.chat(

10 model="phi4-mini",

11 messages=[

12 {"role": "user", "content": prompt}

13 ],

14 options={

15 "temperature": 0.1, # deterministic for structured tasks

16 "num_ctx": 4096, # context window size

17 }

18 )

19 return response["message"]["content"]

21# Structured extraction with small model

22def extract_json(text: str) -> dict:

23 prompt = f"""Extract the following from this text and return ONLY JSON:

24{{

25 "person_name": "...",

26 "date": "YYYY-MM-DD or null",

27 "action": "..."

28}}

30Text: {text}"""

31 result = local_inference(prompt)

32 return json.loads(result)

34# Batch processing: 1000 documents, no API cost

35results = []

36documents = ["Alice met Bob on Jan 5, 2025...",

37 "Carol scheduled meeting for Feb 3..."]

38for doc in documents:

39 extracted = extract_json(doc)

40 results.append(extracted)

42# Speed: Phi-4-mini on Apple Silicon ≈ 40-60 tok/s

43# Cost: $0 (fully local)

44# Privacy: data never leaves your device

44 lines

Steps · 4

What's happening

SLM Landscape

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-4-mini 3.8B: ~2.5GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

Next: Test →

edge AI on-device SLM quantization llama.cpp WebLLM

← PreviousSelf-Hosting LLMs: Running a Model as a Service Next →Hallucination Testing: Measuring What the Model Makes Up

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape11 min

Llama: Open-Weight Models for Production

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

Discussion

LLM Model LandscapeIntermediate

On-Device Inference Pipeline

What fits in device memory, why bandwidth sets generation speed, which runtime to use per platform, and how to ship quantised weights with a cloud fallback that keeps the privacy promise.

By Amit UpadhyayPublished 09 Oct 20268 min

Step 1 / 4

Speed

What's happening

SLM Landscape

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-4-mini 3.8B: ~2.5GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

●edge_models.py

SLM LandscapeL1–5

1# Run Phi-4-mini (3.8B) locally with Ollama

2# ollama pull phi4-mini

4import ollama

5import json

7# Phi-4-mini: 3.8B params, runs on a laptop (Apple Silicon or CPU)

8def local_inference(prompt: str) -> str:

9 response = ollama.chat(

10 model="phi4-mini",

11 messages=[

12 {"role": "user", "content": prompt}

13 ],

14 options={

15 "temperature": 0.1, # deterministic for structured tasks

16 "num_ctx": 4096, # context window size

17 }

18 )

19 return response["message"]["content"]

21# Structured extraction with small model

22def extract_json(text: str) -> dict:

23 prompt = f"""Extract the following from this text and return ONLY JSON:

24{{

25 "person_name": "...",

26 "date": "YYYY-MM-DD or null",

27 "action": "..."

28}}

30Text: {text}"""

31 result = local_inference(prompt)

32 return json.loads(result)

34# Batch processing: 1000 documents, no API cost

35results = []

36documents = ["Alice met Bob on Jan 5, 2025...",

37 "Carol scheduled meeting for Feb 3..."]

38for doc in documents:

39 extracted = extract_json(doc)

40 results.append(extracted)

42# Speed: Phi-4-mini on Apple Silicon ≈ 40-60 tok/s

43# Cost: $0 (fully local)

44# Privacy: data never leaves your device

44 lines

Steps · 4

What's happening

SLM Landscape

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-4-mini 3.8B: ~2.5GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

Next: Test →

edge AI on-device SLM quantization llama.cpp WebLLM

← PreviousSelf-Hosting LLMs: Running a Model as a Service Next →Hallucination Testing: Measuring What the Model Makes Up

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min