LLM Model LandscapeIntermediate

Run a Model in the Browser with WebLLM

What fits in device memory, why bandwidth sets generation speed, which runtime to use per platform, and how to ship quantised weights with a cloud fallback that keeps the privacy promise.

By Amit UpadhyayPublished 09 Oct 20268 min

Following one scenario: “Classify this note offline: “gearbox oil temp high again, topped up, recheck Thursday”.”Field engineers’ tablets at remote wind farms, logging maintenance notesWhy it matters →

What you will build

Run a quantised small model locally with Ollama to classify maintenance notes, with a cloud re-check path.

The same scenario as Read and See. You are done when every line below is true. Tick them off as you go.

Done when

The model runs within the memory budget of the target device
Classification takes about a second per note, offline
Uncertain notes are queued for a cloud re-check
Agreement with the cloud model is reported

0 of 4 done

0/1 steps

Build Steps

Progress

0/1

◆ Before you start

Prerequisites

Python 3.11+Required
python --version
ollamaRequired
Install from ollama.ai

Step 1 of 1

Run models locally with Ollama

Run Llama/Mistral locally with Ollama, benchmark performance, and integrate via OpenAI-compatible API.

Tasks

Install Ollama and pull llama3.2:3b

Run inference via CLI: ollama run llama3.2:3b

Call Ollama via OpenAI-compatible API on localhost:11434

Benchmark tokens/sec for 3b, 7b, and 13b models

Compare edge model vs API model on your eval prompts

🎯 What You'll Learn

▸Pull and run a model with Ollama CLI
▸Use Ollamas OpenAI-compatible API for easy migration
▸Benchmark tokens/second and memory usage

Code

Pythonollama_client.py
1 import time
2 import openai  # pip install openai
3  
4 # Ollama exposes OpenAI-compatible API on localhost:11434
5 # Run: ollama pull llama3.2:3b && ollama serve
6  
7 client = openai.OpenAI(
8     base_url="http://localhost:11434/v1",
9     api_key="ollama",  # Required but ignored
10 )
11  
12 def benchmark_model(model: str, prompt: str, n: int = 3) -> dict:
13     """Benchmark tokens/sec for a local model."""
14     latencies = []
15     token_counts = []
16     
17     for _ in range(n):
18         start = time.perf_counter()
19         response = client.chat.completions.create(
20             model=model,
21             messages=[{"role": "user", "content": prompt}],
22             max_completion_tokens=200,
23             stream=False,
24         )
25         latency = time.perf_counter() - start
26         latencies.append(latency)
27         token_counts.append(response.usage.completion_tokens)
28     
29     avg_latency = sum(latencies) / n
30     avg_tokens = sum(token_counts) / n
31     tokens_per_sec = avg_tokens / avg_latency
32     
33     return {
34         "model": model,
35         "avg_latency_s": round(avg_latency, 2),
36         "avg_output_tokens": round(avg_tokens),
37         "tokens_per_sec": round(tokens_per_sec, 1),
38     }
39  
40 # ── Streaming example ───────────────────────────────────────────────
41 def stream_local(model: str, prompt: str):
42     """Stream tokens from local model."""
43     print(f"\n[{model}] Streaming: ", end="")
44     start = time.perf_counter()
45     token_count = 0
46     
47     stream = client.chat.completions.create(
48         model=model,
49         messages=[{"role": "user", "content": prompt}],
50         max_completion_tokens=200,
51         stream=True,
52     )
53     
54     for chunk in stream:
55         if chunk.choices[0].delta.content:
56             print(chunk.choices[0].delta.content, end="", flush=True)
57             token_count += 1
58     
59     elapsed = time.perf_counter() - start
60     print(f"\n  Tokens/sec: {token_count/elapsed:.1f}")
61  
62 PROMPT = "Explain gradient descent in 3 sentences."
63  
64 # Benchmark available models
65 for model in ["llama3.2:3b"]:
66     try:
67         result = benchmark_model(model, PROMPT)
68         print(result)
69         stream_local(model, PROMPT)
70     except Exception as e:
71         print(f"{model}: {e} (model not pulled — run: ollama pull {model})")
72

💡 Pro Tips

▸Quantized models (Q4_K_M) are 4x smaller with < 5% quality loss for most tasks
▸Use ollama serve + OpenAI client for drop-in replacement of API calls in local dev

Evaluate it

Prove it works before you call it done

Working once on your machine is a demo. Measure it against a fixed set of cases.

Classify 100 labelled notes offline, then compare with a cloud model and the labels.

Pass bar

≥ 90% accuracy on device, with peak memory under the budget

Ship and observe

What changes when it is live

Before it ships, in order

Test on the oldest supported device
Roll out model updates in stages with the app

Put on a dashboard

Agreement with cloud
Memory and crash rate
Time to urgent alert

Break it on purpose

Learn the failure modes while they are cheap

Make one change at a time, predict what will happen, then run it.

ChangeWhat you should see

Swap in a 7B model at 8 bitsMemory use roughly triples, and low-memory devices struggle.

Remove the examples from the promptAccuracy drops on the rarer categories.

Skip the cloud re-checkMistakes on uncertain notes go unnoticed.

What this build costs

Free to run locally; cloud re-checks cost pennies.

Where this leads

Self-hosted models

From one device to your own servers: the same open models, served at scale.

Next topic →

edge AI on-device SLM quantization llama.cpp WebLLM

← PreviousSelf-Hosting LLMs: Running a Model as a Service Next →Hallucination Testing: Measuring What the Model Makes Up

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape11 min

Llama: Open-Weight Models for Production

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

Discussion

Run a quantised small model locally with Ollama to classify maintenance notes, with a cloud re-check path.

The same scenario as Read and See. You are done when every line below is true. Tick them off as you go.

Done when

The model runs within the memory budget of the target device
Classification takes about a second per note, offline
Uncertain notes are queued for a cloud re-check
Agreement with the cloud model is reported

0 of 4 done

1 import time 2 import openai # pip install openai 3 4 # Ollama exposes OpenAI-compatible API on localhost:11434 5 # Run: ollama pull llama3.2:3b && ollama serve 6 7 client = openai.OpenAI( 8 base_url="http://localhost:11434/v1", 9 api_key="ollama", # Required but ignored 10 ) 11 12 def benchmark_model(model: str, prompt: str, n: int = 3) -> dict: 13 """Benchmark tokens/sec for a local model.""" 14 latencies = [] 15 token_counts = [] 16 17 for _ in range(n): 18 start = time.perf_counter() 19 response = client.chat.completions.create( 20 model=model, 21 messages=[{"role": "user", "content": prompt}], 22 max_completion_tokens=200, 23 stream=False, 24 ) 25 latency = time.perf_counter() - start 26 latencies.append(latency) 27 token_counts.append(response.usage.completion_tokens) 28 29 avg_latency = sum(latencies) / n 30 avg_tokens = sum(token_counts) / n 31 tokens_per_sec = avg_tokens / avg_latency 32 33 return { 34 "model": model, 35 "avg_latency_s": round(avg_latency, 2), 36 "avg_output_tokens": round(avg_tokens), 37 "tokens_per_sec": round(tokens_per_sec, 1), 38 } 39 40 # ── Streaming example ─────────────────────────────────────────────── 41 def stream_local(model: str, prompt: str): 42 """Stream tokens from local model.""" 43 print(f"\n[{model}] Streaming: ", end="") 44 start = time.perf_counter() 45 token_count = 0 46 47 stream = client.chat.completions.create( 48 model=model, 49 messages=[{"role": "user", "content": prompt}], 50 max_completion_tokens=200, 51 stream=True, 52 ) 53 54 for chunk in stream: 55 if chunk.choices[0].delta.content: 56 print(chunk.choices[0].delta.content, end="", flush=True) 57 token_count += 1 58 59 elapsed = time.perf_counter() - start 60 print(f"\n Tokens/sec: {token_count/elapsed:.1f}") 61 62 PROMPT = "Explain gradient descent in 3 sentences." 63 64 # Benchmark available models 65 for model in ["llama3.2:3b"]: 66 try: 67 result = benchmark_model(model, PROMPT) 68 print(result) 69 stream_local(model, PROMPT) 70 except Exception as e: 71 print(f"{model}: {e} (model not pulled — run: ollama pull {model})") 72

Learn the failure modes while they are cheap

Make one change at a time, predict what will happen, then run it.

ChangeWhat you should see

Swap in a 7B model at 8 bitsMemory use roughly triples, and low-memory devices struggle.

Remove the examples from the promptAccuracy drops on the rarer categories.

Skip the cloud re-checkMistakes on uncertain notes go unnoticed.

1	import time
2	import openai # pip install openai
3
4	# Ollama exposes OpenAI-compatible API on localhost:11434
5	# Run: ollama pull llama3.2:3b && ollama serve
6
7	client = openai.OpenAI(
8	base_url="http://localhost:11434/v1",
9	api_key="ollama", # Required but ignored
10	)
11
12	def benchmark_model(model: str, prompt: str, n: int = 3) -> dict:
13	"""Benchmark tokens/sec for a local model."""
14	latencies = []
15	token_counts = []
16
17	for _ in range(n):
18	start = time.perf_counter()
19	response = client.chat.completions.create(
20	model=model,
21	messages=[{"role": "user", "content": prompt}],
22	max_completion_tokens=200,
23	stream=False,
24	)
25	latency = time.perf_counter() - start
26	latencies.append(latency)
27	token_counts.append(response.usage.completion_tokens)
28
29	avg_latency = sum(latencies) / n
30	avg_tokens = sum(token_counts) / n
31	tokens_per_sec = avg_tokens / avg_latency
32
33	return {
34	"model": model,
35	"avg_latency_s": round(avg_latency, 2),
36	"avg_output_tokens": round(avg_tokens),
37	"tokens_per_sec": round(tokens_per_sec, 1),
38	}
39
40	# ── Streaming example ───────────────────────────────────────────────
41	def stream_local(model: str, prompt: str):
42	"""Stream tokens from local model."""
43	print(f"\n[{model}] Streaming: ", end="")
44	start = time.perf_counter()
45	token_count = 0
46
47	stream = client.chat.completions.create(
48	model=model,
49	messages=[{"role": "user", "content": prompt}],
50	max_completion_tokens=200,
51	stream=True,
52	)
53
54	for chunk in stream:
55	if chunk.choices[0].delta.content:
56	print(chunk.choices[0].delta.content, end="", flush=True)
57	token_count += 1
58
59	elapsed = time.perf_counter() - start
60	print(f"\n Tokens/sec: {token_count/elapsed:.1f}")
61
62	PROMPT = "Explain gradient descent in 3 sentences."
63
64	# Benchmark available models
65	for model in ["llama3.2:3b"]:
66	try:
67	result = benchmark_model(model, PROMPT)
68	print(result)
69	stream_local(model, PROMPT)
70	except Exception as e:
71	print(f"{model}: {e} (model not pulled — run: ollama pull {model})")
72