Edge & Small Models — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— Edge & Small Language Models
📋Models
⚙️Ollama
🔄Structured
✅Use Cases
🎯 Next: Test →
Step 1 / 4
Speed
SLM Landscape
Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost💡 Analogy
Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.
Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.
Console Output
> Phi-3 Mini 3.8B: 2GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local
●edge_models.py
SLM LandscapeL1–5
1# Run Phi-3 Mini (3.8B) locally with Ollama
2# ollama pull phi3:mini
3 
4import ollama
5import json
6 
7# Phi-3 Mini: 3.8B params, runs on CPU (M1 MacBook Pro)
8def local_inference(prompt: str) -> str:
9    response = ollama.chat(
10        model="phi3:mini",
11        messages=[
12            {"role": "user", "content": prompt}
13        ],
14        options={
15            "temperature": 0.1,     # deterministic for structured tasks
16            "num_ctx": 4096,        # context window size
17        }
18    )
19    return response["message"]["content"]
20 
21# Structured extraction with small model
22def extract_json(text: str) -> dict:
23    prompt = f"""Extract the following from this text and return ONLY JSON:
24{{
25  "person_name": "...",
26  "date": "YYYY-MM-DD or null",
27  "action": "..."
28}}
29 
30Text: {text}"""
31    result = local_inference(prompt)
32    return json.loads(result)
33 
34# Batch processing: 1000 documents, no API cost
35results = []
36documents = ["Alice met Bob on Jan 5, 2025...",
37             "Carol scheduled meeting for Feb 3..."]
38for doc in documents:
39    extracted = extract_json(doc)
40    results.append(extracted)
41 
42# Speed: Phi-3 Mini on M1 Mac ≈ 40-60 tok/s
43# Cost: $0 (fully local)
44# Privacy: data never leaves your device
44 lines
Timeline — 4 Steps
SLM Landscape
Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost💡 Analogy
Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.
Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.
Console Output
> Phi-3 Mini 3.8B: 2GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

Was this page helpful?

Discussion

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-3 Mini 3.8B: 2GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local

Small Language Models (1-7B): run locally on CPU/consumer GPU, zero cost

💡 Analogy

Like a pocket calculator vs mainframe — smaller, local, good enough for most calculations.

Small Language Models (SLMs): 1-7B parameters. Run on MacBook M-series, consumer GPUs (RTX 3070+), or even CPU. No API cost, full privacy.

Console Output

> Phi-3 Mini 3.8B: 2GB RAM (4-bit GGUF)
> Runs on: CPU (slow), M1 Mac (fast), RTX3090 (fast)
> Cost: $0
> Privacy: 100% local