LLM Monitoring — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— Production LLM Monitoring
📋Metrics
⚙️Call
🔄Record
✅Expose
🎯 Next: Test →
Step 1 / 4
Speed
Define Metrics
Create Counter, Histogram metrics for requests, tokens, latency, cost💡 Analogy
Like installing meters in your factory — each measures something different (throughput, quality, cost).
Prometheus metrics: Counter (always increases), Histogram (distribution). Labels (model, status) allow slicing — e.g., latency per model.
Console Output
> Metrics registered:
> llm_requests_total (Counter)
> llm_latency_seconds (Histogram)
> llm_cost_usd_total (Counter)
●llm_monitoring.py
Define MetricsL4–12
1import time
2from prometheus_client import Counter, Histogram, start_http_server
3from openai import OpenAI
4 
5# Prometheus metrics
6REQUEST_COUNT = Counter('llm_requests_total', 'Total LLM requests', ['model', 'status'])
7TOKEN_HISTOGRAM = Histogram('llm_tokens', 'Tokens per request', ['model'],
8                             buckets=[50, 100, 200, 500, 1000, 2000])
9LATENCY_HISTOGRAM = Histogram('llm_latency_seconds', 'LLM latency', ['model'],
10                               buckets=[0.1, 0.3, 0.5, 1.0, 2.0, 5.0, 10.0])
11COST_COUNTER = Counter('llm_cost_usd_total', 'Total LLM cost in USD', ['model'])
12 
13# GPT-4o-mini pricing per 1M tokens
14PRICING = {'gpt-4o-mini': {'input': 0.15, 'output': 0.60}}
15 
16def call_with_monitoring(prompt: str, model="gpt-4o-mini") -> str:
17    client = OpenAI()
18    start = time.perf_counter()
19    try:
20        response = client.chat.completions.create(
21            model=model,
22            messages=[{"role": "user", "content": prompt}]
23        )
24        latency = time.perf_counter() - start
25        usage = response.usage
26        cost = (usage.prompt_tokens * PRICING[model]['input'] / 1_000_000 +
27                usage.completion_tokens * PRICING[model]['output'] / 1_000_000)
28 
29        REQUEST_COUNT.labels(model=model, status='success').inc()
30        LATENCY_HISTOGRAM.labels(model=model).observe(latency)
31        TOKEN_HISTOGRAM.labels(model=model).observe(usage.total_tokens)
32        COST_COUNTER.labels(model=model).inc(cost)
33 
34        return response.choices[0].message.content
35    except Exception as e:
36        REQUEST_COUNT.labels(model=model, status='error').inc()
37        raise
38 
39start_http_server(8000)  # Prometheus scrapes :8000/metrics
39 lines
Timeline — 4 Steps
Define Metrics
Create Counter, Histogram metrics for requests, tokens, latency, cost💡 Analogy
Like installing meters in your factory — each measures something different (throughput, quality, cost).
Prometheus metrics: Counter (always increases), Histogram (distribution). Labels (model, status) allow slicing — e.g., latency per model.
Console Output
> Metrics registered:
> llm_requests_total (Counter)
> llm_latency_seconds (Histogram)
> llm_cost_usd_total (Counter)

Was this page helpful?

Discussion

Create Counter, Histogram metrics for requests, tokens, latency, cost

💡 Analogy

Like installing meters in your factory — each measures something different (throughput, quality, cost).

Prometheus metrics: Counter (always increases), Histogram (distribution). Labels (model, status) allow slicing — e.g., latency per model.

Console Output

> Metrics registered:
> llm_requests_total (Counter)
> llm_latency_seconds (Histogram)
> llm_cost_usd_total (Counter)

Create Counter, Histogram metrics for requests, tokens, latency, cost

💡 Analogy

Like installing meters in your factory — each measures something different (throughput, quality, cost).

Prometheus metrics: Counter (always increases), Histogram (distribution). Labels (model, status) allow slicing — e.g., latency per model.

Console Output

> Metrics registered:
> llm_requests_total (Counter)
> llm_latency_seconds (Histogram)
> llm_cost_usd_total (Counter)