Online Evaluations — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— Online LLM Evaluations
📋Config
⚙️Route
🔄Collect
✅Analyse
🎯 Next: Test →
Step 1 / 4
Speed
A/B Config
Define control/treatment variants with system prompts and 50/50 split💡 Analogy
Like a restaurant testing two recipes — half the tables get the new dish, half get the old one.
Online eval runs two (or more) variants of your LLM simultaneously on real traffic. Control = current system prompt, Treatment = new version.
Console Output
> Variants: control (50%), treatment (50%)
> control: "You are a helpful assistant."
> treatment: "...Be concise, reply in ≤3 sentences."
●online_evals.py
A/B ConfigL10–17
1import random
2from openai import OpenAI
3from dataclasses import dataclass, field
4from collections import defaultdict
5 
6@dataclass
7class OnlineEvalConfig:
8    variants: dict[str, str]   # variant_name → system_prompt
9    traffic_split: dict[str, float]  # variant → fraction (must sum to 1.0)
10    metrics: list[str] = field(default_factory=lambda: ['thumbs_up', 'response_length'])
11 
12config = OnlineEvalConfig(
13    variants={
14        "control":   "You are a helpful assistant.",
15        "treatment": "You are a helpful assistant. Be concise, reply in ≤3 sentences.",
16    },
17    traffic_split={"control": 0.5, "treatment": 0.5},
18)
19 
20def select_variant(config: OnlineEvalConfig) -> str:
21    rand = random.random()
22    cumulative = 0.0
23    for variant, fraction in config.traffic_split.items():
24        cumulative += fraction
25        if rand <= cumulative:
26            return variant
27    return list(config.variants.keys())[-1]
28 
29results = defaultdict(list)
30 
31def chat_with_online_eval(user_message: str) -> dict:
32    variant = select_variant(config)
33    client = OpenAI()
34    response = client.chat.completions.create(
35        model="gpt-4o-mini",
36        messages=[
37            {"role": "system", "content": config.variants[variant]},
38            {"role": "user", "content": user_message},
39        ],
40    )
41    answer = response.choices[0].message.content
42    results[variant].append(len(answer.split()))
43    return {"answer": answer, "variant": variant}
43 lines
Timeline — 4 Steps
A/B Config
Define control/treatment variants with system prompts and 50/50 split💡 Analogy
Like a restaurant testing two recipes — half the tables get the new dish, half get the old one.
Online eval runs two (or more) variants of your LLM simultaneously on real traffic. Control = current system prompt, Treatment = new version.
Console Output
> Variants: control (50%), treatment (50%)
> control: "You are a helpful assistant."
> treatment: "...Be concise, reply in ≤3 sentences."

Was this page helpful?

Discussion

Define control/treatment variants with system prompts and 50/50 split

💡 Analogy

Like a restaurant testing two recipes — half the tables get the new dish, half get the old one.

Online eval runs two (or more) variants of your LLM simultaneously on real traffic. Control = current system prompt, Treatment = new version.

Console Output

> Variants: control (50%), treatment (50%)
> control: "You are a helpful assistant."
> treatment: "...Be concise, reply in ≤3 sentences."

Define control/treatment variants with system prompts and 50/50 split

💡 Analogy

Like a restaurant testing two recipes — half the tables get the new dish, half get the old one.

Online eval runs two (or more) variants of your LLM simultaneously on real traffic. Control = current system prompt, Treatment = new version.

Console Output

> Variants: control (50%), treatment (50%)
> control: "You are a helpful assistant."
> treatment: "...Be concise, reply in ≤3 sentences."