Offline Evaluations Animate

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Offline LLM Evaluations

📋Dataset

⚙️LLM Judge

🔄Score

✅Report

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

●offline_evals.py

Build Eval DatasetL13–17

1import json

2from openai import OpenAI

3from dataclasses import dataclass

5@dataclass

6class EvalResult:

7 question: str

8 expected: str

9 actual: str

10 passed: bool

11 score: float

13EVAL_DATASET = [

14 {"question": "What is 2+2?", "expected": "4"},

15 {"question": "Capital of France?", "expected": "Paris"},

16 {"question": "What is RAG?", "expected": "Retrieval-Augmented Generation"},

17]

19def llm_judge(question: str, expected: str, actual: str) -> float:

20 client = OpenAI()

21 prompt = f"""Score how well the actual answer matches the expected answer.

22Question: {question}

23Expected: {expected}

24Actual: {actual}

25Return a JSON with 'score' (0.0-1.0) and 'reasoning'."""

26 response = client.chat.completions.create(

27 model="gpt-4o-mini",

28 messages=[{"role": "user", "content": prompt}],

29 response_format={"type": "json_object"},

30 )

31 return json.loads(response.choices[0].message.content)["score"]

33def run_offline_eval(model_fn) -> list[EvalResult]:

34 results = []

35 for item in EVAL_DATASET:

36 actual = model_fn(item["question"])

37 score = llm_judge(item["question"], item["expected"], actual)

38 results.append(EvalResult(

39 question=item["question"],

40 expected=item["expected"],

41 actual=actual,

42 passed=score >= 0.8,

43 score=score,

44 ))

45 pass_rate = sum(r.passed for r in results) / len(results)

46 print(f"Pass rate: {pass_rate:.0%} ({sum(r.passed for r in results)}/{len(results)})")

47 return results

47 lines

Steps · 4

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

Was this page helpful?

Discussion

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Offline LLM Evaluations

📋Dataset

⚙️LLM Judge

🔄Score

✅Report

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

●offline_evals.py

Build Eval DatasetL13–17

1import json

2from openai import OpenAI

3from dataclasses import dataclass

5@dataclass

6class EvalResult:

7 question: str

8 expected: str

9 actual: str

10 passed: bool

11 score: float

13EVAL_DATASET = [

14 {"question": "What is 2+2?", "expected": "4"},

15 {"question": "Capital of France?", "expected": "Paris"},

16 {"question": "What is RAG?", "expected": "Retrieval-Augmented Generation"},

17]

19def llm_judge(question: str, expected: str, actual: str) -> float:

20 client = OpenAI()

21 prompt = f"""Score how well the actual answer matches the expected answer.

22Question: {question}

23Expected: {expected}

24Actual: {actual}

25Return a JSON with 'score' (0.0-1.0) and 'reasoning'."""

26 response = client.chat.completions.create(

27 model="gpt-4o-mini",

28 messages=[{"role": "user", "content": prompt}],

29 response_format={"type": "json_object"},

30 )

31 return json.loads(response.choices[0].message.content)["score"]

33def run_offline_eval(model_fn) -> list[EvalResult]:

34 results = []

35 for item in EVAL_DATASET:

36 actual = model_fn(item["question"])

37 score = llm_judge(item["question"], item["expected"], actual)

38 results.append(EvalResult(

39 question=item["question"],

40 expected=item["expected"],

41 actual=actual,

42 passed=score >= 0.8,

43 score=score,

44 ))

45 pass_rate = sum(r.passed for r in results) / len(results)

46 print(f"Pass rate: {pass_rate:.0%} ({sum(r.passed for r in results)}/{len(results)})")

47 return results

47 lines

Steps · 4

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"