AI ObservabilityIntermediate

Golden Dataset Scoring Flow

Build an evaluation dataset from real use, pick the cheapest grader that works, measure the noise floor, and turn the suite into a release gate.

By Amit UpadhyayPublished 13 Oct 202611 min

Step 1 / 4

Speed

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

●offline_evals.py

Build Eval DatasetL13–17

1import json

2from openai import OpenAI

3from dataclasses import dataclass

5@dataclass

6class EvalResult:

7 question: str

8 expected: str

9 actual: str

10 passed: bool

11 score: float

13EVAL_DATASET = [

14 {"question": "What is 2+2?", "expected": "4"},

15 {"question": "Capital of France?", "expected": "Paris"},

16 {"question": "What is RAG?", "expected": "Retrieval-Augmented Generation"},

17]

19def llm_judge(question: str, expected: str, actual: str) -> float:

20 client = OpenAI()

21 prompt = f"""Score how well the actual answer matches the expected answer.

22Question: {question}

23Expected: {expected}

24Actual: {actual}

25Return a JSON with 'score' (0.0-1.0) and 'reasoning'."""

26 response = client.chat.completions.create(

27 model="gpt-6-luna", reasoning_effort="none",

28 messages=[{"role": "user", "content": prompt}],

29 response_format={"type": "json_object"},

30 )

31 return json.loads(response.choices[0].message.content)["score"]

33def run_offline_eval(model_fn) -> list[EvalResult]:

34 results = []

35 for item in EVAL_DATASET:

36 actual = model_fn(item["question"])

37 score = llm_judge(item["question"], item["expected"], actual)

38 results.append(EvalResult(

39 question=item["question"],

40 expected=item["expected"],

41 actual=actual,

42 passed=score >= 0.8,

43 score=score,

44 ))

45 pass_rate = sum(r.passed for r in results) / len(results)

46 print(f"Pass rate: {pass_rate:.0%} ({sum(r.passed for r in results)}/{len(results)})")

47 return results

47 lines

Steps · 4

What's happening

Build Eval Dataset

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

Next: Test →

offline-evals regression-testing datasets LLM-as-judge CI

← PreviousToken Tracking and Cost Management Next →LLM Evaluation Frameworks Compared

Continue reading

All AI Observability articles →

AI Observability11 min

LLM Evaluation Frameworks Compared

Promptfoo, DeepEval, RAGAS, LangSmith and Phoenix compared by what each is good at, with working examples and a way to choose.

AI Observability10 min

Token Tracking and Cost Management

Monitor token usage and costs — per-request tracking, budgets, cost allocation across teams, and the optimization strategies that shrink the bill.

AI Observability14 min

Phoenix: Open-Source LLM Observability

Arize's open-source, OTel-native LLM observability stack — traces, eval workflows, and prompt experiments, self-hosted with no data leaving your VPC.

Discussion

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"

EVAL_DATASET pairs questions with expected answers — the ground truth

💡 Analogy

Like a fixed exam — same questions every semester so you can track improvement over time.

An offline eval dataset contains questions with known correct answers. You run your LLM on these and compare outputs to expected answers.

Console Output

> EVAL_DATASET: 3 samples
> Q1: "What is 2+2?" → expected "4"
> Q2: "Capital of France?" → expected "Paris"
> Q3: "What is RAG?" → expected "Retrieval-Augmented Generation"