AI ObservabilityIntermediate

Build an Eval Pipeline with CI Integration

Build an evaluation dataset from real use, pick the cheapest grader that works, measure the noise floor, and turn the suite into a release gate.

By Amit UpadhyayPublished 13 Oct 202611 min

0/1 steps

Build Steps

Progress

0/1

◆ Before you start

Prerequisites

Python 3.11+Required
python --version
openai, pytestRequired
pip install openai pytest
OPENAI_API_KEYRequired
export OPENAI_API_KEY=<your-api-key>

🧪

Step 1 of 1

Build a pytest-based offline eval suite

Create a repeatable offline evaluation suite using pytest that runs against a golden dataset and gates CI/CD on minimum score thresholds.

Tasks

Create conftest.py with a golden_dataset fixture (10 QA pairs)

Write test_correctness that runs the app and scores with LLM judge

Add assert mean_score >= 3.5 as the gate

Add test_no_hallucination checking responses don't include unknown facts

Run pytest and review the failure report

🎯 What You'll Learn

▸Structure eval datasets as pytest fixtures
▸Use parametrize to run evals over many examples in one test
▸Set pass/fail thresholds to gate deployments

Code

Pythonconftest.py
1 import pytest
2  
3 @pytest.fixture
4 def golden_dataset():
5     return [
6         {"question": "What is the capital of France?", "expected": "Paris"},
7         {"question": "What year did WWII end?", "expected": "1945"},
8         {"question": "What language runs on the JVM?", "expected": "Java"},
9         {"question": "Who wrote Hamlet?", "expected": "Shakespeare"},
10         {"question": "What does HTTP stand for?", "expected": "HyperText Transfer Protocol"},
11     ]
12

💡 Pro Tips

▸Cache LLM responses during offline evals to speed up re-runs
▸Store eval results as JSON artifacts in CI for trend tracking

offline-evals regression-testing datasets LLM-as-judge CI

← PreviousToken Tracking and Cost Management Next →LLM Evaluation Frameworks Compared

Continue reading

All AI Observability articles →

AI Observability11 min

LLM Evaluation Frameworks Compared

Promptfoo, DeepEval, RAGAS, LangSmith and Phoenix compared by what each is good at, with working examples and a way to choose.

AI Observability10 min

Token Tracking and Cost Management

Monitor token usage and costs — per-request tracking, budgets, cost allocation across teams, and the optimization strategies that shrink the bill.

AI Observability14 min

Phoenix: Open-Source LLM Observability

Arize's open-source, OTel-native LLM observability stack — traces, eval workflows, and prompt experiments, self-hosted with no data leaving your VPC.

Discussion