Prerequisites

Python 3.11+Required
python --version
openai, pytestRequired
pip install openai pytest
OPENAI_API_KEYRequired
export OPENAI_API_KEY=sk-...

🧪

Step 1 of 1

Build a pytest-based offline eval suite

Create a repeatable offline evaluation suite using pytest that runs against a golden dataset and gates CI/CD on minimum score thresholds.

Tasks

Create conftest.py with a golden_dataset fixture (10 QA pairs)

Write test_correctness that runs the app and scores with LLM judge

Add assert mean_score >= 3.5 as the gate

Add test_no_hallucination checking responses don't include unknown facts

Run pytest and review the failure report

🎯 What You'll Learn

▸Structure eval datasets as pytest fixtures
▸Use parametrize to run evals over many examples in one test
▸Set pass/fail thresholds to gate deployments

Code

Pythonconftest.py
1 import pytest
2  
3 @pytest.fixture
4 def golden_dataset():
5     return [
6         {"question": "What is the capital of France?", "expected": "Paris"},
7         {"question": "What year did WWII end?", "expected": "1945"},
8         {"question": "What language runs on the JVM?", "expected": "Java"},
9         {"question": "Who wrote Hamlet?", "expected": "Shakespeare"},
10         {"question": "What does HTTP stand for?", "expected": "HyperText Transfer Protocol"},
11     ]
12

💡 Pro Tips

▸Cache LLM responses during offline evals to speed up re-runs
▸Store eval results as JSON artifacts in CI for trend tracking

1 import pytest 2 3 @pytest.fixture 4 def golden_dataset(): 5 return [ 6 {"question": "What is the capital of France?", "expected": "Paris"}, 7 {"question": "What year did WWII end?", "expected": "1945"}, 8 {"question": "What language runs on the JVM?", "expected": "Java"}, 9 {"question": "Who wrote Hamlet?", "expected": "Shakespeare"}, 10 {"question": "What does HTTP stand for?", "expected": "HyperText Transfer Protocol"}, 11 ] 12

Build: Build a pytest-based offline eval suite

Prerequisites

Build a pytest-based offline eval suite

Tasks

🎯 What You'll Learn

Code

💡 Pro Tips

Discussion

Build: Build a pytest-based offline eval suite

Prerequisites

Build a pytest-based offline eval suite

Tasks

🎯 What You'll Learn

Code

💡 Pro Tips

Discussion

1	import pytest
2
3	@pytest.fixture
4	def golden_dataset():
5	return [
6	{"question": "What is the capital of France?", "expected": "Paris"},
7	{"question": "What year did WWII end?", "expected": "1945"},
8	{"question": "What language runs on the JVM?", "expected": "Java"},
9	{"question": "Who wrote Hamlet?", "expected": "Shakespeare"},
10	{"question": "What does HTTP stand for?", "expected": "HyperText Transfer Protocol"},
11	]
12