LLM Evals — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— LLM Evals
📋Rubric
⚙️Grade
🔄Dataset
✅Compare
🎯 Next: Test →
Step 1 / 4
Speed
Grading Rubric
Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)💡 Analogy
Like a professor's grading guide — separate points for structure, argument, and evidence.
A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.
Console Output
> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score
●evals.py
Grading RubricL5–11
1import json
2from openai import OpenAI
3 
4client = OpenAI()
5 
6# Custom eval: grade LLM responses on a specific task
7GRADING_RUBRIC = """Grade the response on 3 criteria (0-10 each):
81. Accuracy: Is the information correct?
92. Completeness: Does it cover all relevant aspects?
103. Clarity: Is it easy to understand?
11 
12Return JSON: {"accuracy": 0-10, "completeness": 0-10, 
13              "clarity": 0-10, "overall": 0-10, "feedback": "..."}"""
14 
15def grade_response(question: str, response: str, reference: str) -> dict:
16    prompt = f"{GRADING_RUBRIC}\n\nQuestion: {question}\nReference: {reference}\nResponse: {response}"
17    result = client.chat.completions.create(
18        model="gpt-4o",   # strong grader
19        messages=[{"role": "user", "content": prompt}],
20        response_format={"type": "json_object"},
21    )
22    return json.loads(result.choices[0].message.content)
23 
24# Run evals across a dataset
25eval_dataset = [
26    {"question": "What is backpropagation?",
27     "reference": "Backpropagation is the algorithm for training neural networks by calculating gradients of the loss function through the chain rule and updating weights."},
28    {"question": "Explain ACID properties",
29     "reference": "ACID: Atomicity (all-or-nothing), Consistency (valid state transitions), Isolation (concurrent transactions), Durability (committed data persists)."},
30]
31 
32def run_eval_suite(model: str) -> dict:
33    scores = []
34    for item in eval_dataset:
35        response = client.chat.completions.create(
36            model=model,
37            messages=[{"role":"user","content":item["question"]}]
38        ).choices[0].message.content
39        grade = grade_response(item["question"], response, item["reference"])
40        scores.append(grade["overall"])
41    return {"model": model, "avg_score": sum(scores)/len(scores)}
41 lines
Timeline — 4 Steps
Grading Rubric
Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)💡 Analogy
Like a professor's grading guide — separate points for structure, argument, and evidence.
A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.
Console Output
> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score

Was this page helpful?

Discussion

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score