LLM Model LandscapeIntermediate

Score → Compare → Decide

Why public benchmarks do not transfer to your workload, how to build a model-selection eval, and why cost per successful task beats price per token.

By Amit UpadhyayPublished 13 Oct 202610 min

Step 1 / 4

Speed

What's happening

Grading Rubric

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score

●evals.py

Grading RubricL5–11

1import json

2from openai import OpenAI

4client = OpenAI()

6# Custom eval: grade LLM responses on a specific task

7GRADING_RUBRIC = """Grade the response on 3 criteria (0-10 each):

81. Accuracy: Is the information correct?

92. Completeness: Does it cover all relevant aspects?

103. Clarity: Is it easy to understand?

12Return JSON: {"accuracy": 0-10, "completeness": 0-10,

13 "clarity": 0-10, "overall": 0-10, "feedback": "..."}"""

15def grade_response(question: str, response: str, reference: str) -> dict:

16 prompt = f"{GRADING_RUBRIC}\n\nQuestion: {question}\nReference: {reference}\nResponse: {response}"

17 result = client.chat.completions.create(

18 model="gpt-6.1-sol", # strong grader

19 messages=[{"role": "user", "content": prompt}],

20 response_format={"type": "json_object"},

21 )

22 return json.loads(result.choices[0].message.content)

24# Run evals across a dataset

25eval_dataset = [

26 {"question": "What is backpropagation?",

27 "reference": "Backpropagation is the algorithm for training neural networks by calculating gradients of the loss function through the chain rule and updating weights."},

28 {"question": "Explain ACID properties",

29 "reference": "ACID: Atomicity (all-or-nothing), Consistency (valid state transitions), Isolation (concurrent transactions), Durability (committed data persists)."},

30]

32def run_eval_suite(model: str) -> dict:

33 scores = []

34 for item in eval_dataset:

35 response = client.chat.completions.create(

36 model=model,

37 messages=[{"role":"user","content":item["question"]}]

38 ).choices[0].message.content

39 grade = grade_response(item["question"], response, item["reference"])

40 scores.append(grade["overall"])

41 return {"model": model, "avg_score": sum(scores)/len(scores)}

41 lines

Steps · 4

What's happening

Grading Rubric

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score

Next: Test →

evals model-selection benchmarks cost-per-task LLM

← PreviousHallucination Testing: Measuring What the Model Makes Up Next →Llama: Open-Weight Models for Production

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape11 min

Llama: Open-Weight Models for Production

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

Discussion

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score

Define multi-criteria rubric: accuracy, completeness, clarity (0-10 each)

💡 Analogy

Like a professor's grading guide — separate points for structure, argument, and evidence.

A rubric defines explicit criteria for grading. Breaking down quality into separate dimensions (accuracy, completeness, clarity) gives more actionable feedback than a single score.

Console Output

> Rubric criteria:
> accuracy: 0-10 (factual correctness)
> completeness: 0-10 (coverage)
> clarity: 0-10 (understandability)
> overall: composite score