LLM Benchmarks — Animate | AI Wisdom

← Back to topic
🎞 ANIMATE · Mode 3 of 5 · +100 XP— LLM Benchmarks
📋MMLU
⚙️HumanEval
🔄Arena
✅Interpret
🎯 Next: Test →
Step 1 / 4
Speed
MMLU Benchmark
MMLU tests breadth of knowledge — 57 subjects from law to astronomy💡 Analogy
Like a comprehensive trivia game — breadth of knowledge tested, not depth.
MMLU measures if a model knows facts across diverse academic subjects. High MMLU = strong general knowledge foundation.
Console Output
> MMLU scores (top models):
> GPT-4o: 88.7%
> Claude 3.5 Sonnet: 88.7%
> Gemini 1.5 Pro: 85.9%
> Human expert baseline: ~90%
●benchmarks.py
MMLU BenchmarkL4–10
1# Key LLM benchmarks and what they measure
2# (2024 data — check leaderboards for latest)
3 
4BENCHMARK_SUMMARY = {
5    "MMLU": {
6        "full_name": "Massive Multitask Language Understanding",
7        "measures": "Academic knowledge across 57 subjects",
8        "score_type": "accuracy %",
9        "top_scores": {"GPT-4o": 88.7, "Claude 3.5 Sonnet": 88.7, "Gemini 1.5 Pro": 85.9},
10        "what_it_tells": "General knowledge breadth",
11    },
12    "HumanEval": {
13        "full_name": "HumanEval (OpenAI)",
14        "measures": "Python code generation correctness (164 problems)",
15        "score_type": "pass@1 % (first attempt correct)",
16        "top_scores": {"GPT-4o": 90.2, "Claude 3.5 Sonnet": 92.0, "Gemini 1.5 Pro": 84.1},
17        "what_it_tells": "Coding ability (simple-medium tasks)",
18    },
19    "MATH": {
20        "measures": "Competition math (algebra, calculus, combinatorics)",
21        "top_scores": {"GPT-4o": 76.6, "Claude 3.5 Sonnet": 71.1, "Gemini 1.5 Pro": 67.7},
22        "what_it_tells": "Mathematical reasoning ability",
23    },
24    "GPQA": {
25        "full_name": "Graduate-Level Google-Proof Q&A",
26        "measures": "Expert-level science questions (PhD level)",
27        "top_scores": {"GPT-4o": 53.6, "Claude 3.5 Sonnet": 59.4, "Gemini 1.5 Pro": 46.2},
28        "what_it_tells": "Deep scientific reasoning (hard baseline: 34%)",
29    },
30}
31 
32# LM-Sys Chatbot Arena: human preference ranking
33# Based on ELO rating from 1M+ human comparisons (not automated)
34ARENA_LEADERBOARD = [
35    {"rank": 1,  "model": "GPT-4o (2024-11-20)", "elo": 1374},
36    {"rank": 2,  "model": "Claude 3.5 Sonnet",    "elo": 1370},
37    {"rank": 3,  "model": "Gemini 1.5 Pro",        "elo": 1299},
38]
38 lines
Timeline — 4 Steps
MMLU Benchmark
MMLU tests breadth of knowledge — 57 subjects from law to astronomy💡 Analogy
Like a comprehensive trivia game — breadth of knowledge tested, not depth.
MMLU measures if a model knows facts across diverse academic subjects. High MMLU = strong general knowledge foundation.
Console Output
> MMLU scores (top models):
> GPT-4o: 88.7%
> Claude 3.5 Sonnet: 88.7%
> Gemini 1.5 Pro: 85.9%
> Human expert baseline: ~90%

Was this page helpful?

Discussion

MMLU tests breadth of knowledge — 57 subjects from law to astronomy

💡 Analogy

Like a comprehensive trivia game — breadth of knowledge tested, not depth.

MMLU measures if a model knows facts across diverse academic subjects. High MMLU = strong general knowledge foundation.

Console Output

> MMLU scores (top models):
> GPT-4o: 88.7%
> Claude 3.5 Sonnet: 88.7%
> Gemini 1.5 Pro: 85.9%
> Human expert baseline: ~90%

MMLU tests breadth of knowledge — 57 subjects from law to astronomy

💡 Analogy

Like a comprehensive trivia game — breadth of knowledge tested, not depth.

MMLU measures if a model knows facts across diverse academic subjects. High MMLU = strong general knowledge foundation.

Console Output

> MMLU scores (top models):
> GPT-4o: 88.7%
> Claude 3.5 Sonnet: 88.7%
> Gemini 1.5 Pro: 85.9%
> Human expert baseline: ~90%