Evaluation 8 min read Updated 2026-10-31
How to Read LLM Benchmarks: MMLU-Pro, SWE-bench, MATH-500 & IFEval
A rigorous field guide to identifying data contamination, variance, test harness differences, and deceptive marketing claims.
Authored by Cadle Research Team · Strictly peer-reviewed with verifiable equations and benchmarks.
EXECUTIVE SUMMARY
Benchmark scores are the primary currency used by AI labs to claim frontier superiority. However, scores vary drastically based on prompt formats, sampling temperatures, few-shot splits, and harness implementations. This guide establishes the gold standard for comparing SWE-bench Verified, MMLU-Pro, MATH-500, and IFEval.
1. The Benchmark Saturation Crisis
Legacy benchmarks like original MMLU (57 multiple-choice subjects) and GSM8K (grade-school math) have saturated beyond 90%, rendering them incapable of distinguishing between frontier reasoning models. Modern benchmarks replace these with rigorous, multi-step problem sets:
- **MMLU-Pro**: Replaces 4-option questions with 10 options and eliminates noisy ambiguous questions. Frontier human baseline is ~85%; top models currently hover between 70% and 82%.
- **SWE-bench Verified**: Requires resolving real GitHub pull requests from open-source repositories with unit-test verification in a sandboxed Docker environment.
- **MATH-500**: A curated 500-problem subset of challenging competition mathematics (AMC, AIME, Olympiad) evaluating step-by-step LaTeX proof derivations.
- **IFEval**: Evaluates strict adherence to verifiable constraints (e.g. 'write exactly 3 paragraphs', 'include keyword X at least 4 times', 'respond in JSON only').
2. The SWE-bench Methodology: Resolved vs. Verified
In SWE-bench, an agent receives an issue description and a full git repository codebase. It must inspect files, reproduce the bug, edit code, and generate a git patch.
- **SWE-bench Lite**: 300 selected issues, but contains several poorly specified tests where passing was impossible.
- **SWE-bench Verified**: A human-audited set of 500 solvable GitHub issues with verified unit tests. Models scoring 50%+ on SWE-bench Verified exhibit genuine software engineering utility.
- **Pass@1 vs. Pass@k**: Always verify whether reported scores reflect **Pass@1** (single attempt) or **Pass@k** with best-of-N voting or execution feedback reranking. Pass@1 is the true measure of interactive coding capability.
3. Contamination and Prompt Sensitivity
Three primary vectors distort published benchmark tables:
1. **Pretraining Contamination**: If the benchmark test questions or solutions were scraped into the training corpus, the model performs associative recall rather than reasoning.
2. **Harness Variance**: Testing via EleutherAI lm-evaluation-harness vs. inspect-ai vs. proprietary lab prompts can cause an 8–12 point swing on the exact same model weights.
3. **Chain-of-Thought (CoT) Prompting**: Disclosing whether CoT was enabled. A model evaluated zero-shot will score 15–25 points lower on math than one permitted reasoning tokens.