What the benchmarks cover
As of October 25, 2026, a typical scenario score rests on a median of 30% of that scenario's intended benchmark weight, and 1 of 25 quality benchmarks has 5 or more models tied at the top score.
A ranking is only as good as the evidence behind it. When a model has no score on a benchmark, its scenario score is computed from the benchmarks it does have, so two models on the same leaderboard can rest on very different amounts of data. This page shows where that evidence is deep, where it is thin, and where a benchmark has stopped telling the top models apart.
Evidence map
Coverage is the share of a scenario's benchmark weight for which a model has a published score. The cards give the median for the models each scenario ranks, and how many of them reach 80%.
- 👨💻 Coding Assistant
- Ranked
- 186
- Median coverage
- 25%
- 80%+ covered
- 10
- 🖥️ Local / Self-Hosted Coding
- Ranked
- 58
- Median coverage
- 10%
- 80%+ covered
- 1
- 💬 Customer Support Bot
- Ranked
- 166
- Median coverage
- 35%
- 80%+ covered
- 0
- 🔬 Research / Analyst
- Ranked
- 146
- Median coverage
- 30%
- 80%+ covered
- 0
- 🧮 Math / Reasoning Tutor
- Ranked
- 153
- Median coverage
- 30%
- 80%+ covered
- 1
- 🏷️ Bulk Classification
- Ranked
- 165
- Median coverage
- 20%
- 80%+ covered
- 6
- 📄 Multimodal Document Q&A
- Ranked
- 31
- Median coverage
- 45%
- 80%+ covered
- 5
- 📊 Data / BI Analyst
- Ranked
- 34
- Median coverage
- 25%
- 80%+ covered
- 0
- ⚡ Realtime Chat / Voice
- Ranked
- 238
- Median coverage
- 75%
- 80%+ covered
- 5
- 🎨 Marketing Image Generation
- Ranked
- 9
- Median coverage
- 100%
- 80%+ covered
- 9
- 🛡️ Production Critical
- Ranked
- 214
- Median coverage
- 15%
- 80%+ covered
- 23
Each cell shows the benchmark's share of that scenario's weight on top, and below it how many of the scenario's ranked models have a score on it. The shading follows the second number, so a heavily weighted benchmark in a pale cell is one most models are ranked without.
| Benchmark | Coding Assistant | Local / Self-Hosted Coding | Customer Support Bot | Research / Analyst | Math / Reasoning Tutor | Bulk Classification | Multimodal Document Q&A | Data / BI Analyst | Realtime Chat / Voice | Marketing Image Generation | Production Critical |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Arena | 15% of weight, 165/186 ranked models scored | Not used | 35% of weight, 165/166 ranked models scored | Not used | Not used | 20% of weight, 165/165 ranked models scored | Not used | Not used | 20% of weight, 165/238 ranked models scored | Not used | Not used |
| MMLU-Pro | Not used | Not used | 10% of weight, 6/166 ranked models scored | 10% of weight, 6/146 ranked models scored | Not used | 35% of weight, 6/165 ranked models scored | Not used | Not used | Not used | Not used | Not used |
| GPQA | Not used | Not used | Not used | 20% of weight, 6/146 ranked models scored | 15% of weight, 6/153 ranked models scored | Not used | 20% of weight, 0/31 ranked models scored | 10% of weight, 6/34 ranked models scored | Not used | Not used | Not used |
| AIME | Not used | Not used | Not used | Not used | 15% of weight, 28/153 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used |
| LiveCB | 30% of weight, 17/186 ranked models scored | 30% of weight, 3/58 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used |
| SWE-bench | 25% of weight, 43/186 ranked models scored | 20% of weight, 14/58 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used |
| IFEval | Not used | 10% of weight, 6/58 ranked models scored | 20% of weight, 6/166 ranked models scored | Not used | 10% of weight, 6/153 ranked models scored | 45% of weight, 6/165 ranked models scored | Not used | 20% of weight, 6/34 ranked models scored | 25% of weight, 6/238 ranked models scored | Not used | 15% of weight, 6/214 ranked models scored |
| MMMU | Not used | Not used | Not used | Not used | Not used | Not used | 25% of weight, 24/31 ranked models scored | Not used | Not used | Not used | Not used |
| MMMU-Pro | Not used | Not used | Not used | Not used | Not used | Not used | 20% of weight, 18/31 ranked models scored | Not used | Not used | Not used | Not used |
| MathVista | Not used | Not used | Not used | Not used | Not used | Not used | 25% of weight, 12/31 ranked models scored | Not used | Not used | Not used | Not used |
| RULER | Not used | Not used | 10% of weight, 28/166 ranked models scored | 25% of weight, 28/146 ranked models scored | Not used | Not used | 10% of weight, 12/31 ranked models scored | 25% of weight, 28/34 ranked models scored | Not used | Not used | 5% of weight, 28/214 ranked models scored |
| Img Arena | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 55% of weight, 9/9 ranked models scored | Not used |
| Prompt Fid. | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 45% of weight, 9/9 ranked models scored | Not used |
| Speed | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 30% of weight, 233/238 ranked models scored | Not used | Not used |
| TTFT | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 25% of weight, 233/238 ranked models scored | Not used | Not used |
| Data Analysis | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 45% of weight, 9/34 ranked models scored | Not used | Not used | Not used |
| FrontierMath | Not used | Not used | Not used | Not used | 30% of weight, 58/153 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used |
| SimpleQA | Not used | Not used | Not used | 10% of weight, 79/146 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used |
| OTIS AIME | Not used | Not used | Not used | Not used | 20% of weight, 136/153 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used |
| HLE | Not used | Not used | Not used | 15% of weight, 37/146 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used |
| Aider | 15% of weight, 34/186 ranked models scored | 25% of weight, 10/58 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used |
| Frontier | 10% of weight, 137/186 ranked models scored | 10% of weight, 50/58 ranked models scored | Not used | 20% of weight, 137/146 ranked models scored | 10% of weight, 137/153 ranked models scored | Not used | Not used | Not used | Not used | Not used | 15% of weight, 137/214 ranked models scored |
| Stability | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 15% of weight, 24/214 ranked models scored |
| Format | Not used | Not used | 15% of weight, 24/166 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 15% of weight, 24/214 ranked models scored |
| Recovery | 5% of weight, 24/186 ranked models scored | 5% of weight, 5/58 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 10% of weight, 24/214 ranked models scored |
| API OK | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 10% of weight, 156/214 ranked models scored |
| Safety | Not used | Not used | 10% of weight, 24/166 ranked models scored | Not used | Not used | Not used | Not used | Not used | Not used | Not used | 15% of weight, 24/214 ranked models scored |
- 75%+ of ranked models scored
- 40–74%
- 15–39%
- Under 15%
Benchmark health
When a measure becomes a target, it stops being a good measure. The contamination label is our judgement of how exposed a question set is to training data. The tie count is measured from the data: when 5 or more models share the top score (out of at least 10 results), the benchmark no longer separates the leaders, whatever its label says. Spread is the normalised gap between #1 and #10.
The “Compressed at top” and “Few results” badges are for information only. They do not change any score: the Analyzer Index v1.0 and every scenario keep their published benchmarks and weights, and a change to either ships as a new index version.
| Benchmark | Models | Question set | Newest score | Label | Top ties | Spread | Used in |
|---|---|---|---|---|---|---|---|
| Chatbot Arena ElopreferenceCrowdsourced pairwise human preference rankings of LLM responses. Higher Elo means more frequently preferred by users. | 165 | 2026-04-01 | 2026-10-25 | Fresh | 1 | 9.6 | Index 10%4 scenariosBest value |
| MMLU ProknowledgeHarder version of MMLU testing knowledge across 57 academic subjects; reduces guessing-friendly answers. | 6Few results | 2024-05-15 | 2026-10-25 | High risk | 1 | — | 3 scenariosBest value |
| GPQA DiamondreasoningGraduate-level Google-proof Q&A in physics, chemistry, and biology. Diamond subset is the hardest tier with PhD-validated answers. | 6Few results | 2023-11-01 | 2026-10-25 | Some risk | 1 | — | Index 5%4 scenariosBest value |
| MATH-500math500 high-school competition math problems requiring multi-step solutions. Scored on final-answer correctness. | 6 | 2021-03-01 | 2026-10-25 | Saturated | 1 | — | Best value |
| AIME 2024mathAmerican Invitational Mathematics Examination 2024 problems. Three-digit integer answers; very hard for non-reasoning models. | 28 | 2024-02-01 | 2025-04-30 | High risk | 1 | 54.0 | 1 scenarioBest value |
| HumanEvalcoding164 hand-written Python programming problems scored by passing unit tests. Saturated for frontier models. | 28 | 2021-07-01 | 2025-04-30 | Saturated | 1 | 18.5 | Best value |
| LiveCodeBenchcodingContinuously refreshed coding benchmark drawing from LeetCode, AtCoder, and Codeforces; reduces benchmark contamination. | 17 | 2026-04-01 | 2026-10-25 | Fresh | 1 | 36.2 | Index 6%2 scenariosBest value |
| SWE-bench VerifiedagenticReal GitHub issues solved end-to-end. Verified subset is a 500-task human-validated slice of SWE-bench. | 43 | 2024-08-15 | 2026-10-25 | Some risk | 1 | 9.1 | Index 10%2 scenariosBest value |
| IFEvalinstruction followingVerifiable instruction-following benchmark; 25 categories of strict formatting / structural directives. | 6Few results | 2023-11-01 | 2026-10-25 | Some risk | 1 | — | 7 scenariosBest value |
| MMMUvisionMassive Multi-discipline Multimodal Understanding; college-exam level questions with images across 30+ subjects. | 31 | 2024-03-15 | 2025-11-24 | Some risk | 1 | 20.8 | 1 scenarioBest value |
| MMMU-ProvisionHarder MMMU variant: text-only-answerable questions removed, ten answer options instead of four, and a vision-only setting where the question is embedded in the image. | 19 | 2024-09-04 | 2026-04-02 | Some risk | 1 | 17.0 | 1 scenarioBest value |
| MathVistavisionMath reasoning over visual contexts (charts, figures, geometry). | 18 | 2023-10-01 | 2025-04-30 | Some risk | 1 | 44.4 | 1 scenarioBest value |
| RULER 128klong contextLong-context retrieval and reasoning suite. We report the 128k token effective-context score. | 28 | 2024-08-01 | 2025-04-30 | Fresh | 1 | 11.6 | 5 scenariosBest value |
| Image Arena Eloimage genCrowdsourced pairwise human preference for image generation models. Users vote on anonymised side-by-side generations; scores are a standard Elo over those votes. | 9Few results | 2026-04-01 | 2026-10-25 | Fresh | 1 | — | 1 scenarioBest value |
| Prompt Adherenceimage genHow well the generated image matches the textual prompt as evaluated by human raters. | 9Few results | 2026-04-01 | 2026-10-25 | Fresh | 1 | — | 1 scenarioBest value |
| Rolling Contamination-Controlled AveragegeneralContamination-controlled average across every task category in the latest rolling release (e.g. reasoning, coding, data analysis, research, browsing, long-horizon tasks). Questions are rotated with each release and graded against objective ground truth, removing the need for LLM-as-judge scoring. Only models with a score in every category get an average. | 9 | 2026-01-08 | 2026-01-08 | Fresh | 1 | — | Index 10%Best value |
| Rolling Data Analysisdata analysisRolling contamination-controlled data-analysis evaluation. Table comprehension, CSV / spreadsheet reasoning, SQL-style joins, and chart interpretation. Refreshed every six months with new tables and questions to minimise contamination. | 9Few results | 2026-01-08 | 2026-01-08 | Fresh | 1 | — | 1 scenarioBest value |
| FrontierMath Tiers 1-3mathMathematical research problems spanning analysis, algebra, combinatorics and number theory. Tiers 1-3 are progressively harder; even frontier reasoning models only solve a small fraction. The hardest publicly reported benchmark for general mathematical reasoning. | 58 | 2026-04-23 | 2026-10-25 | Fresh | 1 | 27.3 | Index 7%1 scenarioBest value |
| SimpleQA VerifiedknowledgeA human-validated factuality benchmark of short factual questions whose answers can be checked against a single ground truth. Penalises hallucinations by scoring confidently-wrong answers below abstentions. | 79 | 2026-04-23 | 2026-10-25 | Fresh | 1 | 9.7 | Index 5%1 scenarioBest value |
| OTIS Mock AIME 2024-2025mathAIME-style competition problems written specifically for the OTIS mock contest, then run as an evaluation by Epoch AI. Closer in spirit to the public AIME but with novel problems unlikely to appear in training data. | 136 | 2026-04-23 | 2026-10-25 | Fresh | 11Compressed at top | 0.0 | Index 8%1 scenarioBest value |
| Humanity's Last ExamreasoningA challenging multi-disciplinary exam aggregating expert-written questions from across academic fields. Designed to discriminate at the very top of the capability range when MMLU-style tests saturate. | 37 | 2026-04-08 | 2026-10-25 | Fresh | 1 | 44.5 | Index 10%1 scenarioBest value |
| ARC-AGI 2reasoningSecond-generation ARC challenge testing fluid reasoning over abstract visual puzzles. Resists training-data memorisation by construction: each puzzle is novel and solutions require multi-step pattern induction. Frontier models are only just starting to score above chance on the harder tier. | 86 | 2026-04-23 | 2026-10-25 | Fresh | 1 | 6.1 | Index 5%Best value |
| Aider PolyglotcodingReal-world refactoring and bug-fix tasks across multiple programming languages, scored by whether the model produces a passing patch in Aider's edit format. Tests practical coding ability beyond single-file generation; harder than HumanEval and not yet saturated. | 34 | 2025-08-05 | 2026-10-25 | Fresh | 1 | 21.1 | Index 6%2 scenariosBest value |
| Terminal-Bench 2agenticLong-horizon shell-and-filesystem tasks executed in a sandboxed terminal, scored by whether the agent's final state matches a target state. Tests practical tool-using ability for everyday devops and data-wrangling work; one of the hardest agentic benchmarks today. | 41 | 2026-04-23 | 2026-10-25 | Fresh | 1 | 26.5 | Index 8%Best value |
| Frontier CompositecompositeSaturation-resistant composite capability score stitched together from ~40 underlying benchmarks using Item Response Theory. Each benchmark is weighted by its fitted difficulty and discriminative slope, so doing well on hard, contamination-resistant evals (FrontierMath, ARC-AGI 2, Humanity's Last Exam) moves the score and saturated benchmarks contribute almost nothing. Imported per-model from Epoch AI's published index; we anchor it to the same min-max scale we use for every other benchmark so it's directly weightable in scenarios. | 137 | 2026-04-23 | 2026-10-25 | Fresh | 2 | 11.2 | Index 10%5 scenariosBest value |
Operational probes
Speed, latency, uptime and reliability measure how a deployed API behaves, not how well a model answers, so the tie and spread checks above do not apply to them. Many models sharing a perfect uptime or format score is good news, not saturation.
| Probe | Models | Newest score | Used in |
|---|---|---|---|
| Output SpeedperformanceMedian output speed in tokens per second over the last 14 days on the model maker's own API, measured nightly with a fixed ~800-token prompt and ~250-token answer. Models not yet probed fall back to a dated manual figure. Higher is faster. | 233 | 2026-10-25 | 1 scenarioBest value |
| Time to First TokenperformanceMedian time from request to first streamed token in milliseconds over the last 14 days on the model maker's own API, using the same nightly probe prompt. Lower is snappier; reasoning models are penalised here because they think before talking. | 233 | 2026-10-25 | 1 scenarioBest value |
| Output StabilityreliabilityHow consistent the model's outputs are across repeated runs of the same task. Higher means lower variance, fewer occasional hallucinations under identical inputs. Useful for production loops that need reproducible behaviour. | 24 | 2026-10-25 | 1 scenarioBest value |
| Format AdherencereliabilityHow reliably the model produces output in the requested format (JSON schemas, markdown structures, exact-string responses). Pairs well with IFEval but reflects how the deployed API is behaving day to day rather than how a frozen test set scores. | 24 | 2026-10-25 | 2 scenariosBest value |
| Recovery RatereliabilityHow often the model self-corrects after producing an incorrect intermediate step (debugging axis upstream). Critical for agentic loops that depend on the model noticing and repairing its own mistakes rather than barrelling forward. | 24 | 2026-10-25 | 3 scenariosBest value |
| API Success RatereliabilityShare of nightly probe requests over the last 14 days that streamed a complete answer without an error, timeout or empty response. Needs at least five attempts before a figure is shown. | 156 | 2026-10-25 | 1 scenarioBest value |
| Safety HandlingreliabilityHow well the model handles safety-sensitive prompts without false-refusing benign requests or producing unsafe output. The upstream signal does not separate refusal counts from substantive content-safety behaviour, so this single axis covers both. | 24 | 2026-10-25 | 2 scenariosBest value |
Does the weighting change the winner?
Every weighted index is a choice. Below, the same 13 Analyzer Index benchmarks are combined three ways, with the same coverage rules. The leader changes between these allocations of the same benchmarks, so treat close calls at the top as ties.
Published weights (v1.0)
The index as shown on the leaderboard.
- 1Claude Opus 5.598.4
- 2Gemini 4 Argon98.3
- 3GPT-6 Astra95.1
- 4Claude Fable 5.193.7
- 5GPT-6.1 Sol93.4
- 6Claude Fable 593.0
- 7Claude Opus 592.4
- 8GPT-5.591.2
- 9GPT-5.6 Sol90.5
- 10Gemini 3.7 Flash88.7
Every benchmark counts equally
Each of the 13 benchmarks gets the same weight.
- 1Gemini 4 Argon+1 places versus published weights98.3
- 2Claude Opus 5.5-1 places versus published weights98.0
- 3GPT-6 Astra96.1
- 4GPT-6.1 Sol+1 places versus published weights94.6
- 5Claude Fable 5.1-1 places versus published weights93.9
- 6Claude Fable 593.3
- 7GPT-5.6 Sol+2 places versus published weights91.6
- 8Claude Opus 5-1 places versus published weights91.4
- 9GPT-5.5-1 places versus published weights91.2
- 10Gemini 3.7 Flash89.2
Every pillar counts equally
Each of the 4 pillars gets the same weight, split evenly inside it.
- 1Claude Opus 5.598.6
- 2Gemini 4 Argon98.4
- 3Claude Fable 5.1+1 places versus published weights95.4
- 4GPT-6 Astra-1 places versus published weights94.5
- 5GPT-6.1 Sol93.5
- 6Claude Fable 593.4
- 7Muse Wondernew places versus published weights93.1
- 8Claude Opus 5-1 places versus published weights93.1
- 9GPT-5.5-1 places versus published weights91.7
- 10GPT-5.6 Sol-1 places versus published weights90.8
Want a different split? Fork this index in the scenario builder.
What benchmarks do not measure
- Uneven coverage. Benchmarks cluster where evaluation is easy, such as coding, maths and exam-style questions. Support, data analysis and regulated drafting lean on benchmarks with far fewer results, so their rankings move more when a new score lands.
- Saturation. Once the leaders all max out a test, it adds weight without adding information. The tie count above flags these.
- Code quality. Coding benchmarks check whether a patch passes tests. They do not measure whether the change is easy to review, fits the codebase, or adds complexity that slows the next change. Recovery Rate is the closest signal we track; a trial on your own repository is still the best test.
- Working with people. Most benchmarks score a model working alone. How much it speeds up a person who uses it, and tasks with a physical side, are largely unmeasured.