Claude Sonnet 4.5

Crowdsourced pairwise human preference rankings of LLM responses. Higher Elo means more frequently preferred by users.

Real GitHub issues solved end-to-end. Verified subset is a 500-task human-validated slice of SWE-bench.

Long-horizon shell-and-filesystem tasks executed in a sandboxed terminal, scored by whether the agent's final state matches a target state. Tests practical tool-using ability for everyday devops and data-wrangling work; one of the hardest agentic benchmarks today.

Massive Multi-discipline Multimodal Understanding; college-exam level questions with images across 30+ subjects.

Harder MMMU variant: text-only-answerable questions removed, ten answer options instead of four, and a vision-only setting where the question is embedded in the image.

Median output speed in tokens per second over the last 14 days on the model maker's own API, measured nightly with a fixed ~800-token prompt and ~250-token answer. Models not yet probed fall back to a dated manual figure. Higher is faster.

Median time from request to first streamed token in milliseconds over the last 14 days on the model maker's own API, using the same nightly probe prompt. Lower is snappier; reasoning models are penalised here because they think before talking.

Mathematical research problems spanning analysis, algebra, combinatorics and number theory. Tiers 1-3 are progressively harder; even frontier reasoning models only solve a small fraction. The hardest publicly reported benchmark for general mathematical reasoning.

AIME-style competition problems written specifically for the OTIS mock contest, then run as an evaluation by Epoch AI. Closer in spirit to the public AIME but with novel problems unlikely to appear in training data.

A human-validated factuality benchmark of short factual questions whose answers can be checked against a single ground truth. Penalises hallucinations by scoring confidently-wrong answers below abstentions.

Second-generation ARC challenge testing fluid reasoning over abstract visual puzzles. Resists training-data memorisation by construction: each puzzle is novel and solutions require multi-step pattern induction. Frontier models are only just starting to score above chance on the harder tier.

Saturation-resistant composite capability score stitched together from ~40 underlying benchmarks using Item Response Theory. Each benchmark is weighted by its fitted difficulty and discriminative slope, so doing well on hard, contamination-resistant evals (FrontierMath, ARC-AGI 2, Humanity's Last Exam) moves the score and saturated benchmarks contribute almost nothing. Imported per-model from Epoch AI's published index; we anchor it to the same min-max scale we use for every other benchmark so it's directly weightable in scenarios.