Mixtral 8x22B

Crowdsourced pairwise human preference rankings of LLM responses. Higher Elo means more frequently preferred by users.

Harder version of MMLU testing knowledge across 57 academic subjects; reduces guessing-friendly answers.

Graduate-level Google-proof Q&A in physics, chemistry, and biology. Diamond subset is the hardest tier with PhD-validated answers.

500 high-school competition math problems requiring multi-step solutions. Scored on final-answer correctness.

American Invitational Mathematics Examination 2024 problems. Three-digit integer answers; very hard for non-reasoning models.

164 hand-written Python programming problems scored by passing unit tests. Saturated for frontier models.

Verifiable instruction-following benchmark; 25 categories of strict formatting / structural directives.

Long-context retrieval and reasoning suite. We report the 128k token effective-context score.

Median output speed in tokens per second over the last 14 days on the model maker's own API, measured nightly with a fixed ~800-token prompt and ~250-token answer. Models not yet probed fall back to a dated manual figure. Higher is faster.

Median time from request to first streamed token in milliseconds over the last 14 days on the model maker's own API, using the same nightly probe prompt. Lower is snappier; reasoning models are penalised here because they think before talking.

Saturation-resistant composite capability score stitched together from ~40 underlying benchmarks using Item Response Theory. Each benchmark is weighted by its fitted difficulty and discriminative slope, so doing well on hard, contamination-resistant evals (FrontierMath, ARC-AGI 2, Humanity's Last Exam) moves the score and saturated benchmarks contribute almost nothing. Imported per-model from Epoch AI's published index; we anchor it to the same min-max scale we use for every other benchmark so it's directly weightable in scenarios.

Share of nightly probe requests over the last 14 days that streamed a complete answer without an error, timeout or empty response. Needs at least five attempts before a figure is shown.