DeepSeek V3 (Thinking)

Crowdsourced pairwise human preference rankings of LLM responses. Higher Elo means more frequently preferred by users.

Real GitHub issues solved end-to-end. Verified subset is a 500-task human-validated slice of SWE-bench.

Median output speed in tokens per second over the last 14 days on the model maker's own API, measured nightly with a fixed ~800-token prompt and ~250-token answer. Models not yet probed fall back to a dated manual figure. Higher is faster.

Median time from request to first streamed token in milliseconds over the last 14 days on the model maker's own API, using the same nightly probe prompt. Lower is snappier; reasoning models are penalised here because they think before talking.

Contamination-controlled average across every task category in the latest rolling release (e.g. reasoning, coding, data analysis, research, browsing, long-horizon tasks). Questions are rotated with each release and graded against objective ground truth, removing the need for LLM-as-judge scoring. Only models with a score in every category get an average.

Rolling contamination-controlled data-analysis evaluation. Table comprehension, CSV / spreadsheet reasoning, SQL-style joins, and chart interpretation. Refreshed every six months with new tables and questions to minimise contamination.

Real-world refactoring and bug-fix tasks across multiple programming languages, scored by whether the model produces a passing patch in Aider's edit format. Tests practical coding ability beyond single-file generation; harder than HumanEval and not yet saturated.