whatmodel.sh / lab vs independent benchmark
How it's worked out
- Each dot is one model on one test where its maker published a score and someone independent (a leaderboard, a paper, another lab running it) measured it too. 611 pairs, 180 models, 103 tests. Gaps are in percentage points on each test's own scale, not the whatmodel score.
- Lab scores are 6.4% higher on average. For each matched pair, we calculate (lab score − independent score) ÷ independent score × 100, then average those percentages, weighting each pair equally. Lower lab scores and ties are included. For example, 60 vs 50 is 20% higher. The median percentage difference is 1.2%; the mean's 95% interval is 3.8%–9.5%, resampling whole models. 611 pairs have a nonzero independent baseline; zero baselines cannot have a relative percentage difference.
- Whenever the scores differ, the lab's is higher 63% of the time: 375 lab higher, 216 independent higher, and 20 exact ties. Every nonzero difference counts, using the scores before display rounding; ties are excluded from that percentage. Its 95% interval is 59%–68%, resampling whole models since one model's pairs aren't independent.
- Across all 611 pairs, including ties, the lab's number is higher 61% of the time (95% interval 57%–66%). The median gap is +0.8 points. The tables below use all pairs, including ties.
- It depends on the test. Standard tests with a fixed format (MMLU, WinoGrande, ARC Challenge, HumanEval) barely move; tests where the setup is part of the result, like the agent harness on SWE-bench Verified, sampling and reasoning budget on AIME, or few-shot prompting on GSM8K and MBPP, show the biggest gaps.
- Against neutral testers (leaderboards, Epoch AI, papers that don't release a model), the maker is higher 56% of 236 pairs; against rival labs' own release documents, 63% of 446. Where a rival and a neutral tester measured the same model on the same test (272 cases), the rival was lower 131 times and higher 130 times (median 0.0).
- Gaps aren't all inflation: settings often differ (reasoning effort, agent harness, few-shot prompts, test subset or version), and labs choose which tests to publish, so these pairs aren't a random sample. When a side has several results for the same model and test, the dot uses their median. Accuracy tests only; arena ratings have no maker-reported twin. Hover a dot for both sources.
By test
Tests with 13 or more pairs.
| Test | pairs | median gap | maker higher |
|---|---|---|---|
| SWE-bench Verified | 13 | +9.0 | 92% |
| MBPP | 13 | +9.0 | 77% |
| ARC Easy | 14 | +4.7 | 79% |
| AIME 2024 | 14 | +3.8 | 86% |
| GSM8K | 19 | +3.3 | 84% |
| AIME 2025 | 20 | +2.6 | 70% |
| MMLU-Pro | 29 | +2.0 | 69% |
| GPQA Diamond | 51 | +1.1 | 67% |
| IFEval (aggregation unspecified) | 13 | +1.0 | 69% |
| PIQA | 16 | +0.5 | 75% |
| MMLU | 20 | 0.0 | 50% |
| HumanEval | 34 | 0.0 | 47% |
| WinoGrande | 25 | -0.1 | 40% |
| ARC Challenge | 14 | -0.6 | 50% |
| τ²-bench Telecom | 18 | -1.3 | 39% |
By maker
Makers with 10 or more pairs. Small groups swing a lot: read these as rough.
| Maker | pairs | median gap | maker higher |
|---|---|---|---|
| Alibaba Cloud | 148 | +1.1 | 61% |
| Meta | 89 | +0.7 | 62% |
| EleutherAI | 48 | -0.1 | 40% |
| 48 | +1.6 | 60% | |
| DeepSeek | 46 | +0.8 | 70% |
| OpenAI | 36 | +0.5 | 58% |
| Z.ai | 27 | +0.9 | 67% |
| Mistral AI | 20 | +1.6 | 75% |
| BigScience | 20 | -11.9 | 10% |
| BigCode | 20 | +2.2 | 95% |
| Liquid AI | 15 | -0.7 | 47% |
| Ai2 | 14 | +8.8 | 93% |
| Microsoft | 14 | 0.0 | 50% |
| IBM | 13 | +1.7 | 77% |
| Anthropic | 12 | +1.6 | 75% |
| All makers | 611 | +0.8 | 61% |
All 611 pairs, biggest maker advantage first
| Model | Test | maker | independent | gap | Sources |
|---|---|---|---|---|---|
| Molmo 7B-D (0924) Ai2 | ChartQA | 84.1 | 27.0 | +57.1 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Qwen2-VL 7B Instruct Alibaba Cloud | ChartQA | 83.0 | 41.2 | +41.8 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Molmo 7B-D (0924) Ai2 | MathVista (testmini) | 51.6 | 12.3 | +39.3 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| gpt-oss-120b OpenAI | SWE-bench Verified | 62.4 | 31.0 | +31.4 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| Qwen3.5 397B-A17B Alibaba Cloud | BrowseComp | 69.0 | 40.5 | +28.5 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Molmo 7B-D (0924) Ai2 | VQAv2 (val) | 85.6 | 57.1 | +28.5 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Ornith 1.0 35B (35B-A3B) DeepReinforce (Ornith) | Terminal-Bench 2.1 | 64.2 | 35.96 | +28.24 | ornith-ai/Ornith-1.5-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| OPT-30B Meta | OpenBookQA | 57.2 | 29.2 | +28.0 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy — authors' own runs |
| DeepSeek R1 0528 DeepSeek | SWE-bench Verified | 57.6 | 29.6 | +28.0 | deepseek-ai/DeepSeek-R1-0528 model card · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| LLaMA 7B Meta | OpenBookQA | 57.2 | 31.4 | +25.8 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Ministral 3 14B Reasoning 2512 Mistral AI | GPQA Diamond | 71.2 | 46.0 | +25.2 | mistralai/Ministral-3-14B-Reasoning-2512 model card · JetBrains/Mellum2-12B-A2.5B-Thinking model card |
| SmolLM3 3B Hugging Face | GSM-Plus | 83.4 | 58.91 | +24.49 | HuggingFaceTB/SmolLM3-3B model card (Evaluation section) · LFM2 Technical Report — Liquid AI own SmolLM3 evaluation |
| OPT-1.3B Meta | SuperGLUE WSC (prompted) | 62.2 | 38.462 | +23.739 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| OPT-1.3B Meta | OpenBookQA | 46.8 | 23.4 | +23.4 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| Magistral Small 1.2 Mistral AI | GPQA Diamond | 70.07 | 47.601 | +22.469 | mistralai/Magistral-Small-2509 model card · Epoch AI — internal evaluations |
| Nemotron 3 Nano 4B NVIDIA | AIME 2025 | 78.5 | 56.3 | +22.2 | nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 model card · openbmb/MiniCPM5-2B model card |
| Qwen2-VL 72B Instruct Alibaba Cloud | ChartQA | 88.3 | 66.6 | +21.7 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| LLaMA 33B Meta | OpenBookQA | 58.6 | 37.5 | +21.1 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Molmo 7B-D (0924) Ai2 | MMMU (val) | 45.3 | 24.3 | +21.0 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Gemma 3 1B Google | GSM8K | 62.8 | 42.15 | +20.65 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| LLaMA 65B Meta | ARC Easy | 78.9 | 58.75 | +20.15 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| gpt-oss-120b OpenAI | AIME 2024 | 95.8 | 75.8 | +20.0 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| DeepSeek V3.1 DeepSeek | AIME 2025 | 88.4 | 68.585 | +19.815 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| Qwen3 Coder 30B-A3B Instruct Alibaba Cloud | SWE-bench Verified | 51.6 | 31.8 | +19.8 | Qwen/Qwen3-Coder-30B-A3B-Instruct model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| Ornith 1.0 35B (35B-A3B) DeepReinforce (Ornith) | SWE-bench Verified | 75.6 | 55.8 | +19.8 | ornith-ai/Ornith-1.5-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| GLM-4.7-Flash Z.ai | SWE-bench Verified | 59.2 | 39.7 | +19.5 | zai-org/GLM-4.7-Flash model card · SERA: Soft-Verified Efficient Repository Agents (arXiv:2601.20789v3) |
| Qwen3.5 2B Alibaba Cloud | MMLU-Pro | 66.5 | 47.05 | +19.45 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| LLaMA 13B Meta | OpenBookQA | 56.4 | 37.2 | +19.2 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Ornith 1.0 35B (35B-A3B) DeepReinforce (Ornith) | SWE-bench Multilingual | 69.3 | 51.67 | +17.63 | ornith-ai/Ornith-1.5-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| LFM2.5 2.6B Liquid AI | LiveCodeBench v6 (window unspecified) | 59.41 | 42.1 | +17.31 | LiquidAI/LFM2.5-2.6B model card · openbmb/MiniCPM5-2B model card |
| Gemma 3 1B Google | IFEval (aggregation unspecified) | 80.2 | 62.9 | +17.3 | Google DeepMind Gemma 3 270M IT model card · LiquidAI/LFM2-1.2B model card |
| Mistral 7B v0.1 (base) Mistral AI | GSM8K | 51.1 | 34.265 | +16.835 | Mistral 7B — authors' own runs · Gemma report — original evaluations |
| Claude Sonnet 4.5 Anthropic | τ²-bench Retail | 86.2 | 69.8 | +16.4 | τ-bench leaderboard: Anthropic's own submissions · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| Ornith 1.0 35B (35B-A3B) DeepReinforce (Ornith) | SWE-bench Pro (split unspecified) | 50.4 | 34.47 | +15.93 | ornith-ai/Ornith-1.5-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| SmolLM3 3B Hugging Face | GPQA Diamond | 41.7 | 26.31 | +15.39 | HuggingFaceTB/SmolLM3-3B model card (Evaluation section) · LFM2 Technical Report — Liquid AI own SmolLM3 evaluation |
| StarCoder2 3B BigCode | MBPP | 57.4 | 42.4 | +15.0 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| StarCoder2 15B BigCode | MBPP | 66.2 | 51.2 | +15.0 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| gpt-oss-120b OpenAI | AIME 2025 | 92.5 | 77.7 | +14.8 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| GPT-5.1 OpenAI | τ²-bench Telecom | 95.6 | 80.9 | +14.7 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| LLaMA 33B Meta | BoolQ | 83.1 | 68.41 | +14.69 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Ministral 8B (2410) Mistral AI | MATH (subset unspecified) | 54.5 | 40.0 | +14.5 | Mistral AI vendor-reported benchmark scores for its own models (Hugging Face model cards) · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| DeepSeek R1 Distill Qwen 14B DeepSeek | GPQA Diamond | 59.1 | 44.697 | +14.403 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| DeepSeek R1 0528 DeepSeek | AIME 2025 | 87.5 | 73.4 | +14.1 | deepseek-ai/DeepSeek-R1-0528 model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Qwen3 Coder 30B-A3B Instruct Alibaba Cloud | SWE-bench Multilingual | 34.7 | 20.67 | +14.03 | Qwen/Qwen3-Coder-30B-A3B-Instruct model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | τ²-bench Airline | 50.0 | 36.0 | +14.0 | Qwen/Qwen3-Next-80B-A3B-Instruct model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| Qwen3.5 2B Alibaba Cloud | LongBench v2 | 38.7 | 24.9 | +13.8 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| Qwen3-VL 32B Thinking Alibaba Cloud | τ²-bench Telecom | 46.9 | 33.3 | +13.6 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| Llama 3.1 Tulu 3 8B Ai2 | BIG-Bench Hard | 69.0 | 55.5 | +13.5 | 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models · microsoft/Phi-4-mini-instruct model card |
| StarCoderBase 3B BigCode | MBPP | 42.6 | 29.4 | +13.2 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| StarCoderBase 15.5B BigCode | MBPP | 50.6 | 37.4 | +13.2 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| StarCoder2 7B BigCode | GSM8K | 40.4 | 27.2 | +13.2 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| CodeGemma 2B (v1.0) Google | MBPP | 43.6 | 30.4 | +13.2 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Nemotron Nano 9B v2 NVIDIA | AIME 2025 | 72.1 | 58.9 | +13.2 | nvidia/NVIDIA-Nemotron-Nano-9B-v2 model card · Olmo 3 |
| Ministral 3 3B Mistral AI | MATH (subset unspecified) | 83.0 | 70.2 | +12.8 | mistralai/Ministral-3-3B-Instruct-2512 model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| Qwen3.5 35B-A3B Alibaba Cloud | SWE-bench Multilingual | 60.3 | 47.67 | +12.63 | Qwen/Qwen3.6-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| StarCoderBase 7B BigCode | MBPP | 47.4 | 34.8 | +12.6 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Magistral Small 1.0 (2506) Mistral AI | GPQA Diamond | 68.18 | 56.061 | +12.119 | mistralai/Magistral-Small-2509 model card · Epoch AI — internal evaluations |
| Claude Sonnet 4.5 Anthropic | τ²-bench Airline | 70.0 | 58.0 | +12.0 | τ-bench leaderboard: Anthropic's own submissions · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| Qwen3 4B Alibaba Cloud | MMLU-Pro | 64.2 | 52.31 | +11.89 | Qwen/Qwen3-4B-Thinking-2507 model card · LFM2 Technical Report — Liquid AI own SmolLM3 evaluation |
| Molmo 72B (0924) Ai2 | ChartQA | 87.3 | 75.6 | +11.7 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Qwen3.5 27B Alibaba Cloud | SWE-bench Multilingual | 69.3 | 57.67 | +11.63 | Qwen/Qwen3.6-27B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| Qwen3.6 27B Alibaba Cloud | OmniDocBench 1.5 overall (higher is better) | 89.4 | 77.8 | +11.6 | Qwen/Qwen3.8-27B model card · meta-models/Muse-Glimmer-30B model card (speed and benchmark tables) |
| EXAONE 4.5 33B LG AI Research | MMLU-Pro | 83.3 | 71.96 | +11.34 | LG AI Research vendor-reported benchmark scores for its own EXAONE models (Hugging Face model cards) · SqueezeBits — original EXAONE 4.5 BF16 evaluations |
| Molmo 72B (0924) Ai2 | VQAv2 (val) | 86.5 | 75.2 | +11.3 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| StarCoderBase 7B BigCode | GSM8K | 14.9 | 3.8 | +11.1 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Qwen3 8B Alibaba Cloud | MMLU-Pro | 74.6 | 63.5 | +11.1 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3.5 2B Alibaba Cloud | CountBench | 91.4 | 80.5 | +10.9 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Granite 4.2 3B IBM | LiveCodeBench v6 (window unspecified) | 69.71 | 58.9 | +10.81 | ibm-granite/granite-4.2-30b model card · openbmb/MiniCPM5-2B model card |
| DeepSeek R1 0528 DeepSeek | LiveCodeBench v6 (Aug 2024–May 2025) | 73.3 | 62.5 | +10.8 | deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Qwen3-VL 2B Instruct Alibaba Cloud | OCRBench | 85.8 | 75.1 | +10.7 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Qwen3.5 35B-A3B Alibaba Cloud | SWE-bench Verified | 69.2 | 58.6 | +10.6 | Qwen/Qwen3.5-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| OPT-1.3B Meta | ARC Challenge | 33.9 | 23.31 | +10.59 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Qwen3 1.7B Alibaba Cloud | MMLU-Pro | 56.5 | 45.94 | +10.56 | Qwen/Qwen3.5-2B model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| Qwen3 1.7B Alibaba Cloud | MMMLU | 57.0 | 46.51 | +10.49 | Qwen/Qwen3.5-2B model card · LiquidAI/LFM2-1.2B model card |
| Qwen3.5 2B Alibaba Cloud | MMStar | 71.7 | 61.4 | +10.3 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| gpt-oss-20b OpenAI | GPQA Diamond | 71.5 | 61.2 | +10.3 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Epoch AI — internal evaluations |
| GPT-Neo 1.3B EleutherAI | HellaSwag | 48.9 | 38.65 | +10.25 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models — authors' own runs |
| Qwen3.6 35B-A3B Alibaba Cloud | SWE-bench Multilingual | 67.2 | 57.0 | +10.2 | Qwen/Qwen3.6-27B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| LFM2.5 2.6B Liquid AI | AIME 2025 | 51.87 | 41.9 | +9.97 | LiquidAI/LFM2.5-2.6B model card · openbmb/MiniCPM5-2B model card |
| Gemma 4 E2B Google | MMLU-Pro | 60.0 | 50.05 | +9.95 | Gemma Team — Gemma 4 Technical Report · openbmb/MiniCPM5-2B model card |
| Qwen3-VL 2B Instruct Alibaba Cloud | ChartQA | 79.1 | 69.3 | +9.8 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Nemotron Nano 9B v2 NVIDIA | AIME 2024 | 81.9 | 72.1 | +9.8 | NVIDIA — Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models · Olmo 3 |
| DeepSeek R1 0528 DeepSeek | MMLU-Pro | 85.0 | 75.3 | +9.7 | deepseek-ai/DeepSeek-R1-0528 model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| LLaMA 13B Meta | BoolQ | 78.0 | 68.502 | +9.498 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | τ-bench Retail | 67.8 | 58.7 | +9.1 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Gemma 4 26B-A4B Google | GPQA Diamond | 82.3 | 73.232 | +9.068 | Gemma Team — Gemma 4 Technical Report · Epoch AI — internal evaluations |
| StarCoder2 7B BigCode | MBPP | 54.4 | 45.4 | +9.0 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Qwen3.6 35B-A3B Alibaba Cloud | SWE-bench Verified | 73.4 | 64.4 | +9.0 | Qwen/Qwen3.6-35B-A3B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| Gemma 4 31B Google | Terminal-Bench Hard | 36.0 | 27.08 | +8.92 | Gemma Team — Gemma 4 Technical Report · NVIDIA ModelOpt quantized-checkpoint cards: baseline (unquantized) measurements of third-party models |
| Devstral Small 2 24B Mistral AI | SWE-bench Verified | 68.0 | 59.1 | +8.9 | mistralai/Devstral-Small-2-24B-Instruct-2512 model card · SERA: Soft-Verified Efficient Repository Agents (arXiv:2601.20789v3) |
| LLaMA 13B Meta | ARC Easy | 76.1 | 67.21 | +8.89 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Gemma 3 4B Google | MMLU-Pro | 43.6 | 34.76 | +8.84 | Google DeepMind Gemma 3 270M IT model card · LFM2 Technical Report — Liquid AI own SmolLM3 evaluation |
| LLaMA 33B Meta | Natural Questions (closed book) | 32.9 | 24.1 | +8.8 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Claude Sonnet 5.5 Anthropic | Terminal-Bench 4.0 | 70.6 | 61.82 | +8.78 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| GLM-5.1 Z.ai | BrowseComp | 68.0 | 59.4 | +8.6 | zai-org/GLM-5.1 model card · NVIDIA — Nemotron 3 Ultra technical report |
| Gemma 4 31B Google | GPQA Diamond | 84.3 | 75.758 | +8.542 | Gemma Team — Gemma 4 Technical Report · Epoch AI — internal evaluations |
| LFM2.5 8B-A1B Liquid AI | Multi-IF (aggregation unspecified) | 79.93 | 71.4 | +8.53 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| OPT-30B Meta | ARC Challenge | 43.1 | 34.642 | +8.458 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| LLaMA 33B Meta | ARC Challenge | 57.8 | 49.464 | +8.336 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| DeepSeek R1 0528 DeepSeek | AIME 2024 | 91.4 | 83.2 | +8.2 | deepseek-ai/DeepSeek-V3.1 model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| InternVL2-4B OpenGVLab | TextVQA | 74.4 | 66.2 | +8.2 | OpenGVLab InternVL2 model cards (InternVL2-4B and InternVL2-26B), Evaluation tables · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table |
| Qwen3.5 2B Alibaba Cloud | AI2D test | 83.3 | 75.2 | +8.1 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| EXAONE 4.0 32B LG AI Research | τ²-bench Retail | 67.5 | 59.5 | +8.0 | LG AI Research vendor-reported benchmark scores for its own EXAONE models (Hugging Face model cards) · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| Nemotron Nano 9B v2 NVIDIA | GPQA (subset unspecified) | 64.0 | 56.2 | +7.8 | nvidia/NVIDIA-Nemotron-Nano-9B-v2 model card · Olmo 3 |
| Qwen3-VL 2B Instruct Alibaba Cloud | MMStar | 58.3 | 50.6 | +7.7 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Gemma 3 1B Google | MATH (subset unspecified) | 48.0 | 40.3 | +7.7 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GLM-4.5-Air Z.ai | MMLU-Pro | 81.4 | 73.9 | +7.5 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Qwen3.5 2B Alibaba Cloud | MMBench-EN dev v1.1 | 83.3 | 76.0 | +7.3 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| GPT-Neo 2.7B EleutherAI | HellaSwag | 55.8 | 48.555 | +7.245 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| gpt-oss-120b OpenAI | GPQA Diamond | 80.1 | 72.879 | +7.221 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Epoch AI — internal evaluations |
| Gemini 2.5 Flash Google | CharXiv (reasoning questions) | 63.7 | 56.5 | +7.2 | Google DeepMind Gemini model page and model cards (lab-reported scores) · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| Gemma 3 27B Google | MATH (subset unspecified) | 89.0 | 81.8 | +7.2 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| DeepSeek R1 Distill Llama 8B DeepSeek | AIME 2024 | 50.4 | 43.3 | +7.1 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| Qwen3.5 2B Alibaba Cloud | GPQA (subset unspecified) | 51.6 | 44.6 | +7.0 | Qwen/Qwen3.5-2B model card · XHToken/Spark-X2.5-4B model card |
| Phi-4-reasoning-plus 14B Microsoft | AIME 2025 | 78.0 | 71.2 | +6.8 | microsoft/Phi-4-reasoning model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3.5 9B Alibaba Cloud | BFCL V4 (December 2025 archive) | 66.1 | 59.315 | +6.785 | Qwen/Qwen3.5-9B model card · JetBrains/Mellum2-12B-A2.5B-Instruct model card |
| Pythia 1B EleutherAI | ARC Easy | 56.9 | 50.2 | +6.7 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Pythia 12B EleutherAI | ARC Easy | 70.2 | 63.721 | +6.479 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LLaMA 7B Meta | ARC Easy | 74.05 | 67.641 | +6.409 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| EXAONE 4.0 32B LG AI Research | τ²-bench Airline | 52.0 | 45.6 | +6.4 | LG AI Research vendor-reported benchmark scores for its own EXAONE models (Hugging Face model cards) · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| Molmo 72B (0924) Ai2 | MathVista (testmini) | 58.6 | 52.2 | +6.4 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Qwen3 1.7B Alibaba Cloud | GPQA (subset unspecified) | 34.0 | 27.72 | +6.28 | Qwen/Qwen3.5-2B model card · LiquidAI/LFM2-1.2B model card |
| Qwen3-VL 2B Instruct Alibaba Cloud | MMLU-Pro | 49.0 | 42.8 | +6.2 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Mistral 7B v0.1 (base) Mistral AI | MBPP | 48.85 | 42.7 | +6.15 | Mistral 7B — authors' own runs · Gemma report — original evaluations |
| StarCoderBase 7B BigCode | HumanEval | 30.5 | 24.4 | +6.1 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| Qwen2-VL 7B Instruct Alibaba Cloud | MMMU (val) | 54.1 | 48.1 | +6.0 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Qwen3.5 2B Alibaba Cloud | IFEval (aggregation unspecified) | 78.6 | 72.7 | +5.9 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| WizardCoder Python 34B V1.0 WizardLM | HumanEval | 73.2 | 67.4 | +5.8 | WizardLM team vendor-reported scores for its own WizardCoder models (Hugging Face card) · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| GLM-5 Z.ai | SWE-bench Verified | 77.8 | 72.078 | +5.722 | zai-org/GLM-5 model card · Epoch AI — internal evaluations |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | LiveCodeBench v6 (Feb–May 2025) | 66.0 | 60.3 | +5.7 | Qwen/Qwen3-30B-A3B-Thinking-2507 model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Qwen3 32B Alibaba Cloud | τ-bench Retail | 46.45 | 40.9 | +5.55 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Pythia 6.9B EleutherAI | ARC Easy | 67.0 | 61.485 | +5.515 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Gemma 4 E4B Google | GPQA Diamond | 58.6 | 53.1 | +5.5 | Gemma Team — Gemma 4 Technical Report · JetBrains/Mellum2.1-12B-A2.5B-Thinking model card |
| LFM2.5 8B-A1B Liquid AI | IFBench (aggregation unspecified) | 56.47 | 51.0 | +5.47 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| Gemma 2B (base) Google | TriviaQA | 53.2 | 47.77 | +5.43 | Gemma report — original evaluations · Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results) |
| Granite 4.0 H 1B IBM | MMLU-Pro | 32.86 | 27.64 | +5.22 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| Qwen3.5 2B Alibaba Cloud | RealWorldQA | 74.5 | 69.3 | +5.2 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Qwen3.5 27B Alibaba Cloud | SWE-bench Verified | 73.7 | 68.6 | +5.1 | Qwen/Qwen3.5-27B model card · Kwaipilot/KAT-Coder-V2.5-Dev model card |
| InternVL2-4B OpenGVLab | MathVista (testmini) | 58.6 | 53.7 | +4.9 | OpenGVLab InternVL2 model cards (InternVL2-4B and InternVL2-26B), Evaluation tables · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table |
| GLM-4.5-Air Z.ai | AIME 2024 | 89.4 | 84.6 | +4.8 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Gemma 3 12B Google | MMLU-Pro | 60.6 | 55.8 | +4.8 | Google DeepMind Gemma 3 270M IT model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| Gemma 3 27B Google | GSM8K | 95.9 | 91.1 | +4.8 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GPT-J 6B EleutherAI | ARC Easy | 67.0 | 62.247 | +4.753 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Gemma 3 12B Google | GSM8K | 94.4 | 89.7 | +4.7 | Google DeepMind Gemma 3 270M IT model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| LLaMA 33B Meta | ARC Easy | 80.0 | 75.337 | +4.663 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| DeepSeek R1 Distill Qwen 32B DeepSeek | MATH-500 | 94.3 | 89.7 | +4.6 | deepseek-ai/DeepSeek-R1 model card · open-thoughts/OpenThinker2-32B model card |
| Qwen2-VL 7B Instruct Alibaba Cloud | MathVista (testmini) | 58.2 | 53.7 | +4.5 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Gemma 3 4B Google | GPQA Diamond | 30.8 | 26.371 | +4.429 | Google DeepMind Gemma 3 270M IT model card · Epoch AI — internal evaluations |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | τ²-bench Telecom | 26.3 | 21.93 | +4.37 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · LiquidAI/LFM2.5-8B-A1B model card |
| gpt-oss-20b OpenAI | AIME 2025 | 91.7 | 87.35 | +4.35 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen2 7B Instruct Alibaba Cloud | MATH (subset unspecified) | 52.9 | 48.6 | +4.3 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · InternLM GitHub repository model cards (Shanghai AI Laboratory) |
| CodeGemma 2B (v1.0) Google | HumanEval | 31.1 | 26.8 | +4.3 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| gpt-oss-120b OpenAI | Humanity's Last Exam | 14.9 | 10.6 | +4.3 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| DeepSeek R1 Distill Llama 70B DeepSeek | GPQA Diamond | 65.2 | 60.972 | +4.228 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| GLM-5.1 Z.ai | Terminal-Bench 2.1 | 63.5 | 59.3 | +4.2 | zai-org/GLM-5.2 model card · NVIDIA — Nemotron 3 Ultra technical report |
| Qwen2-VL 72B Instruct Alibaba Cloud | MMMU (val) | 64.5 | 60.3 | +4.2 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| gpt-oss-20b OpenAI | AIME 2024 | 92.1 | 88.0 | +4.1 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | τ²-bench Retail | 74.6 | 70.5 | +4.1 | Qwen/Qwen3-Next-80B-A3B-Instruct model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| Phi-4-reasoning-plus 14B Microsoft | AIME 2024 | 81.3 | 77.2 | +4.1 | microsoft/Phi-4-reasoning model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| DeepSeek R1 0528 DeepSeek | GPQA Diamond | 81.0 | 76.913 | +4.087 | deepseek-ai/DeepSeek-R1-0528 model card · Epoch AI — internal evaluations |
| Mistral Small 3.1 24B (2503) Mistral AI | GPQA Diamond | 45.96 | 41.919 | +4.041 | mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card · Epoch AI — internal evaluations |
| Qwen3.5 9B Alibaba Cloud | IFBench (aggregation unspecified) | 64.5 | 60.485 | +4.015 | Qwen/Qwen3.5-9B model card · XHToken/Spark-X2.5-4B model card |
| OPT-1.3B Meta | Story Cloze Test | 74.7 | 70.78 | +3.92 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Gemma 3 27B Google | BIG-Bench Hard | 87.6 | 83.7 | +3.9 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| DeepSeek V3.1 DeepSeek | HMMT 2025 (edition unspecified) | 84.2 | 80.4 | +3.8 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Nemotron 3 Nano 4B NVIDIA | MATH-500 | 95.4 | 91.6 | +3.8 | nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 397B-A17B Alibaba Cloud | MultiChallenge | 67.6 | 63.9 | +3.7 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | AIME 2025 | 85.0 | 81.3 | +3.7 | Qwen/Qwen3-30B-A3B-Thinking-2507 model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | τ²-bench Retail | 71.9 | 68.2 | +3.7 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| InternVL2-4B OpenGVLab | MMMU (val) | 47.9 | 44.22 | +3.68 | OpenGVLab InternVL2 model cards (InternVL2-4B and InternVL2-26B), Evaluation tables · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table |
| GPT-NeoX-20B EleutherAI | ARC Easy | 72.3 | 68.687 | +3.613 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| gpt-oss-20b OpenAI | MMLU | 85.3 | 81.7 | +3.6 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| StarCoderBase 3B BigCode | HumanEval | 21.3 | 17.7 | +3.6 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| Inkling (975B-A41B) Thinking Machines Lab | SimpleQA Verified | 43.9 | 40.3 | +3.6 | thinkingmachines/Inkling-Small model card · Epoch AI — internal evaluations |
| Gemma 3 12B Google | GPQA Diamond | 40.9 | 37.329 | +3.571 | Google DeepMind Gemma 3 270M IT model card · Epoch AI — internal evaluations |
| Olmo 3 7B Think Ai2 | IFEval (aggregation unspecified) | 88.2 | 84.7 | +3.5 | Olmo 3 · JetBrains/Mellum2-12B-A2.5B-Thinking model card |
| SmolLM2 1.7B Instruct Hugging Face | GSM8K | 48.8 | 45.3 | +3.5 | SmolLM2 paper (Hugging Face) — authors' own instruct-model evaluations · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GLM-4.5 Z.ai | AIME 2024 | 91.0 | 87.55 | +3.45 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Claude 1.3 Anthropic | GSM8K | 85.2 | 81.8 | +3.4 | Anthropic system cards and model announcements (lab-reported scores) · Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance — authors' own runs |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | τ²-bench Telecom | 32.5 | 29.1 | +3.4 | Qwen/Qwen3-Next-80B-A3B-Instruct model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| GLM-5.1 Z.ai | Humanity's Last Exam | 31.0 | 27.6 | +3.4 | zai-org/GLM-5.1 model card · NVIDIA — Nemotron 3 Ultra technical report |
| Granite 4.0 H Small (32B-A9B) IBM | GSM8K | 87.27 | 83.9 | +3.37 | ibm-granite/granite-4.0-h-small model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| LLaMA 7B Meta | ARC Challenge | 44.75 | 41.38 | +3.37 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Flan-T5 XXL Google | BIG-Bench Hard | 45.3 | 41.95 | +3.35 | Scaling Instruction-Finetuned Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| Qwen3 1.7B Alibaba Cloud | AIME 2025 | 36.8 | 33.485 | +3.315 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · LiquidAI/LFM2.5-1.2B-Thinking model card |
| Qwen3.5 397B-A17B Alibaba Cloud | HMMT February 2026 | 87.9 | 84.6 | +3.3 | Qwen/Qwen3.6-27B model card · LGAI-EXAONE/K-EXAONE-2.0-750B-A37B model card |
| LLaMA 13B Meta | GSM8K | 17.8 | 14.5 | +3.3 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources — authors' own runs |
| Llama 3.1 Tulu 3 8B Ai2 | GSM8K | 87.6 | 84.3 | +3.3 | 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models · microsoft/Phi-4-mini-instruct model card |
| LFM2.5 2.6B Liquid AI | Multi-IF (aggregation unspecified) | 80.07 | 76.8 | +3.27 | LiquidAI/LFM2.5-2.6B model card · openbmb/MiniCPM5-2B model card |
| OPT-175B Meta | Story Cloze Test | 82.9 | 79.66 | +3.24 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| LLaMA 33B Meta | WinoGrande | 76.0 | 72.77 | +3.23 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| DeepSeek V3.1 DeepSeek | MMLU-Redux | 93.7 | 90.5 | +3.2 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Llama 3.1 Tulu 3 8B Ai2 | MMLU | 68.2 | 65.0 | +3.2 | 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models · microsoft/Phi-4-mini-instruct model card |
| CodeGemma 7B (v1.0) Google | MBPP | 56.2 | 53.0 | +3.2 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Flan-T5 Large Google | MMLU | 45.1 | 41.9 | +3.2 | Scaling Instruction-Finetuned Language Models — authors' own runs · Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In — authors' own runs |
| LLaMA 7B Meta | Social IQA | 48.9 | 45.716 | +3.184 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| OPT-30B Meta | Story Cloze Test | 80.3 | 77.14 | +3.16 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Qwen3 1.7B Alibaba Cloud | IFBench (aggregation unspecified) | 26.7 | 23.575 | +3.125 | Qwen/Qwen3.5-2B model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| GLM-4.5 Z.ai | MMLU-Pro | 84.6 | 81.5 | +3.1 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| LLaMA 7B Meta | WinoGrande | 70.0 | 66.93 | +3.07 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| WizardCoder 33B V1.1 WizardLM | HumanEval | 79.9 | 76.85 | +3.05 | WizardLM team vendor-reported scores for its own WizardCoder models (Hugging Face card) · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Granite 4.0 1B IBM | GSM8K | 76.35 | 73.42 | +2.93 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| Claude Opus 4.6 Anthropic | GPQA Diamond | 91.3 | 88.384 | +2.916 | Anthropic system cards and model announcements (lab-reported scores) · Epoch AI — internal evaluations |
| Qwen3 235B-A22B (Apr 2025, hybrid) Alibaba Cloud | BFCL v3 | 70.8 | 67.94 | +2.86 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · Berkeley Function Calling Leaderboard result archive |
| LLaMA 13B Meta | ARC Challenge | 49.55 | 46.725 | +2.825 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Qwen2 7B Instruct Alibaba Cloud | GSM8K | 85.7 | 82.9 | +2.8 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · InternLM GitHub repository model cards (Shanghai AI Laboratory) |
| Nemotron Nano 9B v2 NVIDIA | IFBench (aggregation unspecified) | 37.4 | 34.6 | +2.8 | NVIDIA — Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models · Olmo 3 |
| Qwen3-VL 2B Instruct Alibaba Cloud | CharXiv descriptive questions | 62.3 | 59.5 | +2.8 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| GPT-5 mini OpenAI | τ²-bench Telecom | 74.1 | 71.3 | +2.8 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages (arXiv:2606.28715v2) |
| Qwen3.5 397B-A17B Alibaba Cloud | SWE-bench Verified | 76.3 | 73.6 | +2.7 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| DeepSeek R1 0528 Qwen3 8B DeepSeek | AIME 2024 | 86.0 | 83.3 | +2.7 | deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3.5 4B Alibaba Cloud | IFEval (aggregation unspecified) | 89.8 | 87.1 | +2.7 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 4B Alibaba Cloud | LongBench v2 | 50.0 | 47.3 | +2.7 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| InternVL2-4B OpenGVLab | ChartQA | 81.5 | 78.8 | +2.7 | OpenGVLab InternVL2 model cards (InternVL2-4B and InternVL2-26B), Evaluation tables · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table |
| GPT-4o mini (2024-07-18) OpenAI | CharXiv (reasoning questions) | 36.8 | 34.1 | +2.7 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · CharXiv leaderboard (princeton-nlp/CharXiv README) |
| Qwen3 4B Instruct 2507 Alibaba Cloud | MMLU-Pro | 69.6 | 66.9 | +2.7 | Qwen/Qwen3-4B-Instruct-2507 model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| LLaMA 13B Meta | WinoGrande | 72.8 | 70.128 | +2.672 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | GPQA Diamond | 73.4 | 70.738 | +2.662 | Qwen/Qwen3.5-4B model card · Epoch AI — internal evaluations |
| OPT-30B Meta | SuperGLUE WSC (prompted) | 62.2 | 59.615 | +2.585 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| DeepSeek V3 (Dec '24) DeepSeek | GPQA Diamond | 59.1 | 56.534 | +2.566 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| GPT-5.6 Sol OpenAI | FrontierMath tier 4 v2 | 83.0 | 80.488 | +2.512 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| gpt-oss-120b OpenAI | Aider polyglot (225 exercises, two attempts) | 44.4 | 41.9 | +2.5 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Aider polyglot benchmark runs |
| Phi-1.5 (1.3B) Microsoft | ARC Easy | 75.6 | 73.148 | +2.452 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3.7 Plus Alibaba Cloud | GPQA Diamond | 90.3 | 87.879 | +2.421 | Qwen/Qwen3.8-27B model card · Epoch AI — internal evaluations |
| Olmo 3 7B Instruct Ai2 | IFEval (aggregation unspecified) | 85.6 | 83.2 | +2.4 | Olmo 3 · JetBrains/Mellum2-12B-A2.5B-Instruct model card |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | MMLU-Redux | 93.8 | 91.4 | +2.4 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Granite 4.0 1B IBM | BFCL v3 | 54.82 | 52.43 | +2.39 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| GLM-4.7 Z.ai | GPQA Diamond | 85.7 | 83.333 | +2.367 | GLM-5 Team (Zhipu AI) — GLM-5 technical report · Epoch AI — internal evaluations |
| Qwen3.5 9B Alibaba Cloud | GPQA Diamond | 81.7 | 79.389 | +2.311 | Qwen/Qwen3.5-9B model card · Epoch AI — internal evaluations |
| DeepSeek R1 Distill Qwen 7B DeepSeek | MATH-500 | 92.8 | 90.5 | +2.3 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| Qwen2-VL 72B Instruct Alibaba Cloud | MathVista (testmini) | 70.5 | 68.2 | +2.3 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Kimi K2 Instruct (0711) Moonshot AI | τ²-bench Airline | 56.5 | 54.2 | +2.3 | τ²-bench leaderboard: Moonshot AI's own submission · Meituan LongCat-Flash-Chat model card (evaluation table) |
| LLaMA 7B Meta | MMLU | 35.1 | 32.8 | +2.3 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| Flan-T5 XL Google | MMLU | 52.4 | 50.2 | +2.2 | Scaling Instruction-Finetuned Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| DeepSeek R1 Distill Llama 8B DeepSeek | MATH-500 | 89.1 | 86.9 | +2.2 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| Qwen3.6 35B-A3B Alibaba Cloud | GPQA Diamond | 86.0 | 83.838 | +2.162 | Qwen/Qwen3.6-27B model card · Epoch AI — internal evaluations |
| Granite 4.2 3B IBM | MMLU-Pro | 67.84 | 65.8 | +2.04 | ibm-granite/granite-4.2-30b model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 35B-A3B Alibaba Cloud | IMO-AnswerBench | 76.8 | 74.8 | +2.0 | Qwen/Qwen3.6-35B-A3B model card · NVIDIA — Nemotron-Cascade 2 technical report |
| GLM-4.5 Z.ai | MATH-500 | 98.2 | 96.2 | +2.0 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| BLOOM 7.1B BigScience | COPA | 69.0 | 67.0 | +2.0 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items (arXiv:2410.14897v1) |
| OPT-30B Meta | COPA | 84.0 | 82.0 | +2.0 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| OPT-175B Meta | ARC Challenge | 44.1 | 42.12 | +1.98 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | AIME 2025 | 70.3 | 68.33 | +1.97 | Qwen/Qwen3-235B-A22B-Instruct-2507 model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| LLaMA 65B Meta | PIQA | 82.8 | 80.85 | +1.95 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| Qwen3 235B-A22B (Apr 2025, hybrid) Alibaba Cloud | τ²-bench Telecom | 23.25 | 21.3 | +1.95 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · Moonshot AI Kimi-K2-Instruct model card (instruction-model evaluation table) |
| Qwen3.6 27B Alibaba Cloud | GPQA Diamond | 87.8 | 85.859 | +1.941 | Qwen/Qwen3.6-27B model card · Epoch AI — internal evaluations |
| GLM-5.1 Z.ai | Humanity's Last Exam with tools | 52.3 | 50.4 | +1.9 | zai-org/GLM-5.2 model card · NVIDIA — Nemotron 3 Ultra technical report |
| LLaMA 33B Meta | HellaSwag | 82.8 | 80.925 | +1.875 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Mistral 7B v0.1 (base) Mistral AI | PIQA | 82.6 | 80.794 | +1.806 | Mistral 7B — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| DeepSeek R1 0528 DeepSeek | Humanity's Last Exam | 17.7 | 15.9 | +1.8 | deepseek-ai/DeepSeek-R1-0528 model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Qwen3 8B Alibaba Cloud | BFCL v3 | 68.1 | 66.34 | +1.76 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Berkeley Function Calling Leaderboard result archive |
| Granite 4.0 H Small (32B-A9B) IBM | MMLU | 78.44 | 76.7 | +1.74 | ibm-granite/granite-4.0-h-small model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| DeepSeek R1 Distill Llama 8B DeepSeek | GPQA Diamond | 49.0 | 47.3 | +1.7 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| Qwen3.5 9B Alibaba Cloud | IFEval (aggregation unspecified) | 91.5 | 89.8 | +1.7 | Qwen/Qwen3.5-9B model card · JetBrains/Mellum2-12B-A2.5B-Instruct model card |
| Claude 2.0 Anthropic | HumanEval | 71.2 | 69.5 | +1.7 | Anthropic system cards and model announcements (lab-reported scores) · EvalPlus / EvalPerf published evaluation runs |
| GPT-4o mini (2024-07-18) OpenAI | CharXiv descriptive questions | 76.6 | 74.92 | +1.68 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · CharXiv leaderboard (princeton-nlp/CharXiv README) |
| LLaMA 7B Meta | BoolQ | 75.8 | 74.123 | +1.677 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Pythia 1B EleutherAI | PIQA | 70.7 | 69.1 | +1.6 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| InternVL2-4B OpenGVLab | AI2D test | 78.9 | 77.3 | +1.6 | OpenGVLab InternVL2 model cards (InternVL2-4B and InternVL2-26B), Evaluation tables · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table |
| T0 11B BigScience | WinoGrande | 60.46 | 58.9 | +1.56 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation — authors' own runs |
| Claude Opus 5.5 Anthropic | Terminal-Bench 4.0 | 66.4 | 64.85 | +1.55 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| Qwen3 1.7B Alibaba Cloud | BFCL v3 | 56.6 | 55.055 | +1.545 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Berkeley Function Calling Leaderboard result archive |
| Qwen3 8B Alibaba Cloud | AIME 2025 | 67.3 | 65.8 | +1.5 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Inkling Small (266B-A12B) Thinking Machines Lab | SimpleQA Verified | 20.6 | 19.1 | +1.5 | thinkingmachines/Inkling-Small model card · Epoch AI — internal evaluations |
| OPT-30B Meta | WinoGrande | 69.7 | 68.2 | +1.5 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Molmo 72B (0924) Ai2 | MMMU (val) | 54.1 | 52.7 | +1.4 | Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · Pixtral 12B (Mistral AI, arXiv 2410.07073v2) |
| Claude Instant 1.2 Anthropic | HumanEval | 58.7 | 57.3 | +1.4 | Anthropic system cards and model announcements (lab-reported scores) · EvalPlus / EvalPerf published evaluation runs |
| Ministral 3 8B Mistral AI | MATH (subset unspecified) | 87.6 | 86.2 | +1.4 | mistralai/Ministral-3-8B-Instruct-2512 model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| LLaMA 33B Meta | TriviaQA | 69.9 | 68.5 | +1.4 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Qwen3.5 397B-A17B Alibaba Cloud | GPQA (subset unspecified) | 88.4 | 87.1 | +1.3 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| DeepSeek V3.1 DeepSeek | LiveCodeBench (version unspecified) | 74.8 | 73.5 | +1.3 | DeepSeek-V3.1-Terminus model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Code Llama 34B Python Meta | HumanEval | 53.7 | 52.4 | +1.3 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Phi-1.5 (1.3B) Microsoft | BoolQ | 75.8 | 74.526 | +1.274 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3 235B-A22B (Apr 2025, hybrid) Alibaba Cloud | τ²-bench Retail | 52.65 | 51.4 | +1.25 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| LLaMA 65B Meta | ARC Challenge | 56.0 | 54.78 | +1.22 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| OPT-175B Meta | WinoGrande | 73.7 | 72.5 | +1.2 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| Phi-4-reasoning-plus 14B Microsoft | GPQA Diamond | 69.1 | 67.9 | +1.2 | microsoft/Phi-4-reasoning model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3.5 4B Alibaba Cloud | MMLU-Pro | 79.1 | 78.0 | +1.1 | Qwen/Qwen3.5-4B model card · openbmb/MiniCPM5-2B model card |
| DeepSeek R1 (Jan '25) DeepSeek | AIME 2024 | 79.8 | 78.7 | +1.1 | deepseek-ai/DeepSeek-R1-0528 model card · Microsoft — Phi-4-reasoning Technical Report |
| Gemma 4 E4B Google | MMLU-Pro | 69.4 | 68.3 | +1.1 | Gemma Team — Gemma 4 Technical Report · openbmb/MiniCPM5-2B model card |
| LLaMA 7B Meta | PIQA | 79.25 | 78.183 | +1.067 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| DeepSeek V4.1 Flash DeepSeek | GPQA Diamond | 90.9 | 89.836 | +1.064 | deepseek-ai/DeepSeek-V4.1-Flash model card · Epoch AI — internal evaluations |
| LFM2.5 8B-A1B Liquid AI | IFEval (aggregation unspecified) | 91.84 | 90.8 | +1.04 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 397B-A17B Alibaba Cloud | LiveCodeBench v6 (Feb–May 2025) | 83.6 | 82.6 | +1.0 | Qwen/Qwen3.5-397B-A17B model card · A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1 |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | τ-bench Airline | 48.0 | 47.0 | +1.0 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| GLM-4.5 Z.ai | AIME 2025 | 85.4 | 84.4 | +1.0 | zai-org/GLM-4.6 model card · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| GPT-4o mini (2024-07-18) OpenAI | HumanEval (simple-evals) | 87.2 | 86.2 | +1.0 | OpenAI simple-evals README benchmark results (OpenAI's own models) · Microsoft Research — Phi-4 Technical Report |
| LLaMA 13B Meta | HumanEval | 15.8 | 14.8 | +1.0 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| Inkling Small (266B-A12B) Thinking Machines Lab | GPQA Diamond | 89.5 | 88.51 | +0.99 | thinkingmachines/Inkling-Small model card · Epoch AI — internal evaluations |
| Flan-T5 Base Google | MMLU | 35.9 | 34.95 | +0.95 | Scaling Instruction-Finetuned Language Models — authors' own runs · Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In — authors' own runs |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | IFEval (aggregation unspecified) | 88.9 | 88.0 | +0.9 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| GLM-5.1 Z.ai | τ³-bench | 70.6 | 69.7 | +0.9 | zai-org/GLM-5.1 model card · NVIDIA — Nemotron 3 Ultra technical report |
| OPT-1.3B Meta | BoolQ | 60.5 | 59.6 | +0.9 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| GLM-5.1 Z.ai | CritPt | 4.6 | 3.7 | +0.9 | zai-org/GLM-5.2 model card · NVIDIA — Nemotron 3 Ultra technical report |
| LLaMA 65B Meta | GSM8K | 50.9 | 50.0 | +0.9 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources — authors' own runs |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | GPQA Diamond | 81.1 | 80.225 | +0.875 | Qwen/Qwen3.5-27B model card · Epoch AI — internal evaluations |
| GPT-5 OpenAI | τ²-bench Telecom | 96.7 | 95.83 | +0.87 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs |
| Phi-1.5 (1.3B) Microsoft | WinoGrande | 73.4 | 72.534 | +0.866 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3.5 397B-A17B Alibaba Cloud | GPQA Diamond | 88.4 | 87.579 | +0.821 | Qwen/Qwen3.6-27B model card · Epoch AI — internal evaluations |
| GPT-Neo 2.7B EleutherAI | PIQA | 73.0 | 72.19 | +0.81 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| LLaMA 65B Meta | HellaSwag | 84.2 | 83.39 | +0.81 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| Qwen3.5 397B-A17B Alibaba Cloud | Humanity's Last Exam | 28.7 | 27.9 | +0.8 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Flan-T5 XL Google | BIG-Bench Hard | 41.0 | 40.2 | +0.8 | Scaling Instruction-Finetuned Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| DeepSeek V3 0324 DeepSeek | GPQA Diamond | 68.4 | 67.614 | +0.786 | deepseek-ai/DeepSeek-V3.1 model card · Epoch AI — internal evaluations |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | ZebraLogic | 95.0 | 94.22 | +0.78 | Qwen/Qwen3-235B-A22B-Instruct-2507 model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| GPT-J 6B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 69.0 | 68.232 | +0.768 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3.5 35B-A3B Alibaba Cloud | GPQA Diamond | 84.2 | 83.46 | +0.74 | Qwen/Qwen3.5-35B-A3B model card · Epoch AI — internal evaluations |
| DeepSeek V4 Pro (0813) DeepSeek | GPQA Diamond | 92.4 | 91.667 | +0.733 | deepseek-ai/DeepSeek-V4.1-Flash model card · Epoch AI — internal evaluations |
| LLaMA 33B Meta | PIQA | 82.3 | 81.579 | +0.721 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Mistral 7B v0.1 (base) Mistral AI | WinoGrande | 74.75 | 74.033 | +0.717 | Mistral 7B — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LLaMA 13B Meta | PIQA | 79.65 | 78.943 | +0.707 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Mistral 7B v0.1 (base) Mistral AI | ARC Easy | 80.25 | 79.545 | +0.705 | Mistral 7B — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| DeepSeek R1 Distill Qwen 7B DeepSeek | GPQA Diamond | 49.1 | 48.4 | +0.7 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| DeepSeek R1 Distill Llama 70B DeepSeek | AIME 2024 | 70.0 | 69.3 | +0.7 | deepseek-ai/DeepSeek-R1 model card · Microsoft — Phi-4-reasoning Technical Report |
| GLM-4.5 Z.ai | IFEval prompt-level strict | 86.1 | 85.4 | +0.7 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| StarCoder2 15B BigCode | CRUXEval-I | 48.1 | 47.4 | +0.7 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| SmolLM2 1.7B Instruct Hugging Face | MATH (subset unspecified) | 21.0 | 20.3 | +0.7 | SmolLM2 paper (Hugging Face) — authors' own instruct-model evaluations · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GPT-Neo 2.7B EleutherAI | MathQA | 24.72 | 24.05 | +0.67 | EleutherAI GPT-Neo 2.7B model card — EleutherAI evaluation-harness runs · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models — authors' own runs |
| Gemma 3 1B Google | MMLU-Pro | 14.7 | 14.04 | +0.66 | Google DeepMind Gemma 3 270M IT model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| StarCoder2 15B BigCode | HumanEval | 46.3 | 45.7 | +0.6 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| StarCoder2 3B BigCode | CRUXEval-I | 32.7 | 32.1 | +0.6 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Code Llama 13B Python Meta | HumanEval | 43.3 | 42.7 | +0.6 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Qwen2.5 Coder 32B Instruct Alibaba Cloud | HumanEval | 92.7 | 92.1 | +0.6 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| OPT-1.3B Meta | PIQA | 72.3 | 71.704 | +0.596 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| DeepSeek R1 0528 Qwen3 8B DeepSeek | AIME 2025 | 76.3 | 75.8 | +0.5 | deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Qwen3.5 4B Alibaba Cloud | MMLU-Redux | 88.8 | 88.3 | +0.5 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| Code Llama 13B (base) Meta | MBPP | 47.0 | 46.5 | +0.5 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| GPT-4o mini (2024-07-18) OpenAI | MGSM (simple-evals) | 87.0 | 86.5 | +0.5 | OpenAI simple-evals README benchmark results (OpenAI's own models) · Microsoft Research — Phi-4 Technical Report |
| Granite 4.0 1B IBM | MMLU-Pro | 34.02 | 33.53 | +0.49 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| GPT-J 6B EleutherAI | WinoGrande | 64.65 | 64.167 | +0.483 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Claude Opus 5 Anthropic | Terminal-Bench 4.0 | 52.3 | 51.82 | +0.48 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| GPT-NeoX-20B EleutherAI | PIQA | 77.9 | 77.421 | +0.479 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3.5 397B-A17B Alibaba Cloud | WMT24++ | 78.9 | 78.5 | +0.4 | Qwen/Qwen3.5-397B-A17B model card · Cohere — North Small Translate (arXiv:2609.13916v1) |
| GPT-4o mini (2024-07-18) OpenAI | DROP F1, 3-shot (simple-evals) | 79.7 | 79.3 | +0.4 | OpenAI simple-evals README benchmark results (OpenAI's own models) · Microsoft Research — Phi-4 Technical Report |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | OJBench | 32.5 | 32.1 | +0.4 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Pythia 1B EleutherAI | WinoGrande | 53.7 | 53.3 | +0.4 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| StarCoder2 7B BigCode | CRUXEval-I | 34.6 | 34.2 | +0.4 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| StarCoder2 15B BigCode | CRUXEval-O | 47.1 | 46.7 | +0.4 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| DeepSeek V3.1 DeepSeek | MMLU-Pro | 84.8 | 84.425 | +0.375 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| LLaMA 7B Meta | GSM8K | 11.0 | 10.65 | +0.35 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources — authors' own runs |
| DeepSeek V3 (Dec '24) DeepSeek | IFEval prompt-level strict | 86.1 | 85.77 | +0.33 | deepseek-ai/DeepSeek-R1 model card · OpenCompass CompassAcademic leaderboard (full version) Space: committed predictions file |
| LLaMA 13B Meta | MMLU | 46.9 | 46.57 | +0.33 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| StarCoder2 7B BigCode | HumanEval | 35.4 | 35.1 | +0.3 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| StarCoder2 3B BigCode | HumanEval | 31.7 | 31.4 | +0.3 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| GLM-4.5-Air Z.ai | MATH-500 | 98.1 | 97.8 | +0.3 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Pythia 6.9B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 67.3 | 67.01 | +0.29 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Code Llama 13B (base) Meta | CRUXEval-I | 42.488 | 42.2 | +0.288 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Mistral 7B v0.1 (base) Mistral AI | HumanEval | 28.35 | 28.07 | +0.28 | Mistral 7B — authors' own runs · MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (Hu et al.) |
| GPT-J 6B EleutherAI | PIQA | 75.85 | 75.571 | +0.279 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Phi-1.5 (1.3B) Microsoft | PIQA | 76.6 | 76.333 | +0.267 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Granite 4.0 H 1B IBM | GSM8K | 69.83 | 69.6 | +0.23 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| OPT-30B Meta | MMLU | 26.1 | 25.88 | +0.22 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy — authors' own runs |
| DeepSeek R1 Distill Qwen 1.5B DeepSeek | GPQA Diamond | 33.8 | 33.586 | +0.214 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| Code Llama 7B (base) Meta | CRUXEval-O | 34.213 | 34.0 | +0.212 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Granite 4.2 3B IBM | BFCL V4 (December 2025 archive) | 52.41 | 52.2 | +0.21 | ibm-granite/granite-4.2-30b model card · openbmb/MiniCPM5-2B model card |
| DeepSeek R1 Distill Qwen 7B DeepSeek | AIME 2024 | 55.5 | 55.3 | +0.2 | deepseek-ai/DeepSeek-R1 model card · microsoft/Phi-4-mini-reasoning model card |
| DeepSeek R1 0528 DeepSeek | Aider polyglot (225 exercises, two attempts) | 71.6 | 71.4 | +0.2 | deepseek-ai/DeepSeek-V3.1 model card · Aider polyglot benchmark runs |
| StarCoder2 3B BigCode | CRUXEval-O | 34.2 | 34.0 | +0.2 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| StarCoder2 7B BigCode | CRUXEval-O | 36.0 | 35.8 | +0.2 | StarCoder 2 and The Stack v2: The Next Generation — authors' own runs · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Gemma 7B (base) Google | MATH | 24.3 | 24.1 | +0.2 | Gemma report — original evaluations · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| GPT-4o mini (2024-07-18) OpenAI | MMLU (simple-evals, zero-shot CoT) | 82.0 | 81.8 | +0.2 | OpenAI simple-evals README benchmark results (OpenAI's own models) · Microsoft Research — Phi-4 Technical Report |
| LLaMA 65B Meta | HumanEval | 23.7 | 23.5 | +0.2 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources — authors' own runs |
| Qwen3.5 4B Alibaba Cloud | IFBench (aggregation unspecified) | 59.2 | 59.0 | +0.2 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| GLM-4.5-Air Z.ai | SWE-bench Verified | 57.6 | 57.4 | +0.2 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · SERA: Soft-Verified Efficient Repository Agents (arXiv:2601.20789v3) |
| LFM2.5 2.6B Liquid AI | IFBench (aggregation unspecified) | 59.17 | 59.0 | +0.17 | LiquidAI/LFM2.5-2.6B model card · openbmb/MiniCPM5-2B model card |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | IFEval (aggregation unspecified) | 88.7 | 88.54 | +0.16 | Qwen/Qwen3-Next-80B-A3B-Instruct model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| Pythia 6.9B EleutherAI | PIQA | 75.2 | 75.091 | +0.109 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Qwen3.5 4B Alibaba Cloud | SuperGPQA | 52.9 | 52.8 | +0.1 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| Nemotron 3 Ultra 550B-A55B NVIDIA | Humanity's Last Exam | 26.7 | 26.6 | +0.1 | NVIDIA — Nemotron 3 Ultra technical report · A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1 |
| LLaMA 65B Meta | MMLU | 63.4 | 63.3 | +0.1 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources — authors' own runs |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | HMMT 2025 (edition unspecified) | 83.9 | 83.8 | +0.1 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Mistral 7B v0.1 (base) Mistral AI | HellaSwag | 81.15 | 81.06 | +0.09 | Mistral 7B — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Pythia 2.8B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 64.7 | 64.66 | +0.04 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · RedPajama — original zero-shot 3B comparison |
| GPT-5.6 Sol OpenAI | Terminal-Bench 4.0 | 37.3 | 37.27 | +0.03 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Terminal-Bench 4.0 official leaderboard submissions |
| GPT-NeoX-20B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 72.0 | 71.977 | +0.023 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | MMLU-Pro | 84.4 | 84.4 | 0.0 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| DeepSeek V3 0324 DeepSeek | Aider polyglot (225 exercises, two attempts) | 55.1 | 55.1 | 0.0 | deepseek-ai/DeepSeek-V3.1 model card · Aider polyglot benchmark runs |
| Qwen3 235B-A22B (Apr 2025, hybrid) Alibaba Cloud | Aider polyglot (225 exercises, two attempts) | 59.6 | 59.6 | 0.0 | Qwen/Qwen3-30B-A3B-Instruct-2507 model card · Aider polyglot benchmark runs |
| Qwen3 32B Alibaba Cloud | Aider polyglot (225 exercises, two attempts) | 40.0 | 40.0 | 0.0 | Qwen/Qwen3-Next-80B-A3B-Instruct model card · Aider polyglot benchmark runs |
| Qwen3 4B Instruct 2507 Alibaba Cloud | τ²-bench Telecom | 13.2 | 13.2 | 0.0 | Qwen/Qwen3-4B-Instruct-2507 model card · Synthesize and Reward: Reinforcement Learning for Multi-Step Tool Use in Live Environments (arXiv:2606.03892v2) |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | τ²-bench Airline | 58.0 | 58.0 | 0.0 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Qwen3.5 2B Alibaba Cloud | BFCL V4 (December 2025 archive) | 43.6 | 43.6 | 0.0 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 397B-A17B Alibaba Cloud | Humanity's Last Exam with tools | 48.3 | 48.3 | 0.0 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Qwen3.5 9B Alibaba Cloud | MMLU-Redux | 91.1 | 91.1 | 0.0 | Qwen/Qwen3.5-9B model card · JetBrains/Mellum2-12B-A2.5B-Instruct model card |
| Pythia 12B EleutherAI | SciQ | 90.2 | 90.2 | 0.0 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Magicoder-S-DS 6.7B ise-uiuc | HumanEval | 76.8 | 76.8 | 0.0 | Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| Qwen2.5 Coder 32B Instruct Alibaba Cloud | HumanEval+ | 87.2 | 87.2 | 0.0 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| CodeQwen1.5 7B Chat Alibaba Cloud | HumanEval | 83.5 | 83.5 | 0.0 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| CodeQwen1.5 7B Chat Alibaba Cloud | HumanEval+ | 78.7 | 78.7 | 0.0 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| Command A+ (05-2026) Cohere | Terminal-Bench Hard | 25.0 | 25.0 | 0.0 | Cohere vendor-reported benchmark scores for its own models (cohere.com blog) · Solar Open 2 Technical Report (Upstage, arXiv:2607.20062v2) |
| Command R Plus (08-2024) Cohere | RULER 128k | 64.6 | 64.6 | 0.0 | Cohere — Command A technical report · NVIDIA RULER repository README results table |
| Gemma 3 12B Google | MATH (subset unspecified) | 83.8 | 83.8 | 0.0 | Google DeepMind Gemma 3 270M IT model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| CodeGemma 7B (v1.0) Google | HumanEval | 44.5 | 44.5 | 0.0 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| Code Llama 7B (base) Meta | HumanEval | 33.5 | 33.5 | 0.0 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| gpt-oss-20b OpenAI | SimpleQA | 6.7 | 6.7 | 0.0 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| GPT-Neo 1.3B EleutherAI | WinoGrande | 55.0 | 55.015 | -0.015 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · What Language Model to Train if You Have One Million GPU Hours? — authors' own runs |
| Nova Pro Amazon | BFCL v3 | 60.8 | 60.82 | -0.02 | Amazon vendor-reported benchmark scores for its own Nova models (Amazon Science technical reports) · Berkeley Function Calling Leaderboard result archive |
| GPT-Neo 2.7B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 62.2 | 62.235 | -0.035 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · RedPajama — original zero-shot 3B comparison |
| GPT-NeoX-20B EleutherAI | WinoGrande | 66.1 | 66.14 | -0.04 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Flan-T5 XXL Google | MMLU | 55.1 | 55.145 | -0.045 | Scaling Instruction-Finetuned Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| GPT-Neo 1.3B EleutherAI | MathQA | 24.05 | 24.1 | -0.05 | EleutherAI GPT-Neo 2.7B model card — EleutherAI evaluation-harness runs · What Language Model to Train if You Have One Million GPU Hours? — authors' own runs |
| DeepSeek R1 (Jan '25) DeepSeek | IFEval prompt-level strict | 83.3 | 83.36 | -0.06 | deepseek-ai/DeepSeek-R1 model card · OpenCompass CompassAcademic leaderboard (full version) Space: committed predictions file |
| Pythia 2.8B EleutherAI | PIQA | 73.9 | 73.99 | -0.09 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · RedPajama — original zero-shot 3B comparison |
| GLM-5.1 Z.ai | AIME 2026 | 95.3 | 95.4 | -0.1 | zai-org/GLM-5.1 model card · A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1 |
| LLaMA 33B Meta | MMLU | 57.8 | 57.9 | -0.1 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Code Llama 34B (base) Meta | CRUXEval-O | 42.4 | 42.5 | -0.1 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Qwen3.5 35B-A3B Alibaba Cloud | AIME 2026 | 91.0 | 91.1 | -0.1 | Qwen/Qwen3.6-35B-A3B model card · NVIDIA — Nemotron-Cascade 2 technical report |
| OPT-30B Meta | PIQA | 77.5 | 77.6 | -0.1 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Pythia 12B EleutherAI | WinoGrande | 63.9 | 64.009 | -0.109 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LLaMA 65B Meta | WinoGrande | 77.0 | 77.11 | -0.11 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| GPT-5.6 Sol OpenAI | FrontierMath tiers 1–3 v2 | 89.0 | 89.123 | -0.123 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Seed-OSS 36B Instruct ByteDance Seed | GPQA Diamond | 71.4 | 71.528 | -0.128 | ByteDance-Seed/Seed-OSS-36B-Instruct model card · Epoch AI — internal evaluations |
| Pythia 12B EleutherAI | LAMBADA (OpenAI version), last-word accuracy | 70.5 | 70.638 | -0.138 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LLaMA 7B Meta | HumanEval | 10.5 | 10.65 | -0.15 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · InstructEval (DeCLaRe Lab) README results table |
| Code Llama 13B (base) Meta | CRUXEval-O | 39.738 | 39.9 | -0.163 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | AIME 2025 | 92.3 | 92.5 | -0.2 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| GPT-J 6B EleutherAI | OpenBookQA | 28.8 | 29.0 | -0.2 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Kimi K2 Instruct (0711) Moonshot AI | τ²-bench Retail | 70.6 | 70.8 | -0.2 | τ²-bench leaderboard: Moonshot AI's own submission · Meituan LongCat-Flash-Chat model card (evaluation table) |
| OPT-1.3B Meta | WinoGrande | 59.5 | 59.724 | -0.224 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Code Llama 7B (base) Meta | CRUXEval-I | 35.95 | 36.2 | -0.25 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Gemma 3 4B Google | MMLU | 58.1 | 58.35 | -0.25 | Gemma Team — Gemma 3 Technical Report · LFM2 Technical Report — Liquid AI own SmolLM3 evaluation |
| Pythia 12B EleutherAI | PIQA | 76.0 | 76.279 | -0.279 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| DeepSeek R1 0528 Qwen3 8B DeepSeek | GPQA Diamond | 61.1 | 61.4 | -0.3 | deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| GPT-NeoX-20B EleutherAI | SciQ | 92.8 | 93.1 | -0.3 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Phi-3.5-vision-instruct Microsoft | ChartQA | 81.8 | 82.1 | -0.3 | Microsoft Phi-3.5-vision-instruct model card: Benchmarks table · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Gemma 3 1B Google | BIG-Bench Hard | 39.1 | 39.4 | -0.3 | Google DeepMind Gemma 3 270M IT model card · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| gpt-oss-20b OpenAI | Humanity's Last Exam | 10.9 | 11.25 | -0.35 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| DeepSeek R1 (Jan '25) DeepSeek | AIME 2025 | 70.0 | 70.4 | -0.4 | deepseek-ai/DeepSeek-R1-0528 model card · Microsoft — Phi-4-reasoning Technical Report |
| Qwen3.5 2B Alibaba Cloud | MMLU-Redux | 79.6 | 80.0 | -0.4 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| DeepSeek V3 (Dec '24) DeepSeek | Aider polyglot (225 exercises, two attempts) | 49.6 | 50.0 | -0.4 | deepseek-ai/DeepSeek-R1 model card · Aider polyglot benchmark runs |
| Phi-1.5 (1.3B) Microsoft | OpenBookQA | 37.2 | 37.6 | -0.4 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LLaMA 33B Meta | GSM8K | 35.6 | 36.0 | -0.4 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| GLM-4.5 Z.ai | Humanity's Last Exam | 14.4 | 14.8 | -0.4 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Pythia 6.9B EleutherAI | WinoGrande | 60.9 | 61.308 | -0.408 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| GPT-5 OpenAI | GPQA Diamond | 85.7 | 86.174 | -0.474 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Granite 4.0 H 1B IBM | BFCL v3 | 50.21 | 50.69 | -0.48 | ibm-granite/granite-4.0-h-1b model card · LiquidAI/LFM2.5-1.2B-Thinking model card |
| Qwen3.5 397B-A17B Alibaba Cloud | MMLU-Pro | 87.8 | 88.3 | -0.5 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | SuperGPQA | 56.8 | 57.3 | -0.5 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| GPT-J 6B EleutherAI | SciQ | 91.0 | 91.5 | -0.5 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Code Llama 34B (base) Meta | CRUXEval-I | 47.213 | 47.8 | -0.587 | CRUXEval published evaluation results (authors' generations and scores) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| Qwen3.5 4B Alibaba Cloud | GPQA Diamond | 76.2 | 76.8 | -0.6 | Qwen/Qwen3.5-4B model card · JetBrains/Mellum2-12B-A2.5B-Instruct model card |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | MMMLU | 78.4 | 79.0 | -0.6 | Qwen/Qwen3.5-9B model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Qwen3 32B Alibaba Cloud | SuperGPQA | 48.65 | 49.3 | -0.65 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| GLM-5.2 Z.ai | GPQA Diamond | 91.2 | 91.856 | -0.656 | zai-org/GLM-5.2 model card · Epoch AI — internal evaluations |
| Pythia 2.8B EleutherAI | WinoGrande | 59.4 | 60.06 | -0.66 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · RedPajama — original zero-shot 3B comparison |
| Gemma 3 1B Google | MMLU | 38.8 | 39.49 | -0.69 | Gemma Team — Gemma 3 Technical Report · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GPT-4o mini (2024-07-18) OpenAI | GPQA Diamond (simple-evals) | 40.2 | 40.9 | -0.7 | OpenAI simple-evals README benchmark results (OpenAI's own models) · Microsoft Research — Phi-4 Technical Report |
| Qwen3.5 4B Alibaba Cloud | OJBench | 24.1 | 24.8 | -0.7 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| LFM2.5 8B-A1B Liquid AI | BFCL V4 (December 2025 archive) | 48.5 | 49.2 | -0.7 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| Qwen3-VL 2B Instruct Alibaba Cloud | RealWorldQA | 63.9 | 64.6 | -0.7 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Pythia 6.9B EleutherAI | SciQ | 89.7 | 90.45 | -0.75 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| GPT-Neo 2.7B EleutherAI | PubMedQA | 57.54 | 58.3 | -0.76 | EleutherAI GPT-Neo 2.7B model card — EleutherAI evaluation-harness runs · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models — authors' own runs |
| DeepSeek V3.1 DeepSeek | AIME 2024 | 93.1 | 93.9 | -0.8 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| GPT-NeoX-20B EleutherAI | OpenBookQA | 29.0 | 29.8 | -0.8 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| StarCoder 15.5B BigCode | HumanEval | 33.6 | 34.45 | -0.85 | StarCoder: may the source be with you! — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| DeepSeek R1 (Jan '25) DeepSeek | GPQA Diamond | 71.5 | 72.359 | -0.859 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| o3 OpenAI | GPQA Diamond | 81.0 | 81.859 | -0.859 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Phi-3.5-vision-instruct Microsoft | AI2D test | 78.1 | 79.0 | -0.9 | Microsoft Phi-3.5-vision-instruct model card: Benchmarks table · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Qwen3.5 4B Alibaba Cloud | BFCL V4 (December 2025 archive) | 50.3 | 51.28 | -0.98 | Qwen/Qwen3.5-9B model card · openbmb/MiniCPM5-2B model card |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | MMLU-Pro | 80.9 | 81.9 | -1.0 | Qwen/Qwen3-30B-A3B-Thinking-2507 model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Qwen3 32B Alibaba Cloud | τ²-bench Telecom | 26.3 | 27.3 | -1.0 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Inkling (975B-A41B) Thinking Machines Lab | GPQA Diamond | 87.2 | 88.258 | -1.058 | thinkingmachines/Inkling-Small model card · Epoch AI — internal evaluations |
| GPT-5.6 Terra OpenAI | FrontierMath tiers 1–3 v2 | 84.9 | 85.965 | -1.065 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Granite 4.2 3B IBM | AIME 2025 | 78.33 | 79.4 | -1.07 | ibm-granite/granite-4.2-30b model card · openbmb/MiniCPM5-2B model card |
| Qwen3.5 2B Alibaba Cloud | SuperGPQA | 37.5 | 38.6 | -1.1 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| DeepSeek V4 Flash (0731) DeepSeek | GPQA Diamond | 89.9 | 91.035 | -1.135 | deepseek-ai/DeepSeek-V4.1-Flash model card · Epoch AI — internal evaluations |
| Jamba 1.5 Large AI21 Labs | MMLU-Pro | 48.3 | 49.46 | -1.16 | Jamba-1.5: Hybrid Transformer-Mamba Models at Scale (AI21, arXiv 2408.12570v1) · MMLU-Pro author-released evaluation runs |
| Qwen3.5 397B-A17B Alibaba Cloud | AIME 2026 | 91.3 | 92.5 | -1.2 | Qwen/Qwen3.5-397B-A17B model card · A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1 |
| Qwen3 30B-A3B Thinking 2507 Alibaba Cloud | MMLU-ProX | 76.4 | 77.6 | -1.2 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · NVIDIA — Nemotron 3 Nano technical report |
| GPT-Neo 2.7B EleutherAI | WinoGrande | 56.5 | 57.7 | -1.2 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs |
| Code Llama 7B Python Meta | HumanEval | 38.4 | 39.6 | -1.2 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Code Llama 34B (base) Meta | HumanEval | 48.8 | 50.0 | -1.2 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| Gemma 7B (base) Google | HumanEval | 32.3 | 33.55 | -1.25 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| LLaMA 33B Meta | MATH | 7.1 | 8.4 | -1.3 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| GLM-4.5 Z.ai | BFCL v3 | 77.8 | 79.1 | -1.3 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Mistral 7B v0.1 (base) Mistral AI | MMLU | 61.3 | 62.69 | -1.39 | Mistral 7B — authors' own runs · MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (Hu et al.) |
| Code Llama 70B Instruct Meta | HumanEval | 67.8 | 69.25 | -1.45 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Gemma 2B (base) Google | HumanEval | 22.0 | 23.5 | -1.5 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| OPT-175B Meta | PIQA | 79.5 | 81.07 | -1.57 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Qwen3 1.7B Alibaba Cloud | IFEval (aggregation unspecified) | 72.5 | 74.09 | -1.59 | Qwen/Qwen3.5-2B model card · HuggingFaceTB/SmolLM3-3B model card (Evaluation section) |
| Qwen3.5 2B Alibaba Cloud | OCRBench | 84.5 | 86.1 | -1.6 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Qwen3.5 397B-A17B Alibaba Cloud | SWE-bench Multilingual | 69.3 | 70.9 | -1.6 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Granite 4.0 H Small (32B-A9B) IBM | MMLU-Pro | 55.47 | 57.1 | -1.63 | ibm-granite/granite-4.0-h-small model card · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| Qwen3 4B Alibaba Cloud | BFCL v3 | 61.75 | 63.39 | -1.64 | Qwen/Qwen3-4B-Thinking-2507 model card · Berkeley Function Calling Leaderboard result archive |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | τ²-bench Telecom | 45.6 | 47.3 | -1.7 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Qwen3.5 397B-A17B Alibaba Cloud | MMLU-ProX | 84.7 | 86.4 | -1.7 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Qwen3.6 27B Alibaba Cloud | IFBench (aggregation unspecified) | 69.1 | 70.8 | -1.7 | Qwen/Qwen3.8-27B model card · meta-models/Muse-Glimmer-30B model card (speed and benchmark tables) |
| Nemotron Nano 9B v2 NVIDIA | MMLU | 82.6 | 84.3 | -1.7 | NVIDIA — Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models · Olmo 3 |
| Kimi K2 Instruct (0711) Moonshot AI | τ²-bench Telecom | 65.8 | 67.5 | -1.7 | τ²-bench leaderboard: Moonshot AI's own submission · Meituan LongCat-Flash-Chat model card (evaluation table) |
| gpt-oss-20b OpenAI | MMMLU | 75.7 | 77.4 | -1.7 | OpenAI — gpt-oss-120b & gpt-oss-20b Model Card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| BLOOM 560M BigScience | WinoGrande | 49.88 | 51.601 | -1.721 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| CodeQwen1.5 7B Chat Alibaba Cloud | MBPP+ | 67.2 | 69.0 | -1.8 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| LLaMA 33B Meta | HumanEval | 21.7 | 23.5 | -1.8 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| GLM-5 Z.ai | GPQA Diamond | 86.0 | 87.817 | -1.817 | GLM-5 Team (Zhipu AI) — GLM-5 technical report · Epoch AI — internal evaluations |
| Qwen3 235B-A22B Instruct 2507 Alibaba Cloud | MMLU-Pro | 83.0 | 84.83 | -1.83 | Qwen/Qwen3-235B-A22B-Instruct-2507 model card · Meituan LongCat-Flash-Chat model card (evaluation table) |
| Qwen3 30B-A3B (Apr 2025, hybrid) Alibaba Cloud | BFCL v3 | 63.85 | 65.72 | -1.87 | Qwen/Qwen3-4B-Thinking-2507 model card · Berkeley Function Calling Leaderboard result archive |
| Qwen2.5 Coder 32B Instruct Alibaba Cloud | MBPP+ | 75.1 | 77.0 | -1.9 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · EvalPlus / EvalPerf published evaluation runs |
| Pythia 1B EleutherAI | SciQ | 84.0 | 86.0 | -2.0 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Qwen3 8B Alibaba Cloud | LiveCodeBench v6 (window unspecified) | 51.0 | 53.0 | -2.0 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Code Llama 34B (base) Meta | MBPP | 55.0 | 57.0 | -2.0 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| OPT-1.3B Meta | COPA | 77.0 | 79.0 | -2.0 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models — authors' own runs |
| DeepSeek R1 Distill Qwen 32B DeepSeek | GPQA Diamond | 62.1 | 64.141 | -2.041 | deepseek-ai/DeepSeek-R1 model card · Epoch AI — internal evaluations |
| Claude Fable 5.1 Anthropic | Terminal-Bench 4.0 | 55.8 | 57.88 | -2.08 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| Claude Sonnet 5 Anthropic | Terminal-Bench 4.0 | 10.3 | 12.42 | -2.12 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| GLM-5.1 Z.ai | GPQA Diamond | 86.2 | 88.349 | -2.149 | zai-org/GLM-5.1 model card · Epoch AI — internal evaluations |
| Qwen3.5 397B-A17B Alibaba Cloud | IMO-AnswerBench | 80.9 | 83.1 | -2.2 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| Gemma 2B (base) Google | Natural Questions (closed book) | 12.5 | 14.76 | -2.26 | Gemma report — original evaluations · Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results) |
| Qwen3.5 397B-A17B Alibaba Cloud | IFBench (aggregation unspecified) | 76.5 | 78.8 | -2.3 | Qwen/Qwen3.5-397B-A17B model card · A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1 |
| Gemma 4 E2B Google | AIME 2026 | 37.5 | 39.8 | -2.3 | Gemma Team — Gemma 4 Technical Report · openbmb/MiniCPM5-2B model card |
| Qwen3 32B Alibaba Cloud | MMLU-Pro | 75.5 | 77.85 | -2.35 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| EXAONE 4.0 32B LG AI Research | τ²-bench Telecom | 23.7 | 26.1 | -2.4 | LG AI Research vendor-reported benchmark scores for its own EXAONE models (Hugging Face model cards) · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| GPT-5.6 Terra OpenAI | FrontierMath tier 4 v2 | 68.3 | 70.732 | -2.432 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Llama 3.1 70B Instruct Meta | GSM8K | 92.04 | 94.5 | -2.46 | Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (arXiv:2411.19146v5) · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GPT-5.6 Luna OpenAI | FrontierMath tier 4 v2 | 58.5 | 60.976 | -2.476 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Gemma 4 E4B Google | AIME 2026 | 42.5 | 45.0 | -2.5 | Gemma Team — Gemma 4 Technical Report · openbmb/MiniCPM5-2B model card |
| GPT-J 6B EleutherAI | ARC Challenge | 34.0 | 36.519 | -2.519 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Claude Fable 5 Anthropic | Terminal-Bench 4.0 | 42.0 | 44.55 | -2.55 | Anthropic system cards and model announcements (lab-reported scores) · Terminal-Bench 4.0 official leaderboard submissions |
| BLOOM 176B BigScience | COPA | 55.0 | 57.6 | -2.6 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items (arXiv:2410.14897v1) |
| Qwen3 32B Alibaba Cloud | BFCL v3 | 66.65 | 69.25 | -2.6 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Berkeley Function Calling Leaderboard result archive |
| LFM2 8B-A1B Liquid AI | GPQA Diamond | 29.29 | 31.9 | -2.61 | LFM2 Technical Report — Liquid AI own SmolLM3 evaluation · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| Gemma 4 E2B Google | τ²-bench Telecom | 19.7 | 22.37 | -2.67 | Gemma Team — Gemma 4 Technical Report · LiquidAI/LFM2.5-8B-A1B model card |
| GLM-4.5-Air Z.ai | Humanity's Last Exam | 10.6 | 13.3 | -2.7 | GLM-4.5 Team (Zhipu AI & Tsinghua) — GLM-4.5 technical report · Prime Intellect INTELLECT-3 model card: Evaluation table (Environments Hub runs) |
| Phi-1.5 (1.3B) Microsoft | Social IQA | 52.6 | 55.374 | -2.774 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| GPT-NeoX-20B EleutherAI | ARC Challenge | 38.0 | 40.785 | -2.785 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Gemma 3 1B Google | GPQA Diamond | 19.2 | 22.095 | -2.895 | Google DeepMind Gemma 3 270M IT model card · Epoch AI — internal evaluations |
| GPT-4.1 OpenAI | τ²-bench Telecom | 34.0 | 36.9 | -2.9 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs |
| Mistral Small 3.2 24B Mistral AI | GPQA Diamond | 46.13 | 49.053 | -2.923 | mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card · Epoch AI — internal evaluations |
| GLM-5.1 Z.ai | IMO-AnswerBench | 83.8 | 86.8 | -3.0 | zai-org/GLM-5.2 model card · NVIDIA — Nemotron 3 Ultra technical report |
| Code Llama 70B Python Meta | HumanEval | 57.3 | 60.4 | -3.1 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| DeepSeek R1 Distill Qwen 32B DeepSeek | AIME 2024 | 72.6 | 75.7 | -3.1 | deepseek-ai/DeepSeek-R1 model card · open-thoughts/OpenThinker2-32B model card |
| Phi-4-reasoning-plus 14B Microsoft | MMLU-Pro | 76.0 | 79.2 | -3.2 | microsoft/Phi-4-reasoning model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| Pythia 12B EleutherAI | ARC Challenge | 31.8 | 35.068 | -3.268 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| LFM2.5 8B-A1B Liquid AI | AIME 2025 | 42.53 | 46.0 | -3.47 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| GPT-5.6 Luna OpenAI | FrontierMath tiers 1–3 v2 | 78.6 | 82.105 | -3.505 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Epoch AI — internal evaluations |
| Llama 3.1 70B Instruct Meta | MMLU | 81.66 | 85.2 | -3.54 | Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (arXiv:2411.19146v5) · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| DeepSeek R1 (Jan '25) DeepSeek | Aider polyglot (225 exercises, two attempts) | 53.3 | 56.9 | -3.6 | deepseek-ai/DeepSeek-R1-0528 model card · Aider polyglot benchmark runs |
| Phi-1.5 (1.3B) Microsoft | ARC Challenge | 44.4 | 48.038 | -3.638 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| DeepSeek V3 (Dec '24) DeepSeek | SWE-bench Verified | 42.0 | 45.8 | -3.8 | deepseek-ai/DeepSeek-R1 model card · Cohere — Command A technical report |
| Qwen3 235B-A22B (Apr 2025, hybrid) Alibaba Cloud | τ²-bench Airline | 33.0 | 36.8 | -3.8 | Qwen/Qwen3-235B-A22B-Thinking-2507 model card · HyperCLOVA X 32B Think (NAVER Cloud, arXiv:2601.03286v1) |
| Qwen3 235B-A22B Thinking 2507 Alibaba Cloud | BFCL v3 | 71.9 | 75.7 | -3.8 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| BLOOM 1.1B BigScience | WinoGrande | 50.99 | 54.816 | -3.826 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Qwen3 4B Alibaba Cloud | IFEval (aggregation unspecified) | 81.55 | 85.4 | -3.85 | Qwen/Qwen3-4B-Thinking-2507 model card · HuggingFaceTB/SmolLM3-3B model card (Evaluation section) |
| DeepSeek V3.1 DeepSeek | GPQA Diamond | 80.1 | 84.2 | -4.1 | deepseek-ai/DeepSeek-V3.1 model card · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| Mistral 7B v0.1 (base) Mistral AI | Natural Questions (closed book) | 26.0 | 30.11 | -4.11 | Mixtral of Experts — base-model evaluations · Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results) |
| LFM2.5 2.6B Liquid AI | BFCL V4 (December 2025 archive) | 56.88 | 61.1 | -4.22 | LiquidAI/LFM2.5-2.6B model card · openbmb/MiniCPM5-2B model card |
| Code Llama 13B (base) Meta | HumanEval | 36.0 | 40.25 | -4.25 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| CodeGemma 7B IT (v1.0) Google | HumanEval | 56.1 | 60.4 | -4.3 | CodeGemma: Open Code Models Based on Gemma — authors' own runs · EvalPlus / EvalPerf published evaluation runs |
| Qwen3 32B Alibaba Cloud | IFEval (aggregation unspecified) | 84.1 | 88.4 | -4.3 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Gemma 3 12B Google | MMLU | 71.9 | 76.2 | -4.3 | Gemma Team — Gemma 3 Technical Report · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| LFM2.5 8B-A1B Liquid AI | MATH-500 | 88.76 | 93.2 | -4.44 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| GLM-4.5V Z.ai | CharXiv (reasoning questions) | 58.4 | 63.0 | -4.6 | Z.ai GLM-V GitHub repository benchmark image (GLM-4.6V) · ServiceNow-AI/Apriel-1.6-15b-Thinker model card |
| Mistral 7B v0.1 (base) Mistral AI | TriviaQA | 66.2 | 70.86 | -4.66 | Mistral 7B — authors' own runs · Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results) |
| Qwen3.5 2B Alibaba Cloud | IFBench (aggregation unspecified) | 41.3 | 46.0 | -4.7 | Qwen/Qwen3.5-2B model card · openbmb/MiniCPM5-2B model card |
| DeepSeek V3.2 DeepSeek | τ²-bench aggregate | 80.3 | 85.2 | -4.9 | deepseek-ai/DeepSeek-V3.2 model card · stepfun-ai/Step-3.5-Flash model card |
| Gemma 3 27B Google | MMLU | 76.9 | 81.8 | -4.9 | Gemma Team — Gemma 3 Technical Report · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| GPT-4.1 mini OpenAI | τ²-bench Telecom | 44.0 | 48.9 | -4.9 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs |
| Qwen3-VL 2B Instruct Alibaba Cloud | IFEval (aggregation unspecified) | 68.2 | 73.4 | -5.2 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| Gemma 3 27B Google | GPQA Diamond | 42.4 | 47.727 | -5.327 | Gemma Team — Gemma 4 Technical Report · Epoch AI — internal evaluations |
| o4-mini OpenAI | τ²-bench Telecom | 40.5 | 46.1 | -5.6 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs |
| Qwen3.5 397B-A17B Alibaba Cloud | LongBench v2 | 63.2 | 68.9 | -5.7 | Qwen/Qwen3.5-397B-A17B model card · NVIDIA — Nemotron 3 Ultra technical report |
| o3 OpenAI | τ²-bench Telecom | 58.2 | 64.0 | -5.8 | OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · Meituan LongCat-Flash-Thinking model card (evaluation table) |
| OPT-175B Meta | ARC Easy | 67.0 | 72.97 | -5.97 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| BLOOM 560M BigScience | COPA | 55.0 | 61.0 | -6.0 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| OPT-30B Meta | ARC Easy | 63.9 | 69.996 | -6.096 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Code Llama 13B Instruct Meta | HumanEval | 42.683 | 48.8 | -6.117 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| OPT-1.3B Meta | ARC Easy | 50.8 | 57.0 | -6.2 | OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| LLaMA 7B Meta | HellaSwag | 66.5 | 72.99 | -6.49 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| LFM2.5 8B-A1B Liquid AI | AIME 2026 | 50.0 | 56.7 | -6.7 | LiquidAI/LFM2.5-8B-A1B model card · openbmb/MiniCPM5-2B model card |
| Gemma 7B (base) Google | GSM8K | 46.4 | 53.3 | -6.9 | Gemma report — original evaluations · Granite Code Models: A Family of Open Foundation Models for Code Intelligence — authors' own runs |
| LFM2 8B-A1B Liquid AI | MMLU | 64.84 | 72.1 | -7.26 | LFM2 Technical Report — Liquid AI own SmolLM3 evaluation · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| GPT-J 6B EleutherAI | HellaSwag | 58.95 | 66.242 | -7.292 | EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Mistral 7B v0.1 (base) Mistral AI | ARC Challenge | 55.2 | 62.541 | -7.341 | Mistral 7B — authors' own runs · MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (Hu et al.) |
| Qwen3.5 2B Alibaba Cloud | HallusionBench | 58.0 | 65.5 | -7.5 | Qwen/Qwen3.5-2B model card · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| SmolLM2 1.7B Instruct Hugging Face | BIG-Bench Hard | 32.2 | 39.8 | -7.6 | SmolLM2 paper (Hugging Face) — authors' own instruct-model evaluations · 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models |
| BLOOM 1.7B BigScience | WinoGrande | 49.57 | 57.171 | -7.601 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| LLaMA 13B Meta | HellaSwag | 69.55 | 77.64 | -8.09 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Pythia 6.9B EleutherAI | ARC Challenge | 31.6 | 39.712 | -8.112 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Gemma 4 E4B Google | τ²-bench Telecom | 18.4 | 26.75 | -8.35 | Gemma Team — Gemma 4 Technical Report · LiquidAI/LFM2.5-8B-A1B model card |
| Code Llama 7B Instruct Meta | HumanEval | 34.756 | 43.3 | -8.544 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| Qwen3 32B Alibaba Cloud | LiveCodeBench v6 (Feb–May 2025) | 44.85 | 53.4 | -8.55 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| LLaMA 13B Meta | Social IQA | 50.4 | 58.98 | -8.58 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate — authors' own runs |
| BLOOM 3B BigScience | WinoGrande | 49.96 | 58.6 | -8.64 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs |
| Pythia 1B EleutherAI | ARC Challenge | 24.4 | 33.1 | -8.7 | Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Llama 3.1 Tulu 3 8B Ai2 | TruthfulQA MC2 | 55.0 | 64.1 | -9.1 | 2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models · microsoft/Phi-4-mini-instruct model card |
| Pythia 6.9B EleutherAI | OpenBookQA | 25.2 | 35.4 | -10.2 | trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| Code Llama 34B Instruct Meta | HumanEval | 41.5 | 51.8 | -10.3 | Code Llama: Open Foundation Models for Code — authors' own runs · Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs |
| LLaMA 33B Meta | MBPP | 30.2 | 40.9 | -10.7 | LLaMA: Open and Efficient Foundation Language Models — authors' own runs · Mixtral of Experts — base-model evaluations |
| Code Llama 7B (base) Meta | MBPP | 41.4 | 52.1 | -10.7 | Code Llama: Open Foundation Models for Code — authors' own runs · StarCoder 2 and The Stack v2: The Next Generation — authors' own runs |
| BLOOM 1.1B BigScience | COPA | 57.0 | 68.0 | -11.0 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| Qwen3.6 27B Alibaba Cloud | OSWorld-Verified | 63.9 | 75.6 | -11.7 | Qwen/Qwen3.8-27B model card · meta-models/Muse-Glimmer-30B model card (speed and benchmark tables) |
| BLOOM 560M BigScience | Story Cloze Test | 50.08 | 61.94 | -11.86 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| BLOOM 3B BigScience | COPA | 62.0 | 74.0 | -12.0 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| LFM2 8B-A1B Liquid AI | MMLU-Pro | 37.42 | 49.5 | -12.08 | LFM2 Technical Report — Liquid AI own SmolLM3 evaluation · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| BLOOMZ 176B BigScience | WinoGrande | 58.41 | 72.5 | -14.09 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| Qwen3 32B Alibaba Cloud | τ-bench Airline | 23.0 | 38.0 | -15.0 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · ByteDance-Seed/Seed-OSS-36B-Instruct model card |
| Phi-1.5 (1.3B) Microsoft | HellaSwag | 47.6 | 62.617 | -15.017 | Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Pythia 6.9B EleutherAI | HellaSwag | 48.8 | 63.841 | -15.041 | trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback — authors' own runs · OLMo: Accelerating the Science of Language Models — authors' own runs |
| BLOOM 7.1B BigScience | WinoGrande | 49.41 | 64.641 | -15.231 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3-VL 2B Instruct Alibaba Cloud | HallusionBench | 51.4 | 67.3 | -15.9 | Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · CohereLabs/North-Micro-Vision-Instruct model card (benchmark table) |
| BLOOM 1.7B BigScience | COPA | 54.0 | 70.0 | -16.0 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · EleutherAI lm-evaluation-harness results folder (third-party runs committed to the repo) |
| BLOOM 1.1B BigScience | Story Cloze Test | 47.25 | 63.27 | -16.02 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Qwen3 4B Alibaba Cloud | AIME 2025 | 42.35 | 58.8 | -16.45 | Qwen/Qwen3-4B-Thinking-2507 model card · HuggingFaceTB/SmolLM3-3B model card (Evaluation section) |
| BLOOM 1.7B BigScience | Story Cloze Test | 48.48 | 65.44 | -16.96 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| GPT-NeoX-20B EleutherAI | HellaSwag | 53.5 | 71.43 | -17.93 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Stability AI — original StableLM zero-shot evaluation outputs |
| Qwen3 32B Alibaba Cloud | HMMT 2025 (edition unspecified) | 30.65 | 49.8 | -19.15 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| BLOOM 3B BigScience | Story Cloze Test | 47.51 | 67.79 | -20.28 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| BLOOM 176B BigScience | WinoGrande | 48.62 | 70.3 | -21.68 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — authors' own runs |
| GPT-J 6B EleutherAI | TriviaQA | 17.0 | 39.32 | -22.32 | GPT-NeoX-20B: An Open-Source Autoregressive Language Model — authors' own runs · Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results) |
| BLOOM 7.1B BigScience | Story Cloze Test | 49.44 | 71.99 | -22.55 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
| Qwen3 32B Alibaba Cloud | AIME 2025 | 46.55 | 72.15 | -25.6 | Qwen/Qwen3-Next-80B-A3B-Thinking model card · Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling |
| BLOOM 176B BigScience | Story Cloze Test | 50.99 | 76.89 | -25.9 | Crosslingual Generalization through Multitask Finetuning — authors' own runs · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — authors' own runs |
Every result links to its source on the model's page. The data (CC BY-SA 4.0) and code (MIT) are on GitHub.