whatmodel.sh / Claude Sonnet 4
Claude Sonnet 4
↗ share
+ contribute
Claude Sonnet 4 is a closed-weights language model from Anthropic, released May 2025. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2025-05-22). Dated API snapshot claude-sonnet-4-20250514; distinct from Sonnet 4.5/4.6. Thinking budgets recorded per result. Anthropic model deprecations page: retired June 15, 2026.
$ how it compares
The LLMs ranked either side of Claude Sonnet 4, out of 828 with a score, then more from Anthropic.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 158 |  o3-mini | 65±1 | api $1.93/M | ||
| 159 | Claude 3.7 Sonnet | 65±1 | comparison only | ||
| 160 | Claude Sonnet 4 | 65±1 | comparison only | ||
| 161 |  MiMo-V2.6-Distill-Qwen-9B | 65±2 | local | ||
| 162 |  K-EXAONE (236B-A23B) | 65±1 | local | ||
| more from Anthropic | |||||
| 2 | Claude Opus 5.5 | 87±1 | api $8.00/M | ||
| 4 | Claude Fable 5.1 | 85±1 | api $20.00/M | ||
| 5 | Claude Sonnet 5.5 | 84±1 | api $4.00/M | ||
$ what it's good at
agent tasks▁▁▂▆█▆▅▄▄▃▂▁ #145 of 340 · 22% of its score · 11 tests
Beats o3, Gemini 2.5 Pro, DeepSeek V3.2 Exp here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Large, GPT-4o (2024-05-13) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| τ²-bench Telecom +1 | 46.2% ✎ | ██··███···██ beats 4/6 | 5.5% |
| BrowserGym WorkArena-L2 (AgentLab GenericAgent) | 40.4% ✎ | ▃▅▃·▅▃▃··█·▃ beats 9/11 | 2.7% |
| τ-bench Retail lab | 80.5% ✎ | ▁····▃·▂▇█▂▆ beats 26/28 | 2.4% |
| BrowserGym WorkArena-L1 (AgentLab GenericAgent) | 63.3% ✎ | █···█▄▄·▄█·█ beats 10/10 | 2.1% |
| τ²-bench Retail | 80.0% ✎ | ▅···█·▃····▃ beats 6/6 | 2.0% |
| τ-bench Airline lab | 60.0% ✎ | ▁▁▁·▅▄▁█▇▁▁▇ beats 25/28 | 1.5% |
| τ²-bench Airline | 60.0% ✎ | ▄·▄··▄···█▄▄ beats 6/6 | 1.3% |
| BFCL V4 (2025-08-26 archive, models absent from later archives) | 70.3% ✎ | ▄▄····█▄···▄ beats 5/5 | 1.3% |
3 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BrowserGym MiniWoB (AgentLab GenericAgent) | 70.7% ✎ | ▅····▅▅▃▃·▃█ beats 9/11 | 1.1% |
| Terminal-Bench (pre-2.0 release) | 40.7% ✎ | ▄·▄···▄▄···█ beats 5/5 | 0.9% |
| Galileo Agent Leaderboard v2 (action completion) | 55.0% ✎ | ▅▃·▃▃▅▅█▃█▅▅ beats 16/19 | 0.7% |
coding▁▁▁▃▅▇█▇▅▄▃▁ #140 of 524 · 22% of its score · 6 tests
Beats Qwen3.5 Plus, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, Seed-OSS 36B Instruct, Gemma 4 26B-A4B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Aider polyglot (225 exercises, two attempts) | 61.3% ✎ | ▂·▂▃▅▂▃█▆▄▂▂ beats 36/49 | 10% |
| Luau Leetcode fully solved (Torpedo Software) | 68.3% ✎ | ▁▁▁▁·▁▂▃▅▃█▇ beats 20/49 | 3.1% |
| LiveCodeBench (version unspecified) | 45.6% ✎ | █·······▄▄▄▄ beats 2/5 | 2.9% |
| LiveCodeBench official (Aug 1, 2024–May 1, 2025) | 55.9% ✎ | ▃▃▅▃··▃▅█▅▅▃ beats 5/15 | 2.7% |
| MBPP+ | 80.2% ✎ | ▃·····▃█▃··▃ beats 5/6 | 1.9% |
| HumanEval+ | 94.5% ✎ | ▄▄▄·····▄▄·█ beats 5/6 | 1.8% |
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #127 of 463 * · 5.0% of its score · 2 tests
Beats LongCat 2.0 (1.6T-A48B), Step 3.5 Flash (196B-A11B), GPT-5 mini here, though they rank higher overall. Trails Nemotron-Cascade-8B, Gemini 2.5 Flash, Mistral Large 3 (675B-A41B) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| LMArena (text) | 1401.6 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 247/379 | 3.5% |
| IFEval (aggregation unspecified) | 88.3% ✎ | █··█·█··████ beats 3/6 | 1.4% |
math▁▂▃▅▇▅▅█▄▁▁▁ #195 of 462 · 9.4% of its score · 5 tests
Beats MiniMax-M2.1 here, though they rank higher overall. Trails OpenThinker3 7B, Olmo 3 7B Think, OpenThinker2 32B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| OTIS mock AIME 2024–2025 | 68.9% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 99/193 | 3.0% |
| MATH level 5 | 84.4% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 71/92 | 3.0% |
| AIME 2025 | 37.0% ✎ | ▄▄···▄▄··▄·█ beats 1/6 | 1.7% |
| AIME 2024 | 47.0% ✎ | ▁▁▁▂▁▃▁▅███▃ beats 20/85 | 0.9% |
| MATH-500 | 93.8% ✎ | ▄·▄··█··▄·▄▄ beats 1/6 | 0.8% |
vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #82 of 182 * · 2.4% of its score · 1 test
Beats DeepSeek V4 Flash Vision (Exp), GPT-5.4 nano, GPT-5 mini here, though they rank higher overall. Trails Gemma 4 12B, Qwen3.5 9B, Gemma 4 26B-A4B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena vision (style controlled) | 1206.2 ✎ | ▁▁▂▂▂▂▄▅▅▆██ beats 74/145 | 2.4% |
other tests21 not in a task · 39% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| GPQA Diamond +1 | 79.2% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 125/208 | 9.4% |
| MMLU | 91.8% ✎ | ▄·····█▄▄▄·▄ beats 6/6 | 3.7% |
| MMLU-Pro | 83.7% ✎ | █▄·▄···▄··▄▄ beats 4/6 | 3.0% |
| KCL-Essay vanilla | 35.5% ✎ | ▁···▁▃▂█▃▃▃▂ beats 14/23 | 2.7% |
| EQ-Bench 3 (emotional intelligence, judge Elo) | 1224.3 ✎ | ▁·▂▁▄▅▅▃█▄▁▁ beats 34/62 | 2.2% |
| CMMLU | 86.5% ✎ | █····▄··▄▄▄▄ beats 3/6 | 1.9% |
| AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge) | 75.6% ✎ | ▁▁·▁▂▂▄█▄▅▄▄ beats 59/66 | 1.7% |
| C-Eval | 86.6% ✎ | █····▄·▄·▄▄▄ beats 2/6 | 1.7% |
13 more tests
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Meituan LongCat-Flash-Chat mo… 37%, Epoch AI — internal evaluatio… 12%, Aider polyglot benchmark runs 10% and 15 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json