whatmodel.sh / Claude 3 Sonnet
Claude 3 Sonnet
↗ share
+ contribute
Claude 3 Sonnet is a closed-weights language model from Anthropic, released February 2024. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-02-29). Dated API snapshot claude-3-sonnet-20240229. Anthropic model deprecations page: retired July 21, 2025.
$ how it compares
The LLMs ranked either side of Claude 3 Sonnet, out of 828 with a score, then more from Anthropic.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 438 |  MegaBeam-Mistral-7B-512k | 49±4 | local | ||
| 439 |  GPT-4o Audio Preview (2024-12-17) | 49±2 | comparison only | ||
| 440 | Claude 3 Sonnet | 49±1 | comparison only | ||
| 441 |  AceReason-Nemotron-1.1-7B | 49±2 | local | ||
| 442 |  Mixtral 8x22B Instruct v0.1 | 49±1 | local | ||
| more from Anthropic | |||||
| 2 | Claude Opus 5.5 | 87±1 | api $8.00/M | ||
| 4 | Claude Fable 5.1 | 85±1 | api $20.00/M | ||
| 5 | Claude Sonnet 5.5 | 84±1 | api $4.00/M | ||
$ what it's good at
coding▁▁▁▃▅▇█▇▅▄▃▁ #348 of 524 · 40% of its score · 7 tests
Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, WizardLM 70B V1.0, Yi 1.5 9B Chat here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| HumanEval | 70.7% ✎ | ▁·▁▁▃▄▆▅█▃▁▁ beats 59/99 | 13% |
| HumanEval+ | 64.0% ✎ | ▁·▁▁▁▃▄▄▄█▃▁ beats 58/99 | 11% |
| CRUXEval (ZeroEval, 800 problems) | 66.6% ✎ | ▁▂▆▃▆█▆▃▂·▁▁ beats 24/47 | 5.4% |
| LiveCodeBench official (Jan 1–Nov 1, 2024) | 25.0% ✎ | ▁▁·▁▂▄▆█▅▂·▂ beats 10/31 | 3.8% |
| Aider code editing (133 Exercism Python exercises, two attempts) | 54.9% ✎ | ▁·▁▂▄▁▆▄█▆▂▂ beats 22/62 | 3.6% |
| MBPP+ | 69.3% ✎ | ▁···▁▁▂▅▇█▆▂ beats 62/75 | 2.1% |
| SciCode main problems (official leaderboard) | 1.5% ✎ | ▃···█··▂·▃·▂ beats 5/17 | 0.4% |
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #300 of 463 · 17% of its score · 4 tests
Beats Mistral Small 4 (119B-A6.5B), Ministral 3 14B Reasoning 2512, Qwen2.5 Coder 32B Instruct here, though they rank higher overall. Trails Flan-T5 XXL, Granite 4.0 H 1B, Granite 4.0 1B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| AlpacaEval 2 length-controlled | 34.9% ✎ | ▁▂▂▃▆▆█▆▄▄▁▁ beats 89/122 | 7.6% |
| Arena-Hard (v0.1, GPT-4-1106 judge) | 46.8% ✎ | ▂▁▁▅█▅▄▃▃▅▃▁ beats 36/60 | 5.2% |
| Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024) | 1220.0 ✎ | ▂▁▂▁▃▇█▆▇▄▂▂ beats 87/95 | 1.9% |
| LMArena (text) | 1281.3 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 112/379 | 1.9% |
math▁▂▃▅▇▅▅█▄▁▁▁ #376 of 462 · 6.7% of its score · 3 tests
Beats Athene-70B here, though they rank higher overall. Trails Gemma 7B (base), DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 13B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH level 5 | 18.2% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 16/92 | 3.7% |
| GSM8K (ZeroEval, zero-shot) | 91.5% ✎ | ▁▄·▁▄█▅▅▅███ beats 25/43 | 2.5% |
| OTIS mock AIME 2024–2025 | 2.5% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 19/193 | 0.5% |
vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #160 of 182 · 9.2% of its score · 8 tests
Beats Nova Lite here, though they rank higher overall. Trails InternVL2-26B, Qwen3.5 2B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| CharXiv descriptive questions | 73.7% ✎ | ▄█▄▄▄·██·▄·█ beats 8/12 | 2.2% |
| Arena vision (style controlled) | 1016.4 ✎ | ▁▁▂▃▂▂▄▅▅▆██ beats 13/145 | 1.3% |
| CharXiv descriptive questions (test split) | 72.8% ✎ | █··▄··█··▄·▄ beats 4/6 | 1.3% |
| MMMU val (VHELM, zero-shot exact match) | 44.7% ✎ | ▄▄··▄▄█··▄▄█ beats 0/9 | 1.2% |
| CharXiv (reasoning questions) | 32.2% ✎ | ▂▂▂█▃▂·▂▂··▂ beats 6/13 | 1.2% |
| MathVista testmini (VHELM, zero-shot exact match) | 39.3% ✎ | ▃·▅···▃▃▃·▃█ beats 1/9 | 1.0% |
| CharXiv reasoning questions (test split) | 32.4% ✎ | ▄··▄▄▄█····▄ beats 4/6 | 0.6% |
| PhysBench physical-world understanding (test, Avg) | 36.2% ✎ | ▅▃█··▅▃····▃ beats 0/9 | 0.2% |
other tests8 not in a task · 28% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MMLU-Redux (ZeroEval, 2,778 questions) | 74.9% ✎ | ▁·▁▁▄▄▄▅█▄▂▁ beats 19/53 | 8.7% |
| LiveBench 2024-07-26 | 38.7% ✎ | ▁▁▁▂▂▃▂█▆▃▅▃ beats 42/61 | 7.9% |
| ZebraLogic (ZeroEval, 1,000 puzzles) | 18.7% ✎ | ▁▁▃▆██▁▁·▁▁▁ beats 24/57 | 2.5% |
| BigCodeBench (Instruct) | 42.5% ✎ | ▃▂·▆▆██▆▅▃▃▃ beats 28/33 | 2.4% |
| BigCodeBench (Complete) | 53.4% ✎ | ▂▁▁▄▂▅▇█▆▂▃▂ beats 53/58 | 2.3% |
| WildBench v2 WB-Score (task macro) | 45.4815 WB-Score ✎ | ▁·▁▁·▃█▄▇▆▅▆ beats 34/52 | 2.1% |
| GPQA Diamond | 40.6% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 40/208 | 2.0% |
| Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate) | 83.7% ✎ | ▁▁▃▃▄▄█▆▃▂▂▁ beats 10/140 | 0.1% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · EvalPlus / EvalPerf published… 26%, WildEval ZeroEval published r… 19%, LiveBench model judgments (HF… 8% and 15 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json