whatmodel.sh / Claude 3.5 Sonnet (2024-10-22)
Claude 3.5 Sonnet (2024-10-22)
↗ share
+ contribute
Claude 3.5 Sonnet (2024-10-22) is a closed-weights language model from Anthropic, released October 2024. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-10-22). Dated API snapshot claude-3-5-sonnet-20241022; distinct from the 20240620 snapshot. Anthropic model deprecations page: retired October 28, 2025.
$ how it compares
The LLMs ranked either side of Claude 3.5 Sonnet (2024-10-22), out of 828 with a score, then more from Anthropic.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 236 |  Mistral Medium 3 (2505) | 60±1 | comparison only | ||
| 237 |  Qwen3 235B-A22B (Apr 2025, hybrid) | 60±1 | api $1.23/M +local | ||
| 238 | Claude 3.5 Sonnet (2024-10-22) | 60±1 | comparison only | ||
| 239 |  Granite 4.2 8B | 60±1 | api $0.11/M +local | ||
| 240 |  Solar Open 100B (102B-A12B) | 60±1 | local | ||
| more from Anthropic | |||||
| 2 | Claude Opus 5.5 | 87±1 | api $8.00/M | ||
| 4 | Claude Fable 5.1 | 85±1 | api $20.00/M | ||
| 5 | Claude Sonnet 5.5 | 84±1 | api $4.00/M | ||
$ what it's good at
agent tasks▁▁▂▆█▆▅▄▄▃▂▁ #236 of 340 · 7.9% of its score · 7 tests
Beats o1 (2024-12-17), o3-mini, DeepSeek R1 (Jan '25) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Granite 4.1 3B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BrowserGym WebArena (AgentLab GenericAgent) | 36.2% ✎ | █··▄·▄█·▄··▄ beats 6/7 | 2.1% |
| BrowserGym WorkArena-L1 (AgentLab GenericAgent) | 56.4% ✎ | █···█▄▄·▄█·█ beats 7/10 | 1.8% |
| BrowserGym WorkArena-L2 (AgentLab GenericAgent) | 39.1% ✎ | ▃▅▃·▅▃▃··█·▃ beats 8/11 | 1.5% |
| BrowserGym MiniWoB (AgentLab GenericAgent) | 69.8% ✎ | ▅····▅▅▃▃·▃█ beats 8/11 | 1.0% |
| BFCL v3 | 56.9% ✎ | ▁▁▁▁▁▃▅█▅▄▁▁ beats 66/114 | 0.6% |
| Multi-SWE-bench, 7 languages (MagentLess scaffold) | 4.7% ✎ | ▄···▄▄·▄▄·▄█ beats 3/7 | 0.6% |
| Galileo Agent Leaderboard v1 (tool selection quality) | 80.1% ✎ | ▃▃▃▃▆▅▆▅█▂▅▂ beats 8/31 | 0.4% |
coding▁▁▁▃▅▇█▇▅▄▃▁ #156 of 524 · 26% of its score · 8 tests
Beats Qwen3.5 Plus, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Aider polyglot (225 exercises, two attempts) | 51.6% ✎ | ▂·▂▃▅▂▃█▆▄▂▂ beats 27/49 | 8.1% |
| CRUXEval (ZeroEval, 800 problems) | 83.9% ✎ | ▁▂▆▃▆█▆▃▂·▁▁ beats 43/47 | 4.6% |
| Aider code editing (133 Exercism Python exercises, two attempts) | 84.2% ✎ | ▁·▁▂▄▁▆▄█▆▂▃ beats 62/62 | 4.0% |
| SciCode main problems (official leaderboard) | 4.6% ✎ | ▃···█··▂·▃·▂ beats 13/17 | 2.4% |
| LiveCodeBench official (Aug 1, 2024–May 1, 2025) | 36.4% ✎ | ▃▃▅▃··▃▅█▅▅▃ beats 4/15 | 2.4% |
| HumanEval (Vector Institute, Inspect Evals) | 94.5% ✎ | ▄·▄·█·█·█·▄▄ beats 6/9 | 1.8% |
| WebApp1K-Duo-React pass@1 (onekq-ai) | 75.0% ✎ | ▁··▁·▁▁▁▆█▇█ beats 38/45 | 1.5% |
| Multi-SWE-bench, 7 languages (MagentLess scaffold) | 4.7% ✎ | ▄···▄▄·▄▄·▄█ beats 3/7 | 0.6% |
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #162 of 463 · 13% of its score · 5 tests
Beats EXAONE 4.5 33B, INTELLECT-3, Nemotron 3 Super 120B-A12B here, though they rank higher overall. Trails LFM2.5 8B-A1B, LFM2.5 2.6B, Spark-X2.5 1.7B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge) | 33.0% ✎ | ▂·▃▅▅·▃█▆▂▂▂ beats 9/22 | 4.4% |
| Arena-Hard (v0.1, GPT-4-1106 judge) | 85.2% ✎ | ▂▁▁▅█▅▄▃▃▅▃▁ beats 58/60 | 4.1% |
| LMArena (text) | 1374.0 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 215/379 | 2.9% |
| IFEval prompt-level strict | 85.6% ✎ | ▄▄··▄·▄··█·▄ beats 4/6 | 1.2% |
| Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges) | 24.2% ✎ | ▆▂▂▆▂▂▃█▂▃·▂ beats 7/22 | 0.8% |
math▁▂▃▅▇▅▅█▄▁▁▁ #263 of 462 · 17% of its score · 5 tests
Trails OpenThinker 7B, Olmo 3 7B Instruct, Tulu 2 70B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH level 5 | 56.9% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 50/92 | 5.2% |
| MATH Level 5 (ZeroEval, 721 problems) | 59.4% ✎ | ▁▁▁▆█▂▇▇▄··▂ beats 32/38 | 4.0% |
| MATH (Vector Institute, Inspect Evals) | 79.4% ✎ | ▄··██▄··▄·▄█ beats 5/9 | 3.9% |
| OTIS mock AIME 2024–2025 | 8.5% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 49/193 | 3.0% |
| GSM8K (ZeroEval, zero-shot) | 96.7% ✎ | ▁▄·▁▄█▅▅▅███ beats 43/43 | 1.1% |
vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #109 of 182 * · 3.2% of its score · 2 tests
Beats DeepSeek V4 Flash Vision (Exp), GPT-5 nano here, though they rank higher overall. Trails Mistral Small 4 (119B-A6.5B) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena vision (style controlled) | 1160.7 ✎ | ▁▁▂▂▂▂▄▅▅▆██ beats 55/145 | 2.0% |
| Kaleidoscope (in-language multimodal exams, 18 languages, overall accuracy) | 62.9% ✎ | █·█···█·█·██ beats 5/5 | 1.1% |
other tests8 not in a task · 33% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MMLU-Redux (ZeroEval, 2,778 questions) | 88.9% ✎ | ▁·▁▁▄▄▄▅█▄▃▁ beats 52/53 | 7.6% |
| ZebraLogic (ZeroEval, 1,000 puzzles) | 36.2% ✎ | ▁▁▃▆██▃▁·▁▁▁ beats 51/57 | 7.5% |
| GPQA Diamond | 55.3% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 77/208 | 5.4% |
| GPQA Diamond (Vector Institute, Inspect Evals) | 61.0% ✎ | ▃··▃▃▃▃·▃·▃█ beats 5/9 | 4.7% |
| MMLU-Pro | 77.6% ✎ | ▃▂▅█▃·█▃▅··▂ beats 22/23 | 4.2% |
| MMLU-Pro (Vector Institute, Inspect Evals) | 77.6% ✎ | ▃····▃▅·▃▅·█ beats 5/9 | 2.3% |
| HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge) | 77.5% ✎ | ▁▁▁▁·▂·█▆█▇▁ beats 29/31 | 1.3% |
| Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate) | 95.4% ✎ | ▁▁▂▃▄▄█▆▃▂▂▁ beats 71/140 | 0.1% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · WildEval ZeroEval published r… 25%, Epoch AI — internal evaluatio… 14%, Vector Institute Evaluation L… 13% and 17 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json