whatmodel.sh / Claude 3.5 Sonnet (2024-06-20)

Claude 3.5 Sonnet (2024-06-20)

+ contribute

Claude 3.5 Sonnet (2024-06-20) is a closed-weights language model from Anthropic, released June 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-06-20). Dated API snapshot claude-3-5-sonnet-20240620; distinct from the 20241022 snapshot. Anthropic model deprecations page: retired October 28, 2025.

56.8±0.5comparison only: no current run option verifiedfrom 44 benchmark results across 22 sources

$ how it compares

The LLMs ranked either side of Claude 3.5 Sonnet (2024-06-20), out of 828 with a score, then more from Anthropic.

#	model	score	where	RAM	released
283	QED-Nano LM Provers	57±2	local	~4.1GB	2026-02
284	Llama 4 Maverick (402B-A17B) Meta	57±1	api $0.42/M +local	~65GB	2025-04
285	Claude 3.5 Sonnet (2024-06-20) Anthropic	57±1	comparison only		2024-06
286	Qwen2.5 Coder 32B Instruct Alibaba	57±1	api $0.74/M +local	~21GB	2024-11
287	Qwen3 4B Thinking 2507 Alibaba	57±1	local	~3.9GB	2025-08
	more from Anthropic
2	Claude Opus 5.5 Anthropic	87±1	api $8.00/M		2026-09
4	Claude Fable 5.1 Anthropic	85±1	api $20.00/M		2026-09
5	Claude Sonnet 5.5 Anthropic	84±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▅▇█▇▅▄▃▁ #220 of 524 · 31% of its score · 9 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B, Qwen3 30B-A3B Instruct 2507 here, though they rank higher overall. Trails QwQ-32B-Preview, DeepSeek-V2.5, OpenThinker 32B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval	87.2% ✎	▁·▁▁▃▄▆▅█▃▁▁ beats 92/99	easy · avg 79%	▇▇▇ 6.9%	sharp
HumanEval+	81.7% ✎	▁·▁▁▁▃▄▄▄█▃▁ beats 93/99	easy · avg 74%	▇▇▇ 6.6%	sharp
LiveCodeBench official (Jan 1–Nov 1, 2024)	42.6% ✎	▁▁·▁▂▄▆█▅▂·▂ beats 27/31	hard · avg 26%	▇▇ 5.4%	sharp · at its level
CRUXEval (ZeroEval, 800 problems)	80.8% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 42/47	medium · avg 61%	▇▇ 3.9%	sharp
Aider code editing (133 Exercism Python exercises, two attempts)	77.4% ✎	▁·▁▂▄▁▆▄█▆▃▂ beats 58/62	medium · avg 54%	▇ 3.2%	at its level
MBPP+	74.3% ✎	▁···▁▁▂▅▇█▆▂ beats 70/75	easy · avg 68%	▇ 1.8%	typical
SciCode main problems (official leaderboard)	4.6% ✎	▃···█··▂·▃·▂ beats 14/17	very hard · avg 1.1%	▇ 1.2%	sharp · too hard for it
WebApp1K-Duo-React pass@1 (onekq-ai)	68.0% ✎	▁··▁·▁▁▁▆█▇█ beats 26/45	hard · avg 32%	▇ 1.0%	at its level

1 more test

test	result	vs the others in its run	difficulty	counts	why
WebApp1K-React pass@1 (onekq-ai)	88.1% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 26/31	medium · avg 44%	▇ 0.8%	sharp · at its level · noisy

chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #204 of 463 · 16% of its score · 4 tests

Beats Seed-OSS 36B Instruct, Nemotron-Cascade-14B-Thinking, Qwen3-VL 32B Instruct here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AlpacaEval 2 length-controlled	52.4% ✎	▁▂▂▃▆▆█▆▄▄▁▁ beats 110/122	medium · avg 39%	▇▇▇ 8.6%	at its level
Arena-Hard (v0.1, GPT-4-1106 judge)	79.3% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 53/60	medium · avg 52%	▇▇ 3.8%	sharp
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)	1286.0 ✎	▂▁▂▁▃▇█▆▇▄▂▃ beats 94/95	avg 1201	▇ 2.0%	blunt
LMArena (text)	1343.1 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 182/379	avg 1268	▇ 2.0%	blunt

math▁▂▃▅▇▅▅█▄▁▁▁ #280 of 462 · 11% of its score · 5 tests

Beats Command A (03-2025), GPT-4o (2024-11-20) here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 70B, OpenThinker 7B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	51.7% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 45/92	hard · avg 32%	▇▇ 4.5%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	51.9% ✎	▁▁▁▆█▂▇▇▄··▂ beats 27/38	hard · avg 34%	▇ 2.9%	sharp · at its level
OTIS mock AIME 2024–2025	6.5% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 37/193	very hard · avg 8.5%	▇ 1.6%	sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot)	95.6% ✎	▁▄·▁▄█▅▅▅███ beats 39/43	very easy · avg 89%	▇ 1.1%	sharp · maxed out
MathVista (testmini) +1	64.4% ✎	▂····▂···▂█▆ beats 7/9	medium · avg 53%	▇ 0.8%	at its level · several runs

vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #127 of 182 · 11% of its score · 14 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4o (2024-08-06) here, though they rank higher overall. Trails Llama 4 Scout (109B-A17B), Mistral Small 3.2 24B, Qwen3-VL 4B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CharXiv descriptive questions	84.3% ✎	▄█▄▄▄·██·▄·█ beats 11/12	easy · avg 71%	▇ 1.7%	typical
CharXiv (reasoning questions)	60.2% ✎	▂▂▂█▃▂·▂▂··▃ beats 13/13	medium · avg 36%	▇ 1.4%	at its level
Arena vision (style controlled)	1145.1 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 45/145	avg 1068	▇ 1.4%	blunt
MMMU (val) +1	68.0% ✎	▂·▂······█▄▄ beats 8/9	medium · avg 52%	▇ 1.3%	at its level · several runs
MMMU val (VHELM, zero-shot exact match)	65.5% ✎	▄▄··▄▄█··▄▄█ beats 9/9	medium · avg 51%	▇ 1.2%	at its level
MathVista testmini (VHELM, zero-shot exact match)	57.5% ✎	▃·▅···▃▃▃·▃█ beats 8/9	medium · avg 43%	▇ 1.1%	at its level
CapArena-Auto detailed image captioning (Score_Avg, GPT-4o judge)	21.5000 ✎	▃▃▃·▃█·▅▃▃▃▃ beats 9/12	avg -0.671	▇ 0.9%	typical
MathVista (testmini) +1	64.4% ✎	▂····▂···▂█▆ beats 7/9	medium · avg 53%	▇ 0.8%	at its level · several runs

6 more tests

test	result	vs the others in its run	difficulty	counts	why
MM-MT-Bench (Mistral, GPT-4o judge)	7.5000 MM-MT-Bench score (1-10) ✎	▅··▃·█·▃▃··▅ beats 8/9	avg 5.740 MM-MT-Bench score (1-10)	▇ 0.7%	sharp · noisy
VQAv2 (val)	70.7% ✎	▃···█▃··▅▃▃▃ beats 4/9	easy · avg 74%	▇ 0.3%	blunt
Ai2 Molmo human preference Elo (Sept 2024, 27 VLMs)	1069.0 ✎	▄▄·▄·▄▄▄·▄·█ beats 7/8	avg 1037	▇ 0.2%	blunt
TextVQA	70.5% ✎	▁▁▁▄▅▁▃·▃▃▃█ beats 7/24	easy · avg 76%	▇ 0.1%	blunt
ChartQA +1	87.6% ✎	▄·▄▄▄··▄▄▄▄█ beats 9/9	easy · avg 80%	▇ 0.1%	blunt · noisy · several runs
AI2D test	68.9% ✎	▁▃▅▇▃▇·▄·█·▃ beats 1/28	easy · avg 83%	▇ 0.0%	blunt

other tests10 not in a task · 31% of its score

test	result	vs the others in its run	difficulty	counts	why
LiveBench 2024-07-26	59.9% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 61/61	medium · avg 41%	▇▇▇ 8.6%	at its level
MMLU-Redux (ZeroEval, 2,778 questions)	86.0% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 46/53	easy · avg 77%	▇▇ 6.2%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	33.4% ✎	▁▁▃▆██▁▁·▁▁▁ beats 50/57	hard · avg 18%	▇▇ 4.7%	sharp · at its level
GPQA Diamond	54.0% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 76/208	hard · avg 40%	▇ 3.4%	sharp · at its level
MMLU-Pro	76.1% ✎	▃▂▅█▃·█▃▅··▂ beats 21/23	medium · avg 57%	▇ 3.1%	at its level
WildBench v2 WB-Score (task macro)	54.6951 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 46/52	avg 45.199 WB-Score	▇ 2.2%	sharp
KCL-Essay vanilla	33.7% ✎	▁···▁▃▂█▂▃▃▂ beats 12/23	hard · avg 18%	▇ 1.2%	at its level
EvalPerf correctness (118 tasks, 20240328)	88.0% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 26/29	easy · avg 76%	▇ 0.9%	typical

2 more tests

test	result	vs the others in its run	difficulty	counts	why
Judgemark v2 (LLM-as-judge for creative writing)	78.8% ✎	▃▃··▃▅▅···█▃ beats 9/10	medium · avg 55%	▇ 0.7%	at its level
KCL-MCQA vanilla	45.1% ✎	▂···█▂▆▆██▆▂ beats 14/23	very hard · avg 30%	▇ 0.6%	typical

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · WildEval ZeroEval published r… 19%, EvalPlus / EvalPerf published… 16%, Epoch AI — internal evaluatio… 10% and 19 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇ 19%	Apache-2.0
EvalPlus / EvalPerf published evaluation runs	4	▇▇▇▇▇ 16%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇ 10%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇ 9%	Apache-2.0 (stated in the LiveBench datasheet)
AlpacaEval 1.0 and 2.0 leaderboards (tatsu-lab/alpaca_eval repository)	1	▇▇▇ 9%	Apache-2.0
LiveCodeBench official website results (2024-12-26)	1	▇▇ 5%	CC-BY-SA-4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇ 4%	Apache-2.0
LMArena leaderboard dataset	2	▇ 3%	CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
CharXiv leaderboard (princeton-nlp/CharXiv README)	2	▇ 3%	Apache-2.0
MMLU-Pro author-released evaluation runs	1	▇ 3%	Apache-2.0
VHELM: A Holistic Evaluation of Vision Language Models (Stanford CRFM), arXiv 2410.07112v2	2	▇ 2%	CC BY 4.0
WildBench v2 leaderboard data (Ai2)	1	▇ 2%	Apache-2.0
Pixtral 12B (Mistral AI, arXiv 2410.07073v2)	5	▇ 2%	CC BY 4.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)	1	▇ 2%	CC-BY-4.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 2%	MIT
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities	2	▇ 2%	CC BY 4.0
Microsoft Phi-3.5-vision-instruct model card: Benchmarks table	5	▇ 1%	MIT
SciCode leaderboard (repository README)	1	▇ 1%	Apache-2.0
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (arXiv:2503.12329v1)	1	▇ 1%	CC-BY-4.0
Judgemark v2 results (EQ-Bench)	1	▇ 1%	MIT
Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 56.8