whatmodel.sh / Claude 3.5 Haiku

Claude 3.5 Haiku

+ contribute

Claude 3.5 Haiku is a closed-weights language model from Anthropic, released October 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-10-22). Dated API snapshot claude-3-5-haiku-20241022. Anthropic model deprecations page: retired February 19, 2026.

53.6±0.7comparison only: no current run option verifiedfrom 25 benchmark results across 15 sources

$ how it compares

The LLMs ranked either side of Claude 3.5 Haiku, out of 828 with a score, then more from Anthropic.

#	model	score	where	RAM	released
343	GPT-4 Turbo Preview (1106) OpenAI	54±1	comparison only		2023-11
344	GPT-4 Turbo Preview (0125) OpenAI	54±1	comparison only		2024-01
345	Claude 3.5 Haiku Anthropic	54±1	comparison only		2024-10
346	Llama 3.1 Tulu 3 70B Ai2	54±1	local	~43GB	2024-11
347	DeepSeek-V2.5 DeepSeek	54±1	local	~151GB	2024-09
	more from Anthropic
2	Claude Opus 5.5 Anthropic	87±1	api $8.00/M		2026-09
4	Claude Fable 5.1 Anthropic	85±1	api $20.00/M		2026-09
5	Claude Sonnet 5.5 Anthropic	84±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▂▆█▆▅▄▄▃▂▁ #276 of 340 · 4.1% of its score · 4 tests

Beats DeepSeek R1 0528 Qwen3 8B, Sky-T1 32B Preview, Magistral Small 1.0 (2506) here, though they rank higher overall. Trails Granite 4.0 1B, SmolLM3 3B, Phi-3.5-mini-instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	13.1% ✎	█·▃▃·█▅▅▅▅▃▅ beats 8/18	very hard · avg 5.8%	▇ 1.3%	sharp
BFCL V4 (2025-08-26 archive, models absent from later archives)	43.4% ✎	▄▄····█▄···▄ beats 2/5	hard · avg 27%	▇ 1.3%	at its level
BFCL v3	52.5% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 48/114	medium · avg 52%	▇ 0.9%	blunt · at its level
Galileo Agent Leaderboard v1 (tool selection quality)	76.5% ✎	▃▃▃▃▆▅▆▅█▂▅▂ beats 6/31	easy · avg 79%	▇ 0.7%	blunt

coding▁▁▁▃▅▇█▇▅▄▃▁ #215 of 524 · 22% of its score · 3 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B, Qwen3 30B-A3B Instruct 2507 here, though they rank higher overall. Trails QwQ-32B-Preview here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	68.8% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 26/47	medium · avg 61%	▇▇▇▇ 9.4%	sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)	75.2% ✎	▁·▁▂▄▁▆▄█▆▃▂ beats 57/62	medium · avg 54%	▇▇▇ 7.0%	at its level
Aider polyglot (225 exercises, two attempts)	28.0% ✎	▂·▂▃▅▃▃█▆▄▂▂ beats 15/49	very hard · avg 8.5%	▇▇ 6.1%	sharp

chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #218 of 463 * · 7.1% of its score · 2 tests

Beats Seed-OSS 36B Instruct, Nemotron-Cascade-14B-Thinking, Granite 4.2 8B here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1324.6 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 162/379	avg 1268	▇▇ 3.9%	blunt
Arabic IFEval prompt-level (Inception)	67.1% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 35/38	medium · avg 49%	▇ 3.3%	sharp · at its level

math▁▂▃▅▇▅▅█▄▁▁▁ #298 of 462 · 21% of its score · 4 tests

Beats Molmo 2 8B, Llama 3.3 70B Instruct, Mistral Large 2 (2407) here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), Tulu 2 DPO 70B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	46.4% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 38/92	hard · avg 32%	▇▇▇▇ 10%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	46.5% ✎	▁▁▁▆█▂▇▇▄··▂ beats 24/38	hard · avg 34%	▇▇ 5.9%	sharp · at its level
GSM8K (ZeroEval, zero-shot)	94.5% ✎	▁▄·▁▄█▅▅▅███ beats 34/43	very easy · avg 89%	▇ 3.1%	sharp · maxed out
OTIS mock AIME 2024–2025	4.3% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 28/193	very hard · avg 8.5%	▇ 2.2%	sharp · noisy

vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #130 of 182 * · 3.3% of its score · 2 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4o (2024-08-06), GPT-4 Turbo (2024-04-09) here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Arena vision (style controlled)	1126.8 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 38/145	avg 1068	▇ 2.7%	blunt
MSEarth-Bench Earth-science figure MCQ (Overall accuracy)	47.6% ✎	█·██····██·█ beats 0/5	hard · avg 48%	▇ 0.6%	blunt · at its level

web search▂·▂▇▅█▃▄▁▂▁▁ #12 of 92 * · 1.3% of its score · 1 test

Beats GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	13.1% ✎	█·▃▃·█▅▅▅▅▃▅ beats 8/18	very hard · avg 5.8%	▇ 1.3%	sharp

other tests10 not in a task · 42% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Redux (ZeroEval, 2,778 questions)	79.6% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 27/53	easy · avg 77%	▇▇▇▇▇▇ 15%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	18.7% ✎	▁▁▃▆██▁▁·▁▁▁ beats 25/57	hard · avg 18%	▇▇▇ 7.8%	sharp · at its level
MMLU-Pro	62.1% ✎	▃▂▅█▃·█▃▅··▂ beats 13/23	medium · avg 57%	▇▇▇ 6.5%	at its level
GPQA Diamond	38.1% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 35/208	hard · avg 40%	▇▇ 5.8%	sharp · at its level
KCL-Essay vanilla	21.2% ✎	▁···▁▃▂█▂▃▃▂ beats 3/23	hard · avg 18%	▇ 2.1%	typical
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	61.3% ✎	▁▁▁▁·▂·█▆█▇▁ beats 15/31	medium · avg 48%	▇ 2.0%	at its level
Judgemark v2 (LLM-as-judge for creative writing)	77.6% ✎	▃▃··▃▅▅···█▃ beats 8/10	medium · avg 55%	▇ 1.4%	at its level
KCL-MCQA vanilla	28.0% ✎	▂···█▂▆▆██▆▂ beats 1/23	very hard · avg 30%	▇ 0.9%	typical

2 more tests

test	result	vs the others in its run	difficulty	counts	why
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)	46.9% ✎	▁··▃▁▁▁▅█▅▁▅ beats 2/24	medium · avg 52%	▇ 0.8%	blunt · at its level
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	95.1% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 62/140	very easy · avg 94%	▇ 0.2%	blunt · maxed out · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · WildEval ZeroEval published r… 41%, Epoch AI — internal evaluatio… 18%, Aider code editing benchmark… 7% and 12 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇▇▇▇▇▇▇ 41%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇▇▇ 18%	CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇▇ 7%	Apache-2.0
LMArena leaderboard dataset	2	▇▇ 7%	CC BY 4.0
MMLU-Pro author-released evaluation runs	1	▇▇ 6%	Apache-2.0
Aider polyglot benchmark runs	1	▇▇ 6%	Apache-2.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	2	▇▇ 5%	Apache-2.0
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities	2	▇ 3%	CC BY 4.0
Berkeley Function Calling Leaderboard result archive	2	▇ 2%	Apache-2.0
Judgemark v2 results (EQ-Bench)	1	▇ 1%	MIT
Finance Agent Benchmark (Vals AI, arXiv:2508.00828v1)	1	▇ 1%	CC-BY-4.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)	1	▇ 1%	CC BY 4.0
Galileo Agent Leaderboard (v1 and v2 results)	1	▇ 1%	Apache-2.0
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs (arXiv:2505.20740v3)	1	▇ 1%	CC-BY-4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 53.6