Claude 3 Haiku

+ contribute

Claude 3 Haiku is a closed-weights language model from Anthropic, released March 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-03-07). Dated API snapshot claude-3-haiku-20240307. Anthropic model deprecations page: retired April 20, 2026.

47.8±0.4comparison only: no current run option verifiedfrom 46 benchmark results across 22 sources

$ how it compares

The LLMs ranked either side of Claude 3 Haiku, out of 828 with a score, then more from Anthropic.

#	model	score	where	RAM	released
456	WaveCoder-Ultra-6.7B Microsoft	48±1	local	~5.8GB	2024-04
457	Phi-3.5-MoE-instruct Microsoft	48±1	local	~26GB	2024-08
458	Claude 3 Haiku Anthropic	48±0	comparison only		2024-03
459	GPT-4o mini Audio Preview (2024-12-17) OpenAI	48±2	comparison only		2024-12
460	Gemma 3n E4B Google	48±1	local	~6.2GB	2025-06
	more from Anthropic
2	Claude Opus 5.5 Anthropic	87±1	api $8.00/M		2026-09
4	Claude Fable 5.1 Anthropic	85±1	api $20.00/M		2026-09
5	Claude Sonnet 5.5 Anthropic	84±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▅▇█▇▅▄▃▁ #328 of 524 · 40% of its score · 6 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, Yi 1.5 9B Chat, Mistral NeMo Instruct (2407) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval +1	76.8% ✎	▁·▁▁▃▄▆▅█▃▁▁ beats 76/99	easy · avg 79%	▇▇▇▇▇▇▇▇ 19%	sharp · at its level · several runs
HumanEval+	68.9% ✎	▁·▁▁▁▃▄▄▄█▃▁ beats 67/99	easy · avg 74%	▇▇▇▇ 10%	sharp · at its level
CRUXEval (ZeroEval, 800 problems)	54.8% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 16/47	medium · avg 61%	▇▇ 4.9%	sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)	47.4% ✎	▁·▁▂▄▃▆▄█▆▂▂ beats 13/62	medium · avg 54%	▇ 3.2%	at its level
MBPP+	68.8% ✎	▁···▁▁▂▅▇█▆▂ beats 58/75	easy · avg 68%	▇ 1.9%	at its level
LiveCodeBench official (Aug 1, 2024–May 1, 2025)	20.2% ✎	▃▃▅▃··▃▅█▅▅▃ beats 0/15	hard · avg 25%	▇ 0.9%	typical

chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #310 of 463 · 9.3% of its score · 4 tests

Beats Ministral 3 14B Reasoning 2512, Ministral 3 14B, OpenThinker2 32B here, though they rank higher overall. Trails Flan-T5 XXL, Granite 4.0 H 1B, Granite 4.0 1B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	41.5% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 34/60	medium · avg 52%	▇▇ 4.6%	sharp · at its level
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)	1209.0 ✎	▂▁▂▁▃▇█▆▇▄▂▂ beats 83/95	avg 1201	▇ 1.7%	blunt
LMArena (text)	1261.6 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 97/379	avg 1268	▇ 1.7%	blunt
Arabic IFEval prompt-level (Inception)	53.7% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 29/38	medium · avg 49%	▇ 1.4%	sharp · at its level

math▁▂▃▅▇▅▅█▄▁▁▁ #389 of 462 · 8.6% of its score · 5 tests

Trails Qwen3 0.6B, Gemma 7B (base), LFM2 700M here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	14.9% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 12/92	hard · avg 32%	▇ 3.0%	sharp
GSM8K (ZeroEval, zero-shot)	88.8% ✎	▁▄·▁▄█▅▅▅███ beats 21/43	very easy · avg 89%	▇ 2.4%	sharp
MATH Level 5 (ZeroEval, 721 problems)	15.1% ✎	▁▁▁▆█▂▇▇▄··▂ beats 3/38	hard · avg 34%	▇ 2.0%	sharp · at its level
MathVista (testmini) +1	44.8% ✎	▂····▂···▃█▆ beats 2/9	medium · avg 52%	▇ 0.8%	at its level · small run · several runs
OTIS mock AIME 2024–2025	1.8% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 13/193	very hard · avg 8.5%	▇ 0.4%	sharp · too hard for it · noisy

vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #168 of 182 · 11% of its score · 16 tests

Trails Qwen3.5 0.8B, Qwen3-VL 2B Instruct, Qwen2.5-VL 3B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CharXiv descriptive questions	65.1% ✎	▄█▄▄▄·██·▄·█ beats 5/12	easy · avg 71%	▇ 2.1%	at its level
CharXiv descriptive questions (test split)	64.5% ✎	█··▄··█··▄·▄ beats 2/6	easy · avg 71%	▇ 1.2%	at its level
Arena vision (style controlled)	1000.0 ✎	▁▁▂▃▂▂▄▅▅▆██ beats 10/145	avg 1068	▇ 1.1%	blunt
MMMU val (VHELM, zero-shot exact match)	48.1% ✎	▄▄··▄▄█··▄▄█ beats 1/9	medium · avg 51%	▇ 1.1%	at its level
CharXiv (reasoning questions)	31.8% ✎	▂▂▂█▃▂·▂▂··▂ beats 5/13	medium · avg 36%	▇ 1.0%	at its level
MMMU (val) +1	50.4% ✎	▂·▂······█▄▄ beats 3/9	medium · avg 50%	▇ 1.0%	at its level · small run · several runs
MathVista testmini (VHELM, zero-shot exact match)	35.2% ✎	▃·▅···▃▃▃·▃█ beats 0/9	medium · avg 43%	▇ 0.9%	at its level
MathVista (testmini) +1	44.8% ✎	▂····▂···▃█▆ beats 2/9	medium · avg 52%	▇ 0.8%	at its level · small run · several runs

8 more tests

test	result	vs the others in its run	difficulty	counts	why
MM-MT-Bench (Mistral, GPT-4o judge)	5.4600 MM-MT-Bench score (1-10) ✎	▅··▃·█·▃▃··▅ beats 4/9	avg 5.740 MM-MT-Bench score (1-10)	▇ 0.6%	sharp · noisy
CharXiv reasoning questions (test split)	30.5% ✎	▄··▄▄▄█····▄ beats 3/6	hard · avg 32%	▇ 0.5%	blunt · at its level
VQAv2 (val)	68.4% ✎	▃···█▃··▅▃▃▃ beats 3/9	easy · avg 74%	▇ 0.3%	blunt · at its level
PhysBench physical-world understanding (test, Avg)	39.4% ✎	▅▃█··▅▃····▃ beats 3/9	hard · avg 43%	▇ 0.2%	blunt
Ai2 Molmo human preference Elo (Sept 2024, 27 VLMs)	999.0 ✎	▄▄·▄·▄▄▄·▄·█ beats 2/8	avg 1037	▇ 0.2%	blunt
ChartQA +1	69.6% ✎	▄·▄▄▄··▄▄▄▄█ beats 5/9	easy · avg 79%	▇ 0.1%	blunt · noisy · small run · several runs
TextVQA	62.7% ✎	▁▃▁▄▅▁▃·▃▃▃█ beats 1/24	easy · avg 76%	▇ 0.1%	blunt · small run
AI2D test	60.3% ✎	▃▁▅▇▃▇·▄·█·▃ beats 0/28	easy · avg 83%	▇ 0.0%	blunt · noisy · small run

other tests12 not in a task · 31% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Redux (ZeroEval, 2,778 questions)	72.3% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 16/53	easy · avg 77%	▇▇▇ 7.9%	at its level
LiveBench 2024-07-26	35.9% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 39/61	medium · avg 41%	▇▇▇ 6.8%	at its level
EvoEval (overall)	52.1% ✎	▂▂▃▄▄█▁▄▅▃▁▃ beats 36/42	medium · avg 55%	▇▇ 5.3%	at its level
BigCodeBench (Instruct)	39.2% ✎	▃▂·▆▆██▆▅▃▃▃ beats 23/33	medium · avg 41%	▇ 2.1%	at its level
ZebraLogic (ZeroEval, 1,000 puzzles)	14.3% ✎	▁▁▃▆██▁▁·▁▁▁ beats 20/57	hard · avg 18%	▇ 2.0%	sharp
BigCodeBench (Complete)	48.6% ✎	▂▁▁▄▂▅▇█▆▂▂▂ beats 48/58	medium · avg 50%	▇ 2.0%	at its level
WildBench v2 WB-Score (task macro)	38.8936 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 26/52	avg 45.199 WB-Score	▇ 1.9%	sharp
GPQA Diamond	36.3% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 30/208	hard · avg 40%	▇ 1.6%	sharp

4 more tests

test	result	vs the others in its run	difficulty	counts	why
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	55.3% ✎	▁▁▁▁·▂·█▆█▇▁ beats 11/31	medium · avg 48%	▇ 0.8%	at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)	74.2% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 214/276	medium · avg 73%	▇ 0.4%	at its level · noisy
ARC-AGI-1 public evaluation set (ARC Prize model_baseline, pass@2)	7.1% ✎	▁·▇▃▁·█▇▁·▇▃ beats 1/27	very hard · avg 4.1%	▇ 0.2%	sharp · too hard for it · noisy
ARC-AGI-2 public evaluation set (ARC Prize model_baseline, pass@2)	0.0% ✎	█··▂▅▁▃▁▃·▁▃ beats 4/24	very hard · avg 0.3%	▇ 0.0%	sharp · too hard for it · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · EvalPlus / EvalPerf published… 25%, WildEval ZeroEval published r… 19%, Top Leaderboard Ranking = Top… 12% and 19 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
EvalPlus / EvalPerf published evaluation runs	3	▇▇▇▇▇▇▇ 25%	Apache-2.0
WildEval ZeroEval published results	5	▇▇▇▇▇▇ 19%	Apache-2.0
Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM — authors' own runs	2	▇▇▇▇ 12%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇ 7%	Apache-2.0 (stated in the LiveBench datasheet)
Epoch AI — internal evaluations	3	▇ 5%	CC BY 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇ 5%	Apache-2.0
BigCodeBench paper (Zhuo et al., ICLR 2025), arXiv 2406.15877v4	2	▇ 4%	CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
CharXiv leaderboard (princeton-nlp/CharXiv README)	2	▇ 3%	Apache-2.0
LMArena leaderboard dataset	2	▇ 3%	CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	2	▇ 2%	Apache-2.0
VHELM: A Holistic Evaluation of Vision Language Models (Stanford CRFM), arXiv 2410.07112v2	2	▇ 2%	CC BY 4.0
WildBench v2 leaderboard data (Ai2)	1	▇ 2%	Apache-2.0
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (arXiv:2406.18521v1)	2	▇ 2%	CC BY-SA 4.0
Pixtral 12B (Mistral AI, arXiv 2410.07073v2)	5	▇ 2%	CC BY 4.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)	1	▇ 2%	CC-BY-4.0
Phi-3 Technical Report (Microsoft, arXiv 2404.14219v3)	5	▇ 1%	CC BY 4.0
LiveCodeBench official website results (2025-08-01)	1	▇ 1%	CC-BY-SA-4.0
Open Portuguese LLM Leaderboard (archived)	1	▇ 0%	Apache-2.0
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (arXiv:2501.16411v2)	1	▇ 0%	CC-BY-4.0
Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column	1	▇ 0%	Apache-2.0
ARC Prize Foundation model_baseline outputs on ARC-AGI-1/2 public evaluation sets	2	▇ 0%	MIT
prior · pull toward the middle		▇ 0%
weighted average		= 47.8