whatmodel.sh / GPT-4o (2024-05-13)

GPT-4o (2024-05-13)

+ contribute

GPT-4o (2024-05-13) is a closed-weights language model from OpenAI, released May 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-05-13). Dated API snapshot gpt-4o-2024-05-13 (not the default gpt-4o snapshot; the model page price is not stated per snapshot, so no price). OpenAI's deprecations page schedules its shutdown for October 23, 2026 (announced 2026-04-22).

57.0±0.4#236 of 719 LLMsfrom 59 benchmark results across 25 sources

$ how it compares

The LLMs ranked either side of GPT-4o (2024-05-13), out of 828 with a score, then more from OpenAI.

#	model	score	where	RAM	released
280	Qwen2.5-Max (2025-01-25) Alibaba	57±1	comparison only		2025-01
281	Qwen2.5-Omni 7B Alibaba	57±2	local	~6.4GB	2025-03
282	GPT-4o (2024-05-13) OpenAI	57±0	api $7.50/M		2024-05
283	QED-Nano LM Provers	57±2	local	~4.1GB	2026-02
284	Llama 4 Maverick (402B-A17B) Meta	57±1	api $0.42/M +local	~65GB	2025-04
	more from OpenAI
1	GPT-6 Astra OpenAI	87±1	api $20.00/M		2026-09
3	GPT-6.1 Sol OpenAI	85±1	api $4.00/M		2026-09
7	GPT-6 Sol OpenAI	83±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #129 of 300 * · 0.3% of its score · 1 test

Beats o3, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Large here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BFCL (version not stated)	82.9% ✎	█·········██ beats 5/5	easy · avg 78%	▇ 0.3%	blunt

coding▁▁▁▃▆██▇▆▄▃▁ #185 of 458 · 16% of its score · 8 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B, Qwen3 30B-A3B Instruct 2507 here, though they rank higher overall. Trails QwQ-32B-Preview, OpenThinker 32B, Command A Reasoning (08-2025) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LiveCodeBench official (Jan 1–Nov 1, 2024)	40.1% ✎	▁▁·▁▂▄▆█▅▂·▂ beats 23/31	hard · avg 26%	▇▇ 5.0%	sharp · at its level
CRUXEval (ZeroEval, 800 problems)	86.1% ✎	▁▂▆▃▆█▆▃▃·▁▁ beats 44/47	medium · avg 61%	▇ 3.6%	sharp
Aider code editing (133 Exercism Python exercises, two attempts)	72.9% ✎	▁·▁▂▄▁▆▄█▆▂▂ beats 56/62	medium · avg 54%	▇ 2.9%	at its level
HumanEval+	84.9% ✎	▁·▁▁▂▃▄▄█▅▄▃ beats 118/145	easy · avg 78%	▇ 1.7%	sharp
SciCode main problems (official leaderboard)	1.5% ✎	▃···█··▂·▃·▂ beats 9/17	very hard · avg 1.1%	▇ 1.1%	sharp · too hard for it
WebApp1K-React pass@1 (onekq-ai)	87.0% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 24/31	medium · avg 44%	▇ 0.7%	sharp · at its level · noisy
HumanEval (simple-evals) lab	91.0% ✎	▆█▆▆▂··▂···▂ beats 9/15	easy · avg 84%	▇ 0.6%	typical
LiveCodeBench v2	50.5% ✎	▄··█·▄···▄▄█ beats 2/7	medium · avg 50%	▇ 0.5%	at its level

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #170 of 393 · 16% of its score · 6 tests

Beats MiniCPM5 2B, Seed-OSS 36B Instruct, GPT-5 nano here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AlpacaEval 2 length-controlled	57.5% ✎	▁▂▂▃▆▆█▆▄▄▁▁ beats 115/122	medium · avg 39%	▇▇▇ 7.9%	at its level
Arena-Hard (v0.1, GPT-4-1106 judge)	79.2% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 52/60	medium · avg 52%	▇ 3.4%	sharp
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)	1297.0 ✎	▂▁▂▁▃▇█▆▇▄▂▃ beats 95/95	avg 1201	▇ 1.8%	blunt
LMArena (text)	1346.1 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 187/379	avg 1268	▇ 1.8%	blunt
Arena-Hard (version/judge unspecified)	69.0% ✎	▃▃▂▅▃▆▅█▅▅▂▃ beats 19/30	medium · avg 60%	▇ 1.1%	at its level
IFEval (aggregation unspecified)	81.8% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 59/141	easy · avg 75%	▇ 0.3%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #239 of 398 · 12% of its score · 9 tests

Beats Command A (03-2025), GPT-4o (2024-11-20) here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 70B, OpenThinker 7B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	51.0% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 44/92	hard · avg 32%	▇▇ 4.1%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	54.8% ✎	▁▁▁▆█▂▇▇▄··▂ beats 30/38	hard · avg 34%	▇ 2.7%	sharp · at its level
OTIS mock AIME 2024–2025	6.3% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 35/193	very hard · avg 8.5%	▇ 1.5%	sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot)	95.4% ✎	▁▄·▁▄█▅▅▅███ beats 36/43	very easy · avg 89%	▇ 1.0%	sharp · maxed out
MATH (simple-evals, original test set, pre-o1 rows) lab	76.6% ✎	█····▄··▄··█ beats 5/5	medium · avg 63%	▇ 0.8%	at its level
MathVista (testmini) +1	64.6% ✎	▂····▂···▂█▆ beats 8/9	medium · avg 53%	▇ 0.7%	at its level · several runs
MGSM (simple-evals) lab	89.9% ✎	▁····▁▅▁█▁▃▁ beats 9/16	easy · avg 82%	▇ 0.6%	blunt
AIME 2024	8.7% ✎	▁▁▁▃▁▃▁▅███▃ beats 8/85	medium · avg 42%	▇ 0.3%	sharp · at its level · noisy

1 more test

test	result	vs the others in its run	difficulty	counts	why
MATH-500	75.8% ✎	▁▃▂▃▅▇▅▇█▅▂▁ beats 19/84	easy · avg 84%	▇ 0.2%	noisy

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #98 of 147 · 11% of its score · 14 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4o (2024-08-06), Gemini 2.5 Flash-Lite here, though they rank higher overall. Trails Llama 4 Maverick (402B-A17B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CharXiv descriptive questions	84.5% ✎	▄█▄▄▄·██·▄·█ beats 12/12	easy · avg 71%	▇ 1.5%	typical
CharXiv (reasoning questions)	47.1% ✎	▂▂▂█▃▂·▂▃··▂ beats 12/13	medium · avg 36%	▇ 1.3%	at its level
Arena vision (style controlled)	1161.8 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 57/145	avg 1068	▇ 1.2%	blunt
MMMU (val) +1	68.6% ✎	▂·▂······█▄▄ beats 9/9	medium · avg 52%	▇ 1.2%	at its level · several runs
MMMU val (VHELM, zero-shot exact match)	64.0% ✎	▄▄··▄▄█··▄▄█ beats 8/9	medium · avg 51%	▇ 1.1%	at its level
MathVista testmini (VHELM, zero-shot exact match)	55.1% ✎	▃·▅···▃▃▃·▃█ beats 6/9	medium · avg 43%	▇ 1.0%	at its level
CharXiv descriptive questions (test split)	84.9% ✎	█··▄··█··▄·▄ beats 6/6	easy · avg 71%	▇ 0.9%	typical
MathVista (testmini) +1	64.6% ✎	▂····▂···▂█▆ beats 8/9	medium · avg 53%	▇ 0.7%	at its level · several runs

6 more tests

test	result	vs the others in its run	difficulty	counts	why
CharXiv reasoning questions (test split)	47.0% ✎	▄··▄▄▄█····▄ beats 6/6	hard · avg 32%	▇ 0.7%	blunt · at its level
MM-MT-Bench (Mistral, GPT-4o judge)	7.7200 MM-MT-Bench score (1-10) ✎	▅··▃·█·▃▃··▅ beats 9/9	avg 5.740 MM-MT-Bench score (1-10)	▇ 0.6%	sharp · noisy
VQAv2 (val)	77.8% ✎	▃···█▃··▅▃▃▃ beats 7/9	easy · avg 74%	▇ 0.3%	blunt
TextVQA	75.6% ✎	▁▁▁▄▅▁▃·▃▃▃█ beats 12/24	easy · avg 76%	▇ 0.1%	blunt
ChartQA +1	85.1% ✎	▄·▄▄▄··▄▄▄▄█ beats 8/9	easy · avg 80%	▇ 0.1%	blunt · noisy · several runs
AI2D test	82.8% ✎	▁▁▅▇▃▇·▄·█·▃ beats 13/28	easy · avg 83%	▇ 0.0%	blunt

other tests17 not in a task · 45% of its score

test	result	vs the others in its run	difficulty	counts	why
LiveBench 2024-07-26	54.6% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 59/61	medium · avg 41%	▇▇▇ 7.9%	at its level
MMLU-Redux (ZeroEval, 2,778 questions)	88.0% ✎	▁·▁▁▄▄▄▅█▄▃▁ beats 50/53	easy · avg 77%	▇▇ 5.6%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	28.2% ✎	▁▁▃▆██▁▁·▁▁▁ beats 45/57	hard · avg 18%	▇▇ 4.3%	sharp · at its level
GPQA Diamond +1	48.9% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 64/208	hard · avg 40%	▇▇ 4.3%	sharp · at its level · several runs
MMLU-Pro +1	72.1% ✎	▃▂▅█▃·█▃▅··▂ beats 19/23	medium · avg 57%	▇▇ 4.0%	at its level · several runs
GPQA Diamond (simple-evals) lab	49.9% ✎	█▃▃█·▃▃▅▃▃▃▅ beats 5/16	hard · avg 40%	▇▇ 4.0%	sharp · at its level
BigCodeBench (Instruct)	49.9% ✎	▃▂·▆▆██▆▅▃▃▃ beats 33/33	medium · avg 41%	▇ 2.4%	at its level
CLIcK Korean (Daekeun Kim)	80.5% ✎	▆▄▄▄█▄▂·▄█·▄ beats 13/23	medium · avg 58%	▇ 2.3%	at its level

9 more tests

test	result	vs the others in its run	difficulty	counts	why
BigCodeBench (Complete)	60.2% ✎	▂▁▁▄▂▅▇█▆▂▂▃ beats 58/58	medium · avg 50%	▇ 2.1%	at its level
WildBench v2 WB-Score (task macro)	59.2982 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 51/52	avg 45.199 WB-Score	▇ 2.1%	sharp
HAE-RAE 1.0 (Daekeun Kim)	85.7% ✎	▄▄▄▂▂█▂▄▂█▂▄ beats 13/22	medium · avg 63%	▇ 1.9%	at its level
KMMLU-HARD (Daekeun Kim, zero-shot)	39.6% ✎	█···▁▁▃▂▃·▆▂ beats 10/23	very hard · avg 29%	▇ 1.3%	sharp
MMLU (simple-evals, zero-shot CoT) lab	87.2% ✎	▂▂·█▄▄▂·█▂·▂ beats 8/16	easy · avg 80%	▇ 1.2%	typical
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	74.0% ✎	▁▁▁▁·▂·█▆█▇▁ beats 25/31	medium · avg 48%	▇ 0.9%	at its level
FlenQA (3K-token subset)	90.8% ✎	▄···▄····▄·█ beats 1/4	easy · avg 86%	▇ 0.7%	sharp · noisy
DROP F1, 3-shot (simple-evals) lab	83.7% ✎	▂▂▄▆▆█·▂···▄ beats 12/16	easy · avg 78%	▇ 0.2%	blunt
SimpleQA (simple-evals) lab	39.0% ✎	▄·▁▁▁▁··█▁·▁ beats 8/14	hard · avg 18%	▇ 0.2%	at its level · noisy

$ where to run it

Through an API

api · $7.50/M · $5 in / $15 out per 1M tokens · 128k context · developers.openai.com ↗ · price sourceUSD per 1M tokens, OpenAI API Standard processing, model gpt-4o-2024-05-13 (listed as its own row in the text-token pricing table; deprecated snapshot, shutdown scheduled 2026-10-23): input $5, output $15; no cached-input rate listed. Batch ($2.50/$7.50) and Priority ($8.75/$26.25) tiers excluded. Currency: USD. OpenAI official pricing page, observed 2026-10-07.

$ how the score was worked out · WildEval ZeroEval published r… 17%, Epoch AI — internal evaluatio… 9%, LiveBench model judgments (HF… 8% and 22 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇ 17%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇ 9%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇ 8%	Apache-2.0 (stated in the LiveBench datasheet)
AlpacaEval 1.0 and 2.0 leaderboards (tatsu-lab/alpaca_eval repository)	1	▇▇ 8%	Apache-2.0
OpenAI simple-evals README benchmark results (OpenAI's own models) · lab-reported	7	▇▇ 7%	MIT
Daekeun Kim Korean LLM benchmark runs	3	▇▇ 5%	MIT
Microsoft — Phi-4-reasoning Technical Report	5	▇▇ 5%	CC BY 4.0
LiveCodeBench official website results (2024-12-26)	1	▇ 5%	CC-BY-SA-4.0
BigCodeBench paper (Zhuo et al., ICLR 2025), arXiv 2406.15877v4	2	▇ 4%	CC BY 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇ 3%	Apache-2.0
LMArena leaderboard dataset	2	▇ 3%	CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
CharXiv leaderboard (princeton-nlp/CharXiv README)	2	▇ 3%	Apache-2.0
MMLU-Pro author-released evaluation runs	1	▇ 3%	Apache-2.0
open-thoughts/OpenThinker-7B model card	4	▇ 2%	Apache-2.0
VHELM: A Holistic Evaluation of Vision Language Models (Stanford CRFM), arXiv 2410.07112v2	2	▇ 2%	CC BY 4.0
WildBench v2 leaderboard data (Ai2)	1	▇ 2%	Apache-2.0
Pixtral 12B (Mistral AI, arXiv 2410.07073v2)	5	▇ 2%	CC BY 4.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)	1	▇ 2%	CC-BY-4.0
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (arXiv:2406.18521v1)	2	▇ 2%	CC BY-SA 4.0
Microsoft Phi-3.5-vision-instruct model card: Benchmarks table	5	▇ 1%	MIT
SciCode leaderboard (repository README)	1	▇ 1%	Apache-2.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	1	▇ 1%	Apache-2.0
WebApp1K models leaderboard (onekq-ai)	1	▇ 1%	MIT
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (Team GLM, arXiv 2406.12793v2)	1	▇ 0%	CC BY 4.0
prior · pull toward the middle		▇ 0%
weighted average		= 57.0