whatmodel.sh / GPT-4o (2024-08-06)

GPT-4o (2024-08-06)

+ contribute

GPT-4o (2024-08-06) is a closed-weights language model from OpenAI, released August 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-08-06). Dated API snapshot gpt-4o-2024-08-06 (default snapshot of 'gpt-4o'). No shutdown listed on the deprecations page.

57.4±0.5#229 of 719 LLMsfrom 56 benchmark results across 29 sources

$ how it compares

The LLMs ranked either side of GPT-4o (2024-08-06), out of 828 with a score, then more from OpenAI.

#	model	score	where	RAM	released
271	Nemotron 3 Nano 30B-A3B NVIDIA	58±1	api $0.087/M +local	~20GB	2025-12
272	Ministral 3 14B Reasoning 2512 Mistral	58±1	local	~10GB	2025-12
273	GPT-4o (2024-08-06) OpenAI	57±1	api $4.38/M		2024-08
274	Gemini 2.0 Flash Thinking Experimental (01-21) Google	57±2	comparison only		2025-01
275	DeepSeek R1 0528 Qwen3 8B DeepSeek	57±1	api $0.068/M +local	~6.3GB	2025-05
	more from OpenAI
1	GPT-6 Astra OpenAI	87±1	api $20.00/M		2026-09
3	GPT-6.1 Sol OpenAI	85±1	api $4.00/M		2026-09
7	GPT-6 Sol OpenAI	83±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #189 of 300 · 6.5% of its score · 6 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Mellum2 12B-A2.5B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
ContractBench success rate (33 observation-contract tool tasks)	23.2% ✎	▆··▆▄▂▄▂▆▆█▄ beats 7/23	very hard · avg 14%	▇ 1.5%	sharp · at its level
BrowserGym WebArena (AgentLab GenericAgent)	31.4% ✎	█··▄·▄█·▄··▄ beats 5/7	hard · avg 18%	▇ 1.3%	at its level
BrowserGym WorkArena-L1 (AgentLab GenericAgent)	45.5% ✎	█···█▄▄·▄█·█ beats 5/10	hard · avg 29%	▇ 1.2%	at its level
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	20.0% ✎	█·▃▃·█▅▅▅▅▃▅ beats 10/18	very hard · avg 5.8%	▇ 1.0%	sharp
BrowserGym WorkArena-L2 (AgentLab GenericAgent)	8.5% ✎	▃▅▃·▅▃▃··█·▃ beats 5/11	very hard · avg 3.9%	▇ 0.8%	sharp · noisy
BrowserGym MiniWoB (AgentLab GenericAgent)	63.8% ✎	▅····▅▅▃▃·▃█ beats 2/11	medium · avg 55%	▇ 0.7%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #186 of 458 · 16% of its score · 7 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B, Qwen3 30B-A3B Instruct 2507 here, though they rank higher overall. Trails QwQ-32B-Preview, OpenThinker 32B, Command A Reasoning (08-2025) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts)	23.1% ✎	▂·▂▃▅▂▃█▆▄▂▂ beats 14/49	very hard · avg 8.5%	▇▇ 4.7%	sharp · at its level
CRUXEval (ZeroEval, 800 problems)	87.0% ✎	▁▂▆▃▆█▆▃▃·▁▁ beats 45/47	medium · avg 61%	▇▇ 3.8%	sharp
Aider code editing (133 Exercism Python exercises, two attempts)	71.4% ✎	▁·▁▂▄▁▆▄█▆▂▂ beats 52/62	medium · avg 54%	▇ 3.1%	at its level
LiveCodeBench official (Aug 1, 2024–May 1, 2025)	29.5% ✎	▃▃▅▃··▃▅█▅▅▃ beats 3/15	hard · avg 25%	▇ 1.6%	at its level
HumanEval (Vector Institute, Inspect Evals)	90.9% ✎	▄·▄·█·█·█·▄▄ beats 4/9	easy · avg 79%	▇ 1.5%	typical
WebApp1K-React pass@1 (onekq-ai)	88.5% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 27/31	medium · avg 44%	▇ 0.8%	sharp · at its level · noisy
HumanEval (simple-evals) lab	90.2% ✎	▆█▆▆▂··▂···▂ beats 7/15	easy · avg 84%	▇ 0.6%	typical

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #176 of 393 · 7.1% of its score · 3 tests

Beats Seed-OSS 36B Instruct, Nemotron-Cascade-14B-Thinking, Qwen3-VL 32B Instruct here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	77.9% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 49/60	medium · avg 52%	▇ 3.6%	sharp
LMArena (text)	1335.5 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 172/379	avg 1268	▇ 2.0%	blunt
Arabic IFEval prompt-level (Inception)	70.8% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 38/38	medium · avg 49%	▇ 1.5%	sharp · at its level

math▁▂▃▅▆▅▄█▄▁▁▁ #235 of 398 · 17% of its score · 8 tests

Trails Tulu 2 DPO 70B, OpenThinker 7B, Olmo 3 7B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	53.3% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 47/92	hard · avg 32%	▇▇ 4.4%	sharp · at its level
MATH (Vector Institute, Inspect Evals)	70.5% ✎	▄··██▄··▄·▄█ beats 4/9	medium · avg 53%	▇ 3.2%	sharp
MATH Level 5 (ZeroEval, 721 problems)	55.3% ✎	▁▁▁▆█▂▇▇▄··▂ beats 31/38	hard · avg 34%	▇ 3.0%	sharp · at its level
FrontierMath tiers 1–3 v2	0.4% ✎	▃··▁▁▃▄▄█▅▃▃ beats 0/82	very hard · avg 1.2%	▇ 1.9%	sharp · too hard for it
OTIS mock AIME 2024–2025	6.4% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 36/193	very hard · avg 8.5%	▇ 1.8%	sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot)	96.2% ✎	▁▄·▁▄█▅▅▅███ beats 42/43	very easy · avg 89%	▇ 1.0%	sharp · maxed out
MATH (simple-evals, original test set, pre-o1 rows) lab	75.9% ✎	█····▄··▄··█ beats 4/5	medium · avg 63%	▇ 0.9%	at its level
MGSM (simple-evals) lab	90.0% ✎	▁····▁▅▁█▁▃▁ beats 11/16	easy · avg 82%	▇ 0.6%	blunt

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #106 of 147 · 7.6% of its score · 7 tests

Beats DeepSeek V4 Flash Vision (Exp) here, though they rank higher overall. Trails Mistral Small 3.1 24B (2503), Llama 4 Scout (109B-A17B), Qwen3-VL 8B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena vision (style controlled)	1118.3 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 33/145	avg 1068	▇ 1.4%	blunt
MMMU val (VHELM, zero-shot exact match)	63.0% ✎	▄▄··▄▄█··▄▄█ beats 7/9	medium · avg 51%	▇ 1.2%	at its level
CharXiv (reasoning questions)	45.9% ✎	▂▅▃▅▅▆▆▅▆█▆▂ beats 23/100	hard · avg 33%	▇ 1.2%	at its level · small run
MathVista testmini (VHELM, zero-shot exact match)	56.7% ✎	▃·▅···▃▃▃·▃█ beats 7/9	medium · avg 43%	▇ 1.1%	at its level
CharXiv descriptive questions	79.5% ✎	▂▄▄▅▇▄█▇▇▂▁▂ beats 25/58	easy · avg 66%	▇ 1.0%	small run
CapArena-Auto detailed image captioning (Score_Avg, GPT-4o judge)	44.0000 ✎	▃▃▃·▃█·▅▃▃▃▃ beats 11/12	avg -0.671	▇ 0.9%	typical
Kaleidoscope (in-language multimodal exams, 18 languages, overall accuracy)	58.3% ✎	█·█···█·█·██ beats 4/5	medium · avg 47%	▇ 0.8%	at its level

web search▂▁▂▆▄█▃▃▃▁·▁ #6 of 80 * · 1.0% of its score · 1 test

Beats GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	20.0% ✎	█·▃▃·█▅▅▅▅▃▅ beats 10/18	very hard · avg 5.8%	▇ 1.0%	sharp

other tests25 not in a task · 46% of its score

test	result	vs the others in its run	difficulty	counts	why
LiveBench 2024-07-26	56.7% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 60/61	medium · avg 41%	▇▇▇ 8.7%	at its level
MMLU-Redux (ZeroEval, 2,778 questions)	88.3% ✎	▁·▁▁▄▄▄▅█▄▃▁ beats 51/53	easy · avg 77%	▇▇ 6.0%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	31.7% ✎	▁▁▃▆██▁▁·▁▁▁ beats 49/57	hard · avg 18%	▇▇ 4.8%	sharp · at its level
GPQA Diamond (simple-evals) lab	53.1% ✎	█▃▃█·▃▃▅▃▃▃▅ beats 7/16	hard · avg 40%	▇▇ 4.4%	sharp · at its level
GPQA Diamond	49.2% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 69/208	hard · avg 40%	▇ 3.5%	sharp · at its level
MMLU-Pro	74.7% ✎	▃▂▅█▃·█▃▅··▂ beats 20/23	medium · avg 57%	▇ 3.1%	at its level
GPQA Diamond (Vector Institute, Inspect Evals)	51.0% ✎	▃··▃▃▃▃·▃·▃█ beats 4/9	hard · avg 41%	▇ 3.1%	sharp · at its level
MMLU-Pro (Vector Institute, Inspect Evals)	74.5% ✎	▃····▃▅·▃▅·█ beats 4/9	medium · avg 60%	▇ 1.7%	at its level

17 more tests

test	result	vs the others in its run	difficulty	counts	why
MMLU (simple-evals, zero-shot CoT) lab	88.7% ✎	▂▂·█▄▄▃·█▂·▂ beats 10/16	easy · avg 80%	▇ 1.3%	typical
KCL-Essay vanilla	24.3% ✎	▁···▁▃▃█▂▃▃▂ beats 5/23	hard · avg 18%	▇ 1.3%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	44.2% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 91/97	hard · avg 32%	▇ 1.1%	blunt · at its level
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	74.5% ✎	▁▁▁▁·▂·█▆█▇▁ beats 26/31	medium · avg 48%	▇ 1.0%	at its level
ARC-AGI-1 public evaluation set (ARC Prize model_baseline, pass@2)	11.4% ✎	▁·▇▃▁·█▇▁·▇▃ beats 4/27	very hard · avg 4.1%	▇ 0.9%	sharp · noisy
Polish EQ-Bench v2 (SpeakLeash)	75.1% ✎	▁··▁▁▂▂▆▅▆█▆ beats 66/70	medium · avg 64%	▇ 0.8%	typical
EvalPerf correctness (118 tasks, 20240328)	86.7% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 24/29	easy · avg 76%	▇ 0.8%	typical
Open Persian LLM Leaderboard v2 average (21 tasks)	76.3% ✎	▂▄▂▃█▄▃▆▃▅▂▃ beats 80/81	medium · avg 58%	▇ 0.8%	blunt · at its level
KCL-MCQA vanilla	43.3% ✎	▂···█▂▆▆██▆▂ beats 12/23	very hard · avg 30%	▇ 0.6%	typical
Icelandic LLM Leaderboard average (Miðeind, Sept 2026 suite)	41.1% ✎	▁▁▃▂▂▄▁▃▄▄▄█ beats 15/38	very hard · avg 14%	▇ 0.5%	typical
SimpleQA Verified	26.0% ✎	▁▁▃▂▃▃▆▆█▃▂▅ beats 17/78	very hard · avg 8.0%	▇ 0.4%	typical
Open PT-LLM Leaderboard average (9 Portuguese tasks)	83.8% ✎	▃▁▁▁▁▁▁▁▂█▅▃ beats 272/276	medium · avg 73%	▇ 0.4%	at its level · noisy
Chess Puzzles (Epoch)	13.0% ✎	█▁▁▁▁▃▁▄▂▂▁▁ beats 63/135	very hard · avg 5.9%	▇ 0.2%	sharp · too hard for it · noisy
DROP F1, 3-shot (simple-evals) lab	79.8% ✎	▂▂▄▆▆█·▂···▄ beats 4/16	easy · avg 78%	▇ 0.2%	blunt
SimpleQA (simple-evals) lab	40.1% ✎	▄·▁▁▁▁··█▁·▁ beats 9/14	hard · avg 18%	▇ 0.2%	at its level · noisy
ARC-AGI-2 public evaluation set (ARC Prize model_baseline, pass@2)	0.0% ✎	█··▂▅▁▃▁▃·▁▃ beats 3/24	very hard · avg 0.3%	▇ 0.1%	sharp · too hard for it · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)	90.4% ✎	▁▂▁▂▅█▂▃▄▃▁▁ beats 47/98	very easy · avg 90%	▇ 0.0%	blunt

$ where to run it

Through an API

api · $4.38/M · $2.5 in / $10 out per 1M tokens · 128k context · developers.openai.com ↗ · price sourceUSD per 1M tokens, OpenAI API standard processing, model page 'gpt-4o' whose default snapshot is gpt-4o-2024-08-06: input $2.5, output $10; cached-input, Batch and other tiers excluded. Currency: USD. OpenAI official model page, observed 2026-09-28.

$ how the score was worked out · WildEval ZeroEval published r… 19%, Epoch AI — internal evaluatio… 12%, Vector Institute Evaluation L… 9% and 26 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇ 19%	Apache-2.0
Epoch AI — internal evaluations	6	▇▇▇▇ 12%	CC BY 4.0
Vector Institute Evaluation Leaderboard (Inspect Evals runs)	4	▇▇▇ 9%	Apache-2.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇ 9%	Apache-2.0 (stated in the LiveBench datasheet)
OpenAI simple-evals README benchmark results (OpenAI's own models) · lab-reported	7	▇▇ 8%	MIT
Aider polyglot benchmark runs	1	▇ 5%	Apache-2.0
ServiceNow BrowserGym leaderboard (AgentLab GenericAgent runs)	4	▇ 4%	MIT
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇ 4%	Apache-2.0
LMArena leaderboard dataset	2	▇ 3%	CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
MMLU-Pro author-released evaluation runs	1	▇ 3%	Apache-2.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	2	▇ 2%	Apache-2.0
VHELM: A Holistic Evaluation of Vision Language Models (Stanford CRFM), arXiv 2410.07112v2	2	▇ 2%	CC BY 4.0
Meridian Labs inspect_charxiv evaluation report (GitHub README)	2	▇ 2%	MIT
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities	2	▇ 2%	CC BY 4.0
LiveCodeBench official website results (2025-08-01)	1	▇ 2%	CC-BY-SA-4.0
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)	1	▇ 2%	CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
ARC Prize Foundation model_baseline outputs on ARC-AGI-1/2 public evaluation sets	2	▇ 1%	MIT
Finance Agent Benchmark (Vals AI, arXiv:2508.00828v1)	1	▇ 1%	CC-BY-4.0
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (arXiv:2503.12329v1)	1	▇ 1%	CC-BY-4.0
SpeakLeash Polish EQ-Bench leaderboard	1	▇ 1%	Apache-2.0
EvalPlus / EvalPerf published evaluation runs	1	▇ 1%	Apache-2.0
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (arXiv:2504.07072v2)	1	▇ 1%	CC-BY-4.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
WebApp1K models leaderboard (onekq-ai)	1	▇ 1%	MIT
Miðeind Icelandic LLM Leaderboard	1	▇ 0%	CC BY 4.0
Open Portuguese LLM Leaderboard (archived)	1	▇ 0%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 57.4