whatmodel.sh / o3
o3
↗ share
+ contribute
o3 is a closed-weights language model from OpenAI, released April 2025.
$ how it compares
The LLMs ranked either side of o3, out of 828 with a score, then more from OpenAI.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 100 |  Ring-2.6-1T | 69±2 | comparison only | ||
| 101 |  A.X K2 (688B-A33B) | 69±1 | local | ||
| 102 | o3 | 69±1 | api $3.50/M | ||
| 103 |  MiMo-V2.5 (310B-A15B) | 69±1 | api $0.21/M +local | ||
| 104 |  Qwen3.5 Plus | 69±2 | api $0.90/M | ||
| more from OpenAI | |||||
| 1 | GPT-6 Astra | 87±1 | api $20.00/M | ||
| 3 | GPT-6.1 Sol | 85±1 | api $4.00/M | ||
| 7 | GPT-6 Sol | 83±1 | api $4.00/M | ||
$ what it's good at
agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #133 of 300 · 19% of its score · 7 tests
Trails SmolLM3 3B, Jamba 1.5 Large, GPT-4o (2024-05-13) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| τ²-bench Telecom lab +2 | 58.2% ✎ | ▃·▅▅▃▃··█▃▃▅ beats 5/13 | 5.7% |
| SWE-bench Verified | 62.3% ✎ | ▁··▁·▁▃▁▄█▄▁ beats 5/29 | 5.3% |
| τ²-bench Retail +1 | 72.8% ✎ | █···█·█·██·█ beats 5/5 | 2.5% |
| Finance Agent Benchmark v1 (537 questions, class-balanced accuracy) | 46.8% ✎ | █·▃▃·█▅▅▅▅▃▅ beats 18/18 | 2.3% |
| τ²-bench Airline +1 | 62.5% ✎ | ▄·····█··▄·█ beats 3/5 | 1.5% |
| BFCL V4 (December 2025 archive) | 48.6% ✎ | ▁▁▁▂██▅▅▆▂▂▂ beats 55/77 | 1.1% |
| Galileo Agent Leaderboard v1 (tool selection quality) | 86.1% ✎ | ▃▃▃▃▆▅▆▅█▂▅▂ beats 15/31 | 0.2% |
coding▁▁▁▃▆██▇▆▄▃▁ #94 of 458 · 14% of its score · 6 tests
Beats Grok 4.20 (0309 reasoning), Grok 4.3, Muse Glimmer 30B here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, o4-mini, Qwen3 235B-A22B Thinking 2507 here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| SWE-bench Verified | 62.3% ✎ | ▁··▁·▁▃▁▄█▄▁ beats 5/29 | 5.3% |
| Aider polyglot (225 exercises, two attempts) | 81.3% ✎ | ▂·▂▃▅▂▃█▆▄▃▂ beats 46/49 | 5.0% |
| LiveCodeBench official (Aug 1, 2024–May 1, 2025) | 75.8% ✎ | ▃▃▅▃··▃▅█▅▅▃ beats 14/15 | 1.5% |
| LiveCodeBench (version unspecified) | 76.2% ✎ | ▄······▄█▄·▄ beats 4/5 | 1.4% |
| LiveCodeBench v6 (Feb–May 2025) | 58.6% ✎ | ▁▁▃▁█▅▄▇▅▅▃█ beats 13/38 | 0.7% |
| HumanEval (simple-evals) lab | 88.4% ✎ | ▆█▆▆▂··▂···▂ beats 5/15 | 0.4% |
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #71 of 393 · 8.3% of its score · 6 tests
Beats LongCat 2.0 (1.6T-A48B), GPT-5, Mistral Large 4 (public preview) here, though they rank higher overall. Trails Gemma 4 26B-A4B, Nemotron Cascade 2 30B-A3B, Gemma 4 31B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge) | 85.9% ✎ | ▂·▃▅▅·▃█▆▂▂▃ beats 22/22 | 2.8% |
| LMArena (text) | 1431.6 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 287/379 | 2.4% |
| Arena-Hard v2 aggregate | 80.8% ✎ | ▁▁··▄▃▁▃█▁▁▇ beats 20/22 | 1.3% |
| IFEval prompt-level strict | 90.2% ✎ | ███··█·█···█ beats 4/5 | 0.7% |
| Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges) | 88.8% ✎ | ▆▂▂▆▂▂▃█▂▃·▃ beats 22/22 | 0.6% |
| IFEval (aggregation unspecified) | 92.1% ✎ | ▁▁▃▁▂▄▄▆█▃▂▁ beats 123/141 | 0.5% |
math▁▂▃▅▆▅▄█▄▁▁▁ #82 of 398 · 21% of its score · 9 tests
Beats Claude Sonnet 4.6, Claude Opus 4.5, Qwen3.5 397B-A17B here, though they rank higher overall. Trails Granite 4.2 30B, INTELLECT-3, EXAONE 4.5 33B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| FrontierMath tiers 1–3 v2 | 33.3% ✎ | ▂··▁▁▃▄▄█▅▃▃ beats 22/82 | 11% |
| HMMT 2025 (edition unspecified) +1 | 71.9% ✎ | ▄··▄··▄·▄··█ beats 0/5 | 3.6% |
| AIME 2024 lab | 90.0% ✎ | ▁▁▁▂▁▃▁▅███▃ beats 73/85 | 1.6% |
| OTIS mock AIME 2024–2025 | 83.9% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 121/193 | 1.3% |
| MATH level 5 | 97.8% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 89/92 | 1.0% |
| MATH-500 (simple-evals) lab | 98.1% ✎ | ▃··▅▅▃···▃·█ beats 8/9 | 0.9% |
| PolyMATH | 49.7% ✎ | ▁···▃▁·▇█▄▄▄ beats 11/23 | 0.6% |
| MATH-500 | 98.4% ✎ | █···██·█·█·█ beats 2/5 | 0.6% |
1 more test
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MGSM (simple-evals) lab | 92.0% ✎ | ▁····▁▅▁█▁▃▁ beats 15/16 | 0.5% |
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #67 of 147 * · 1.6% of its score · 1 test
Beats DeepSeek V4 Flash Vision (Exp), GPT-5, Inkling Small (266B-A12B) here, though they rank higher overall. Trails Gemma 4 26B-A4B, Claude Haiku 4.5, Qwen3.6 Flash here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena vision (style controlled) | 1215.7 ✎ | ▁▁▂▂▂▂▄▅▅▆██ beats 81/145 | 1.6% |
web search▂▁▂▆▄█▃▃▁▁·▃ #1 of 80 * · 3.2% of its score · 2 tests
Beats GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 here, though they rank higher overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Finance Agent Benchmark v1 (537 questions, class-balanced accuracy) | 46.8% ✎ | █·▃▃·█▅▅▅▅▃▅ beats 18/18 | 2.3% |
| Arena search systems (style controlled) | 1191.9 ✎ | ▁·▁▁▂▂▂█▅▂▁▁ beats 14/30 | 1.0% |
other tests29 not in a task · 41% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| GPQA Diamond +2 | 81.8% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 131/208 | 10% |
| GPQA Diamond (simple-evals) lab | 83.4% ✎ | █▃▃█·▃▃▅▃▃▃▅ beats 16/16 | 4.2% |
| MMLU-Pro +1 | 85.3% ✎ | ▄·▄···█·▄··▄ beats 4/5 | 2.6% |
| MMLU-Redux +1 | 93.1% ✎ | ████··█····█ beats 5/5 | 2.3% |
| OJBench +1 | 38.4% ✎ | ▄······▄▄·▄█ beats 3/5 | 2.1% |
| KCL-Essay vanilla | 51.2% ✎ | ▁···▁▃▂█▂▃▃▂ beats 20/23 | 1.8% |
| EQ-Bench 3 (emotional intelligence, judge Elo) | 1288.3 ✎ | ▁·▂▁▄▅▅▃█▄▁▁ beats 43/62 | 1.5% |
| Mystery Game Puzzles (Epoch) | 29.0% ✎ | █··▂▃▇▄█▂▃▁▁ beats 47/74 | 1.3% |
21 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| VitaBench | 35.3% ✎ | █····██·██·█ beats 5/5 | 1.2% |
| Chess Puzzles (Epoch) | 34.0% ✎ | █▁▁▁▁▂▁▄▃▂▁▁ beats 111/135 | 1.2% |
| INCLUDE | 86.6% ✎ | ▂····▂▂▄▄▄█▄ beats 36/36 | 1.1% |
| ZebraLogic | 94.3% ✎ | ██··█·██···█ beats 2/5 | 1.1% |
| AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge) | 82.2% ✎ | ▁▁·▁▂▂▄█▄▅▄▄ beats 64/66 | 1.0% |
| KCL-MCQA vanilla | 50.2% ✎ | ▂···█▂▆▆██▆▂ beats 17/23 | 1.0% |
| MMLU (simple-evals, zero-shot CoT) lab | 93.3% ✎ | ▂▂·█▄▄▂·█▂·▃ beats 16/16 | 1.0% |
| SimpleQA Verified | 49.4% ✎ | ▁▁▃▂▃▃▆▆█▃▂▅ beats 53/78 | 0.9% |
| MMMLU lab | 88.8% ✎ | ▂▁▁▂▃▂▄▄▅██▃ beats 43/46 | 0.8% |
| HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge) | 85.6% ✎ | ▁▁▁▁·▂·█▆█▇▃ beats 31/31 | 0.7% |
| BeyondAIME | 63.2% ✎ | █··▄····▄··█ beats 1/5 | 0.7% |
| MMLU-ProX | 83.3% ✎ | ▁·▁▁▁▂▁▃▂█▅▄ beats 30/35 | 0.7% |
| SimpleQA lab | 54.0% ✎ | ▄▄▄▆█▆███·▂█ beats 30/32 | 0.6% |
| Multi-IF (aggregation unspecified) | 80.3% ✎ | ▁▁▁▁▁▁▃▄█▄▁▂ beats 32/41 | 0.6% |
| Creative Writing v3 | 87.7% ✎ | ▂····▂▆··▆█▆ beats 14/14 | 0.6% |
| WritingBench | 85.3% ✎ | ▁·▁▂▁··▅█▂▂▁ beats 14/20 | 0.5% |
| HealthBench Hard lab | 31.6% ✎ | ▁····▁▁▂█·▁▁ beats 7/13 | 0.3% |
| SimpleQA (simple-evals) lab | 48.6% ✎ | ▄·▁▁▁▁··█▃·▁ beats 13/14 | 0.3% |
| DROP F1, 3-shot (simple-evals) lab | 89.8% ✎ | ▂▂▄▆▆█·▂···▄ beats 15/16 | 0.2% |
| miniF2F (pass@1) | 15.2% ✎ | ██······▄··▄ beats 3/5 | 0.1% |
| Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate) | 93.2% ✎ | ▁▁▂▃▄▄█▆▃▂▂▁ beats 44/140 | 0.1% |
$ where to run it
Through an API
api · $3.50/M · $2 in / $8 out per 1M tokens · 200k context · platform.openai.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.
$ how the score was worked out · Epoch AI — internal evaluatio… 25%, Meituan LongCat-Flash-Thinkin… 19%, Qwen/Qwen3-235B-A22B-Thinking… 16% and 14 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json