whatmodel.sh / o3-mini
o3-mini
↗ share
+ contribute
o3-mini is a closed-weights language model from OpenAI, released January 2025. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2025-01-31). Epoch label o3-mini-2025-01-31 (the only o3-mini snapshot). Reasoning effort recorded per result. OpenAI's deprecations page schedules its shutdown for October 23, 2026 (announced 2026-04-22).
$ how it compares
The LLMs ranked either side of o3-mini, out of 828 with a score, then more from OpenAI.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 156 |  Laguna XS 2.1 (33B-A3B) | 66±2 | local | ||
| 157 |  LongCat-Flash-Chat (560B-A27B) | 66±1 | local | ||
| 158 | o3-mini | 65±1 | api $1.93/M | ||
| 159 |  Claude 3.7 Sonnet | 65±1 | comparison only | ||
| 160 | Claude Sonnet 4 | 65±1 | comparison only | ||
| more from OpenAI | |||||
| 1 | GPT-6 Astra | 87±1 | api $20.00/M | ||
| 3 | GPT-6.1 Sol | 85±1 | api $4.00/M | ||
| 7 | GPT-6 Sol | 83±1 | api $4.00/M | ||
$ what it's good at
agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #238 of 300 · 8.3% of its score · 7 tests
Trails Granite 4.0 1B, SmolLM3 3B, Phi-3.5-mini-instruct here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Finance Agent Benchmark v1 (537 questions, class-balanced accuracy) | 12.8% ✎ | █·▃▃·█▅▅▅▅▃▅ beats 7/18 | 2.3% |
| τ²-bench Retail | 75.7% ✎ | █▄··▄··▄·▄·▄ beats 6/6 | 1.7% |
| SWE-bench Verified | 22.6% ✎ | █·█▄·······█ beats 0/6 | 1.5% |
| τ²-bench Airline | 61.3% ✎ | ▄▄······▄█▄▄ beats 6/6 | 1.3% |
| Multi-SWE-bench, 7 languages (MagentLess scaffold) | 3.6% ✎ | ▄···▄▄·▄▄·▄█ beats 1/7 | 0.7% |
| BFCL v3 | 51.3% ✎ | ▁▁▁▁▁▃▅█▅▄▁▁ beats 38/114 | 0.5% |
| Galileo Agent Leaderboard v1 (tool selection quality) | 84.7% ✎ | ▃▃▃▃▆▅▆▅█▂▅▂ beats 14/31 | 0.3% |
coding▁▁▁▃▆██▇▆▄▃▁ #123 of 458 · 27% of its score · 10 tests
Beats Qwen3.5 Plus, Gemini 2.5 Pro, Kimi K2 Thinking here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, Seed-OSS 36B Instruct, Gemma 4 26B-A4B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Aider polyglot (225 exercises, two attempts) +1 | 60.4% ✎ | ▂·▂▃▅▂▃█▆▄▂▂ beats 35/49 | 15% |
| SciCode main problems (official leaderboard) | 9.2% ✎ | ▃···█··▂·▃·▃ beats 17/17 | 3.9% |
| LiveCodeBench official (Aug 1, 2024–May 1, 2025) | 67.4% ✎ | ▃▃▅▃··▃▅█▅▅▃ beats 11/15 | 2.0% |
| SWE-bench Verified | 22.6% ✎ | █·█▄·······█ beats 0/6 | 1.5% |
| WebApp1K-Duo-React pass@1 (onekq-ai) | 69.6% ✎ | ▁··▁·▁▁▁▆█▇█ beats 30/45 | 1.3% |
| HumanEval+ | 94.0% ✎ | ▁·▁▁▂▃▄▄█▅▄▃ beats 143/145 | 1.3% |
| HumanEval (Vector Institute, Inspect Evals) | 98.2% ✎ | ▄·▄·█·█·█·▄▄ beats 9/9 | 1.0% |
| Multi-SWE-bench, 7 languages (MagentLess scaffold) | 3.6% ✎ | ▄···▄▄·▄▄·▄█ beats 1/7 | 0.7% |
2 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| HumanEval (simple-evals) lab | 97.6% ✎ | ▆█▆▆▂··▃···▂ beats 14/15 | 0.5% |
| WebApp1K-React pass@1 (onekq-ai) | 96.1% ✎ | ▃▁▁█▁▄▅▃▄▇▁▄ beats 31/31 | 0.5% |
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #138 of 393 · 9.0% of its score · 5 tests
Trails LFM2.5 8B-A1B, LFM2.5 2.6B, Spark-X2.5 1.7B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge) | 66.1% ✎ | ▂·▃▅▅·▃█▆▃▂▂ beats 20/22 | 4.0% |
| LMArena (text) | 1363.6 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 206/379 | 2.7% |
| Arena-Hard (version/judge unspecified) | 81.9% ✎ | ▃▃▂▅▃▆▅█▅▅▃▃ beats 28/30 | 1.2% |
| Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges) | 43.0% ✎ | ▆▂▂▆▂▃▃█▂▃·▂ beats 11/22 | 0.7% |
| IFEval (aggregation unspecified) | 91.5% ✎ | ▁▁▃▁▂▄▄▆█▃▂▁ beats 120/141 | 0.3% |
math▁▂▃▅▆▅▄█▄▁▁▁ #120 of 398 · 21% of its score · 9 tests
Beats Claude Sonnet 4.6, MiniMax-M3, MiMo-V2.5 (310B-A15B) here, though they rank higher overall. Trails Olmo 3.1 32B Think, Gemini 2.5 Flash, Qwen3-VL 32B Thinking here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| FrontierMath tiers 1–3 v2 | 18.6% ✎ | ▂··▁▁▃▄▄█▅▃▃ beats 11/82 | 8.8% |
| OTIS mock AIME 2024–2025 | 76.9% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 110/193 | 2.3% |
| MATH level 5 | 96.5% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 85/92 | 2.2% |
| MATH (Vector Institute, Inspect Evals) | 96.9% ✎ | ▄··██▄··▄·▄█ beats 8/9 | 2.1% |
| FrontierMath tier 4 v2 | 0.0% ✎ | ▃▂▁▂█▅▂▂▁▁▁▁ beats 0/63 | 1.7% |
| MATH-500 (simple-evals) lab | 97.9% ✎ | ▃··▅▅▃···▃·█ beats 7/9 | 1.4% |
| AIME 2025 | 82.5% ✎ | █··▄··█··▄·▄ beats 6/6 | 1.1% |
| AIME 2024 | 88.0% ✎ | ▄···▄█·▄▄··▄ beats 6/6 | 0.7% |
1 more test
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MGSM (simple-evals) lab | 92.0% ✎ | ▁····▁▅▁█▁▃▁ beats 14/16 | 0.6% |
web search▂▁▂▆▄█▃▃▁▁·▁ #8 of 80 * · 2.3% of its score · 1 test
Beats GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 here, though they rank higher overall. Trails GPT-4o (2024-08-06), GPT-4.1 mini, GPT-4.1 here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Finance Agent Benchmark v1 (537 questions, class-balanced accuracy) | 12.8% ✎ | █·▃▃·█▅▅▅▅▃▅ beats 7/18 | 2.3% |
other tests19 not in a task · 36% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| GPQA Diamond +1 | 77.0% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 121/208 | 8.4% |
| ZebraLogic (ZeroEval, 1,000 puzzles) | 91.7% ✎ | ▁▁▃▆██▁▁·▁▁▃ beats 57/57 | 6.2% |
| GPQA Diamond (simple-evals) lab | 77.2% ✎ | █▃▃█·▃▃▅▃▃▃▅ beats 14/16 | 5.8% |
| GPQA Diamond (Vector Institute, Inspect Evals) | 73.7% ✎ | ▃··▃▃▃▃·▃·▃█ beats 7/9 | 4.2% |
| MMLU-Pro (Vector Institute, Inspect Evals) | 79.2% ✎ | ▃····▃▅·▃▅·█ beats 6/9 | 1.7% |
| AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge) | 59.8% ✎ | ▁▁·▁▂▂▄█▄▅▄▄ beats 50/66 | 1.3% |
| MMLU (simple-evals, zero-shot CoT) lab | 86.9% ✎ | ▂▂·█▄▄▂·█▂·▂ beats 7/16 | 1.3% |
| MMLU-Pro | 79.4% ✎ | ▁▁·▁▁▁▃▂▄▅█▁ beats 124/188 | 1.2% |
11 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge) | 55.1% ✎ | ▁▁▁▁·▂·█▆█▇▁ beats 10/31 | 1.0% |
| Chess Puzzles (Epoch) | 17.0% ✎ | █▁▁▁▁▂▃▄▂▂▁▁ beats 71/135 | 0.9% |
| SimpleQA Verified | 15.3% ✎ | ▁▁▃▃▃▃▆▆█▃▂▅ beats 8/78 | 0.9% |
| EuroEval HellaSwag (English), validation split, 5-shot generative MC, accuracy | 75.8% ✎ | ▂··▂▂·▂▆▆█·▄ beats 8/15 | 0.8% |
| Mystery Game Puzzles (Epoch) | 7.0% ✎ | █··▂▃▇▄█▂▃▁▁ beats 2/74 | 0.8% |
| Multi-IF (aggregation unspecified) | 87.3% ✎ | ▄·····▄█▄▄·▄ beats 6/6 | 0.7% |
| FlenQA (3K-token subset) | 96.8% ✎ | ▄···▄····▄·█ beats 2/4 | 0.6% |
| SimpleQA (simple-evals) lab | 13.8% ✎ | ▄·▃▁▁▁··█▁·▁ beats 3/14 | 0.3% |
| DROP F1, 3-shot (simple-evals) lab | 80.6% ✎ | ▂▂▄▆▆█·▂···▄ beats 5/16 | 0.2% |
| Omni-MATH | 74.6% ✎ | █·█·█·██··██ beats 3/6 | 0.1% |
| Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate) | 99.2% ✎ | ▁▁▂▃▄▄█▆▃▂▂▃ beats 138/140 | 0.1% |
$ where to run it
Through an API
api · $1.93/M · $1.1 in / $4.4 out per 1M tokens · 200k context · developers.openai.com ↗ · price sourceUSD per 1M tokens, OpenAI API standard processing, model page 'o3-mini' whose default snapshot is o3-mini-2025-01-31: input $1.1, output $4.4; cached-input, Batch and other tiers excluded. Currency: USD. OpenAI official model page, observed 2026-09-28.
$ how the score was worked out · Epoch AI — internal evaluatio… 22%, ServiceNow-AI/Apriel-1.6-15b-… 12%, Microsoft — Phi-4-reasoning T… 10% and 16 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json