o1-mini

+ contribute

o1-mini is a closed-weights language model from OpenAI, released September 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-09-12). Epoch label o1-mini-2024-09-12 (the only o1-mini snapshot). OpenAI deprecations page: o1-mini removed 2025-10-27. Reasoning effort is an API setting of the same checkpoint (recorded per result).

60.8±0.5comparison only: no current run option verifiedfrom 41 benchmark results across 18 sources

$ how it compares

The LLMs ranked either side of o1-mini, out of 828 with a score, then more from OpenAI.

#	model	score	where	RAM	released
221	HyperCLOVA X SEED 32B Think NAVER Cloud	61±2	comparison only		2025-12
222	Devstral Small 2 24B Mistral	61±3	local	~17GB	2025-12
223	o1-mini OpenAI	61±1	comparison only		2024-09
224	Gemini 2.0 Pro Experimental (02-05) Google	61±1	comparison only		2025-02
225	Qwen3 30B-A3B Thinking 2507 Alibaba	61±1	api $0.75/M +local	~20GB	2025-07
	more from OpenAI
1	GPT-6 Astra OpenAI	87±1	api $20.00/M		2026-09
3	GPT-6.1 Sol OpenAI	85±1	api $4.00/M		2026-09
7	GPT-6 Sol OpenAI	83±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▂▆█▆▅▄▄▃▂▁ #173 of 340 · 5.6% of its score · 4 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BrowserGym WebArena (AgentLab GenericAgent)	28.6% ✎	█··▄·▄█·▄··▄ beats 3/7	hard · avg 18%	▇ 1.8%	at its level
BrowserGym WorkArena-L1 (AgentLab GenericAgent)	56.7% ✎	█···█▄▄·▄█·█ beats 8/10	hard · avg 29%	▇ 1.5%	at its level
BrowserGym WorkArena-L2 (AgentLab GenericAgent)	14.9% ✎	▃▅▃·▅▃▃··█·▃ beats 7/11	very hard · avg 3.9%	▇ 1.4%	sharp · noisy
BrowserGym MiniWoB (AgentLab GenericAgent)	67.8% ✎	▅····▅▅▃▃·▃█ beats 7/11	medium · avg 55%	▇ 0.9%	blunt · at its level

coding▁▁▁▃▅▇█▇▅▄▃▁ #171 of 524 · 41% of its score · 12 tests

Beats Gemini 2.5 Pro, Gemini 3.1 Flash-Lite, Qwen3-Coder-Next (80B-A3B) here, though they rank higher overall. Trails Ministral 3 8B Reasoning 2512, Phi-4-reasoning 14B, Qwen3 235B-A22B (Apr 2025, hybrid) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts)	32.9% ✎	▂·▂▃▅▃▃█▆▄▂▂ beats 17/49	very hard · avg 8.5%	▇▇▇ 7.3%	sharp · at its level
LiveCodeBench official (Jan 1–Nov 1, 2024)	67.2% ✎	▁▁·▁▂▄▆█▅▂·▃ beats 30/31	hard · avg 26%	▇▇▇ 7.1%	sharp · at its level
HumanEval+	89.0% ✎	▁·▁▁▁▃▄▄▄█▃▃ beats 98/99	easy · avg 74%	▇▇ 5.9%	sharp
HumanEval	96.3% ✎	▁·▁▁▃▄▆▅█▃▁▃ beats 98/99	easy · avg 79%	▇▇ 5.9%	sharp · maxed out
CRUXEval (ZeroEval, 800 problems)	93.8% ✎	▁▂▆▃▆█▆▃▂·▃▁ beats 46/47	medium · avg 61%	▇ 3.7%	sharp
Aider code editing (133 Exercism Python exercises, two attempts)	70.7% ✎	▁·▁▂▄▁▆▄█▆▂▂ beats 49/62	medium · avg 54%	▇ 3.3%	typical
SciCode main problems (official leaderboard)	1.5% ✎	▃···█··▂·▃·▂ beats 7/17	very hard · avg 1.1%	▇ 2.3%	sharp · too hard for it
MBPP+	78.8% ✎	▁···▁▁▂▅▇█▆▃ beats 74/75	easy · avg 68%	▇ 2.0%	typical

4 more tests

test	result	vs the others in its run	difficulty	counts	why
WebApp1K-Duo-React pass@1 (onekq-ai)	66.7% ✎	▁··▁·▁▁▁▆█▇█ beats 24/45	hard · avg 32%	▇ 1.3%	at its level
WebApp1K-React pass@1 (onekq-ai)	93.9% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 29/31	medium · avg 44%	▇ 0.7%	sharp · noisy
LiveCodeBench v2	72.8% ✎	▄··█·▄···▄▄█ beats 7/7	medium · avg 50%	▇ 0.7%	at its level
HumanEval (simple-evals) lab	92.4% ✎	▆█▆▆▂··▂···▂ beats 10/15	easy · avg 84%	▇ 0.7%	typical

chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #196 of 463 * · 5.6% of its score · 2 tests

Beats MiniCPM5 2B, Granite 4.2 30B, Seed-OSS 36B Instruct here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, LFM2.5 8B-A1B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	92.0% ✎	▂▁▁▅█▅▄▃▃▅▃▃ beats 60/60	medium · avg 52%	▇ 3.1%	sharp
LMArena (text)	1336.8 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 175/379	avg 1268	▇ 2.5%	blunt

math▁▂▃▅▇▅▅█▄▁▁▁ #183 of 462 · 15% of its score · 9 tests

Beats MiniMax-M2.1, Mistral Medium 3.5 (128B), GPT-4.1 here, though they rank higher overall. Trails Olmo 3 7B Think, Nemotron Nano 9B v2, Qwen3-VL 4B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	89.2% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 77/92	hard · avg 32%	▇▇ 4.1%	sharp
MATH Level 5 (ZeroEval, 721 problems)	89.3% ✎	▁▁▁▆█▂▇▇▄··▃ beats 38/38	hard · avg 34%	▇ 3.3%	sharp · at its level
OTIS mock AIME 2024–2025	46.9% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 75/193	very hard · avg 8.5%	▇ 2.6%	sharp · at its level · noisy
MATH-500 (simple-evals) lab	90.0% ✎	▃··▅▅▃···▃·█ beats 5/9	medium · avg 61%	▇ 2.1%	sharp
HMMT February 2025	38.0% ✎	▁▁▁▃▁▅▅▃██▁▄ beats 3/31	hard · avg 33%	▇ 1.2%	at its level · small run
GSM8K (ZeroEval, zero-shot)	96.1% ✎	▁▄·▁▄█▅▅▅███ beats 41/43	very easy · avg 89%	▇ 0.9%	sharp · maxed out
MGSM (simple-evals) lab	89.9% ✎	▁····▁▅▁█▁▃▁ beats 10/16	easy · avg 82%	▇ 0.7%	blunt
AIME 2024	64.0% ✎	▁▁▁▂▁▃▁▅███▃ beats 30/85	medium · avg 42%	▇ 0.3%	sharp · at its level · noisy

1 more test

test	result	vs the others in its run	difficulty	counts	why
MATH-500	85.6% ✎	▁▃▂▃▅▇▅▇█▅▂▁ beats 28/84	easy · avg 84%	▇ 0.2%	maxed out · noisy

other tests13 not in a task · 33% of its score

test	result	vs the others in its run	difficulty	counts	why
ZebraLogic (ZeroEval, 1,000 puzzles)	52.6% ✎	▁▁▃▆██▁▃·▁▁▁ beats 53/57	hard · avg 18%	▇▇▇ 6.5%	sharp · at its level
GPQA Diamond +1	62.4% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 90/208	hard · avg 40%	▇▇▇ 6.3%	sharp · at its level · several runs
MMLU-Redux (ZeroEval, 2,778 questions)	86.7% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 48/53	easy · avg 77%	▇▇ 6.1%	typical
GPQA Diamond (simple-evals) lab	60.0% ✎	█▃▃█·▃▃▅▃▃▃▅ beats 8/16	hard · avg 40%	▇▇ 5.9%	sharp · at its level
LiveBench 2024-08-31	59.1% ✎	▄█·▄·▄▄·▄··▄ beats 6/7	medium · avg 42%	▇ 1.5%	at its level
MMLU (simple-evals, zero-shot CoT) lab	85.2% ✎	▂▂·█▄▄▂·█▂·▂ beats 3/16	easy · avg 80%	▇ 1.5%	typical
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)	70.5% ✎	▁▁·▁▂▂▄█▄▅▄▄ beats 55/66	hard · avg 31%	▇ 1.4%	at its level
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	60.0% ✎	▁▁▁▁·▂·█▆█▇▁ beats 14/31	medium · avg 48%	▇ 1.1%	at its level

5 more tests

test	result	vs the others in its run	difficulty	counts	why
MMMLU lab	78.6% ✎	▂▁▁▂▃▂▄▄▅██▃ beats 27/46	medium · avg 66%	▇ 1.0%	at its level · small run
EvalPerf correctness (118 tasks, 20240328)	88.8% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 29/29	easy · avg 76%	▇ 0.9%	typical
SimpleQA (simple-evals) lab	7.6% ✎	▄·▁▁▁▁··█▁·▁ beats 1/14	hard · avg 18%	▇ 0.2%	at its level · noisy
DROP F1, 3-shot (simple-evals) lab	83.9% ✎	▂▂▄▆▆█·▂···▄ beats 13/16	easy · avg 78%	▇ 0.2%	blunt
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	98.6% ✎	▁▁▂▃▄▄█▆▃▃▂▁ beats 130/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · WildEval ZeroEval published r… 20%, EvalPlus / EvalPerf published… 15%, Epoch AI — internal evaluatio… 11% and 15 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇ 20%	Apache-2.0
EvalPlus / EvalPerf published evaluation runs	4	▇▇▇▇ 15%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇ 11%	CC BY 4.0
OpenAI simple-evals README benchmark results (OpenAI's own models) · lab-reported	7	▇▇▇ 11%	MIT
Aider polyglot benchmark runs	1	▇▇ 7%	Apache-2.0
LiveCodeBench official website results (2024-12-26)	1	▇▇ 7%	CC-BY-SA-4.0
ServiceNow BrowserGym leaderboard (AgentLab GenericAgent runs)	4	▇▇ 6%	MIT
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇ 3%	Apache-2.0
open-thoughts/OpenThinker-7B model card	4	▇ 3%	Apache-2.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	2	▇ 2%	Apache-2.0
SciCode leaderboard (repository README)	1	▇ 2%	Apache-2.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 2%	MIT
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇ 2%	Apache-2.0 (stated in the LiveBench datasheet)
Microsoft — Phi-4-reasoning Technical Report	1	▇ 1%	CC BY 4.0
OpenAI — OpenAI o1 System Card (arXiv:2412.16720v2) · lab-reported	1	▇ 1%	CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 60.8