whatmodel.sh / o1 (2024-12-17)

o1 (2024-12-17)

+ contribute

o1 (2024-12-17) is a closed-weights language model from OpenAI, released December 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-12-17). Dated API snapshot o1-2024-12-17; distinct from o1-preview. Reasoning effort recorded per result. OpenAI's deprecations page schedules its shutdown for October 23, 2026 (announced 2026-04-22).

66.0±0.6#130 of 719 LLMsfrom 39 benchmark results across 19 sources

$ how it compares

The LLMs ranked either side of o1 (2024-12-17), out of 828 with a score, then more from OpenAI.

#	model	score	where	RAM	released
147	GLM-4.6 Z.ai	66±1	api $1.00/M +local	~58GB	2025-09
148	Kimi K2 Instruct (0711) Moonshot AI	66±1	api $1.00/M +local	~160GB	2025-07
149	o1 (2024-12-17) OpenAI	66±1	api $26.25/M		2024-12
150	MiniMax-M2.5 MiniMax	66±1	api $0.53/M +local	~75GB	2026-02
151	Qwen3-Max (2025-09-23) Alibaba	66±1	api $2.40/M		2025-09
	more from OpenAI
1	GPT-6 Astra OpenAI	87±1	api $20.00/M		2026-09
3	GPT-6.1 Sol OpenAI	85±1	api $4.00/M		2026-09
7	GPT-6 Sol OpenAI	83±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #224 of 300 · 4.5% of its score · 4 tests

Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	21.4% ✎	█·▃▃·█▅▅▅▅▃▅ beats 12/18	very hard · avg 5.8%	▇ 2.7%	sharp · at its level
Multi-SWE-bench, 7 languages (MagentLess scaffold)	6.4% ✎	▄···▄▄·▄▄·▄█ beats 7/7	very hard · avg 2.6%	▇ 0.8%	blunt · too hard for it
BFCL v3	56.1% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 62/114	medium · avg 52%	▇ 0.6%	blunt · at its level
Galileo Agent Leaderboard v1 (tool selection quality)	87.6% ✎	▃▃▃▃▆▅▆▅█▂▅▂ beats 18/31	easy · avg 79%	▇ 0.3%	blunt

coding▁▁▁▃▆██▇▆▄▃▁ #116 of 458 · 21% of its score · 5 tests

Beats Qwen3.5 Plus, Gemini 2.5 Pro, Kimi K2 Thinking here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, Solar Pro 4, DeepSeek R1 0528 here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts)	61.7% ✎	▂·▂▃▅▂▃█▆▄▂▂ beats 37/49	very hard · avg 8.5%	▇▇▇ 8.5%	sharp · at its level
LiveCodeBench official (Jan 1–Nov 1, 2024)	72.4% ✎	▁▁·▁▂▄▆█▅▂·▃ beats 31/31	hard · avg 26%	▇▇▇ 7.9%	sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)	84.2% ✎	▁·▁▂▄▁▆▄█▆▂▃ beats 61/62	medium · avg 54%	▇ 2.9%	typical
HumanEval (Vector Institute, Inspect Evals)	97.0% ✎	▄·▄·█·█·█·▄▄ beats 8/9	easy · avg 79%	▇ 1.1%	maxed out
Multi-SWE-bench, 7 languages (MagentLess scaffold)	6.4% ✎	▄···▄▄·▄▄·▄█ beats 7/7	very hard · avg 2.6%	▇ 0.8%	blunt · too hard for it

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #106 of 393 · 8.6% of its score · 3 tests

Beats LongCat 2.0 (1.6T-A48B), GPT-5.4 nano, Step 3.5 Flash (196B-A11B) here, though they rank higher overall. Trails Gemini 2.5 Flash, Mistral Large 3 (675B-A41B), GPT-4.1 here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge)	61.0% ✎	▂·▃▅▅·▃█▆▂▂▂ beats 19/22	very hard · avg 12%	▇▇ 4.6%	sharp · at its level
LMArena (text)	1402.3 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 248/379	avg 1268	▇ 3.2%	blunt
Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges)	59.9% ✎	▆▂▂▆▂▂▃█▂▃·▂ beats 15/22	hard · avg 19%	▇ 0.8%	sharp · at its level · noisy

math▁▂▃▅▆▅▄█▄▁▁▁ #135 of 398 · 22% of its score · 8 tests

Beats Claude Sonnet 4.6, MiniMax-M3, MiMo-V2.5 (310B-A15B) here, though they rank higher overall. Trails Olmo 3 32B Think, Olmo 3.1 32B Think, Nemotron-Cascade-8B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
FrontierMath tiers 1–3 v2	14.7% ✎	▂··▁▁▃▄▄█▅▃▃ beats 8/82	very hard · avg 1.2%	▇▇▇▇ 11%	sharp
OTIS mock AIME 2024–2025	73.3% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 108/193	very hard · avg 8.5%	▇ 2.5%	sharp · at its level · noisy
MATH level 5	94.7% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 81/92	hard · avg 32%	▇ 2.4%	sharp · maxed out
MATH (Vector Institute, Inspect Evals)	95.9% ✎	▄··██▄··▄·▄█ beats 7/9	medium · avg 53%	▇ 2.3%	sharp
MATH-500 (simple-evals) lab	96.4% ✎	▃··▅▅▃···▃·█ beats 6/9	medium · avg 61%	▇ 1.6%	sharp · maxed out
AIME 2025	71.4% ✎	█··▄··█··▄·▄ beats 4/6	hard · avg 34%	▇ 1.2%	sharp · noisy
AIME 2024	74.6% ✎	▄···▄█·▄▄··▄ beats 2/6	medium · avg 43%	▇ 0.7%	sharp · noisy
MGSM (simple-evals) lab	89.3% ✎	▁····▁▅▁█▁▃▁ beats 7/16	easy · avg 82%	▇ 0.7%	blunt

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #82 of 147 · 6.3% of its score · 4 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-5 mini here, though they rank higher overall. Trails GPT-4.1 mini, Gemini 2.5 Flash, Mistral Large 3 (675B-A41B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena vision (style controlled)	1192.8 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 67/145	avg 1068	▇ 2.2%	blunt
CharXiv (reasoning questions) lab	55.1% ✎	▄··▂█··▂···▄ beats 3/9	hard · avg 34%	▇ 2.0%	at its level
CharXiv descriptive questions lab	88.9% ✎	▄▄·····▄·▄█▄ beats 5/6	easy · avg 72%	▇ 1.6%	typical
PhysBench physical-world understanding (test, Avg)	55.1% ✎	▅▃█··▅▃····▃ beats 9/9	hard · avg 43%	▇ 0.5%	blunt · at its level

web search▂▁▂▆▄█▃▃▁▃·▁ #4 of 80 * · 2.7% of its score · 1 test

Beats GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 here, though they rank higher overall. Trails GPT-4.1 here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	21.4% ✎	█·▃▃·█▅▅▅▅▃▅ beats 12/18	very hard · avg 5.8%	▇ 2.7%	sharp · at its level

other tests15 not in a task · 38% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +1	76.8% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 119/208	hard · avg 40%	▇▇▇▇ 9.5%	sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzles)	81.0% ✎	▁▁▃▆██▁▁·▁▃▁ beats 56/57	hard · avg 18%	▇▇▇ 7.1%	sharp · at its level
GPQA Diamond (simple-evals) lab	75.7% ✎	█▃▃█·▃▃▅▃▃▃▅ beats 13/16	hard · avg 40%	▇▇▇ 6.6%	sharp · at its level
GPQA Diamond (Vector Institute, Inspect Evals)	75.5% ✎	▃··▃▃▃▃·▃·▃█ beats 8/9	hard · avg 41%	▇▇ 4.8%	sharp · at its level
MMLU-Pro (Vector Institute, Inspect Evals)	84.5% ✎	▃····▃▅·▃▅·█ beats 9/9	medium · avg 60%	▇ 2.0%	typical
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)	84.3% ✎	▁▁·▁▂▂▄█▄▅▄▄ beats 66/66	hard · avg 31%	▇ 1.5%	at its level
MMLU (simple-evals, zero-shot CoT) lab	91.8% ✎	▂▂·█▄▄▂·█▃·▂ beats 15/16	easy · avg 80%	▇ 1.5%	typical
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	79.6% ✎	▁▁▁▁·▂·█▆█▇▁ beats 30/31	medium · avg 48%	▇ 1.2%	typical

7 more tests

test	result	vs the others in its run	difficulty	counts	why
Chess Puzzles (Epoch)	15.0% ✎	█▁▁▁▁▂▃▄▂▂▁▁ beats 69/135	very hard · avg 5.9%	▇ 1.1%	sharp · noisy
SimpleQA Verified	41.1% ✎	▁▁▃▂▃▃▆▆█▃▂▅ beats 37/78	very hard · avg 8.0%	▇ 1.0%	at its level
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)	65.4% ✎	▁··▃▁▁▁▅█▅▁▅ beats 18/24	medium · avg 52%	▇ 0.6%	blunt · at its level
SimpleQA (simple-evals) lab	42.6% ✎	▄·▁▁▁▁··█▁·▁ beats 12/14	hard · avg 18%	▇ 0.3%	at its level · noisy
DROP F1, 3-shot (simple-evals) lab	90.2% ✎	▂▂▄▆▆█·▂···▄ beats 16/16	easy · avg 78%	▇ 0.2%	blunt
Omni-MATH	67.5% ✎	█·█·█·██··██ beats 2/6	medium · avg 64%	▇ 0.2%	blunt · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	97.6% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 116/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy

$ where to run it

Through an API

api · $26.25/M · $15 in / $60 out per 1M tokens · 200k context · developers.openai.com ↗ · price sourceUSD per 1M tokens, OpenAI API standard processing, model page 'o1' whose default snapshot is o1-2024-12-17: input $15, output $60; cached-input, Batch and other tiers excluded. Currency: USD. OpenAI official model page, observed 2026-09-28.

$ how the score was worked out · Epoch AI — internal evaluatio… 23%, OpenAI simple-evals README be… 11%, Vector Institute Evaluation L… 10% and 16 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Epoch AI — internal evaluations	6	▇▇▇▇▇▇▇ 23%	CC BY 4.0
OpenAI simple-evals README benchmark results (OpenAI's own models) · lab-reported	6	▇▇▇ 11%	MIT
Vector Institute Evaluation Leaderboard (Inspect Evals runs)	4	▇▇▇ 10%	Apache-2.0
Aider polyglot benchmark runs	1	▇▇▇ 9%	Apache-2.0
LiveCodeBench official website results (2024-12-26)	1	▇▇ 8%	CC-BY-SA-4.0
WildEval ZeroEval published results	1	▇▇ 7%	Apache-2.0
Microsoft — Phi-4-reasoning Technical Report	4	▇▇ 7%	CC BY 4.0
Arena-Hard-Auto v2.0-Preview leaderboard (repository README)	2	▇▇ 5%	Apache-2.0
LMArena leaderboard dataset	2	▇▇ 5%	CC BY 4.0
OpenAI vendor-reported benchmark scores for its own models (openai.com announcement posts and Deployment Safety Hub system cards) · lab-reported	2	▇ 4%	Factual vendor-reported score (not licensed content)
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 3%	Apache-2.0
Finance Agent Benchmark (Vals AI, arXiv:2508.00828v1)	1	▇ 3%	CC-BY-4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	2	▇ 3%	Apache-2.0
Multi-SWE-bench experiments (ByteDance Seed baseline runs)	1	▇ 1%	Apache-2.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)	1	▇ 1%	CC BY 4.0
Berkeley Function Calling Leaderboard result archive	1	▇ 1%	Apache-2.0
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (arXiv:2501.16411v2)	1	▇ 1%	CC-BY-4.0
Galileo Agent Leaderboard (v1 and v2 results)	1	▇ 0%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 66.0