whatmodel.sh / o1-preview

o1-preview

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

o1-preview is a closed-weights language model from OpenAI, released September 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-09-12). Epoch label o1-preview-2024-09-12 (the only o1-preview snapshot). OpenAI deprecations page: o1-preview removed 2025-07-28.

62.8±0.6comparison only: no current run option verifiedfrom 27 benchmark results across 13 sources

$ how it compares

The LLMs ranked either side of o1-preview, out of 828 with a score, then more from OpenAI.

#modelscorewhereRAMreleased
191Qwen3-Next 80B-A3B Instruct Alibaba63±1api $0.34/M +local~27GB2025-09
192Granite 4.2 30B IBM63±1api $0.28/M +local~19GB2026-08
193o1-preview OpenAI63±1comparison only2024-09
194Gemini 2.5 Flash Preview (04-17) Google63±1comparison only2025-04
195GLM-4.5-Air Z.ai63±1api $0.42/M +local~35GB2025-07
more from OpenAI
1GPT-6 Astra OpenAI87±1api $20.00/M2026-09
3GPT-6.1 Sol OpenAI85±1api $4.00/M2026-09
7GPT-6 Sol OpenAI83±1api $4.00/M2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasksnot ranked on this task yet · 1 test
testresultvs the others in its rundifficultycountswhy
SWE-bench Verified lab41.3% ✎▁▁▁▁▁▁▂▄█▆▁▁ beats 28/131hard · avg 19%—not counted yet: no other model in the same run to compare it with
coding▁▁▁▃▅▇█▇▅▄▃▁ #151 of 524 · 41% of its score · 11 tests

Beats Qwen3.5 Plus, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall. Trails Devstral Small 2 24B, Llama 3.1 Nemotron Ultra 253B v1, Nemotron 3.5 Lightning 30B-A3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench official (Jan 1–Nov 1, 2024)52.5% ✎▁▁·▁▂▄▆█▅▃·▂ beats 29/31hard · avg 26%▇▇▇▇ 10%sharp · at its level
HumanEval+89.0% ✎▁·▁▁▁▃▄▄▄█▃▃ beats 99/99easy · avg 74%▇▇▇ 6.8%sharp · maxed out
HumanEval96.3% ✎▁·▁▁▃▄▆▅█▃▁▃ beats 99/99easy · avg 79%▇▇▇ 6.6%sharp · maxed out
CRUXEval (ZeroEval, 800 problems)95.9% ✎▁▂▆▃▆█▆▃▂·▁▃ beats 47/47medium · avg 61%▇▇ 4.2%sharp
Aider code editing (133 Exercism Python exercises, two attempts)79.7% ✎▁·▁▂▄▁▆▄█▆▃▂ beats 59/62medium · avg 54%▇▇ 4.1%typical
SciCode main problems (official leaderboard)7.7% ✎▃···█··▂·▃·▃ beats 16/17very hard · avg 1.1%▇▇ 4.0%sharp · too hard for it
MBPP+80.2% ✎▁···▁▁▂▅▇█▆▃ beats 75/75easy · avg 68%▇ 2.5%typical
WebApp1K-Duo-React pass@1 (onekq-ai)65.2% ✎▁··▁·▁▁▁▆█▇█ beats 21/45hard · avg 32%▇ 1.8%at its level
3 more tests
testresultvs the others in its rundifficultycountswhy
WebApp1K-React pass@1 (onekq-ai)95.2% ✎▃▁▁█▁▄▅▃▄▇▁▄ beats 30/31medium · avg 44%▇ 0.8%sharp · noisy
HumanEval (simple-evals) lab92.4% ✎▆█▆▆▂··▂···▂ beats 11/15easy · avg 84%▇ 0.8%maxed out
SWE-bench Verified lab41.3% ✎▁▁▁▁▁▁▂▄█▆▁▁ beats 28/131hard · avg 19%—not counted yet: no other model in the same run to compare it with
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #140 of 463 * · 6.8% of its score · 2 tests

Beats LongCat 2.0 (1.6T-A48B), MiMo-V2-Flash (309B-A15B), MiniMax-M2.1 here, though they rank higher overall. Trails LFM2.5 2.6B, Nemotron-Cascade-8B, DeepSeek V3 0324 here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LMArena (text)1388.5 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 231/379avg 1268▇ 3.5%blunt
Arena-Hard (v0.1, GPT-4-1106 judge)90.4% ✎▂▁▁▅█▅▄▃▃▅▃▃ beats 59/60medium · avg 52%▇ 3.3%sharp
math▁▂▃▅▇▅▅█▄▁▁▁ #200 of 462 · 15% of its score · 5 tests

Beats MiniMax-M2.1 here, though they rank higher overall. Trails Phi-4-mini-reasoning 3.8B, OpenThinker3 7B, Olmo 3 7B Think here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MATH level 581.6% ✎▂▂▅▆▄█▆▃▅▂▃▄ beats 67/92hard · avg 32%▇▇ 4.4%sharp
MATH Level 5 (ZeroEval, 721 problems)84.5% ✎▁▁▁▆█▂▇▇▄··▃ beats 37/38hard · avg 34%▇▇ 4.0%sharp
OTIS mock AIME 2024–202531.1% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 66/193very hard · avg 8.5%▇ 3.5%sharp · at its level · noisy
MATH-500 (simple-evals) lab85.5% ✎▃··▅▅▃···▃·█ beats 3/9medium · avg 61%▇ 2.4%sharp
MGSM (simple-evals) lab90.8% ✎▁····▁▅▁█▃▃▁ beats 13/16easy · avg 82%▇ 0.9%blunt
other tests10 not in a task · 37% of its score
testresultvs the others in its rundifficultycountswhy
ZebraLogic (ZeroEval, 1,000 puzzles)71.4% ✎▁▁▃▆██▁▁·▃▁▁ beats 54/57hard · avg 18%▇▇▇▇ 8.9%sharp · at its level
GPQA Diamond (simple-evals) lab73.3% ✎█▃▃█·▃▃▅▃▃▃▅ beats 12/16hard · avg 40%▇▇▇ 8.1%sharp · at its level
MMLU-Redux (ZeroEval, 2,778 questions)92.8% ✎▁·▁▁▄▄▄▅█▄▂▃ beats 53/53easy · avg 77%▇▇▇ 7.3%typical
GPQA Diamond50.3% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 70/208hard · avg 40%▇▇▇ 6.3%sharp · at its level
LiveBench 2024-08-3166.0% ✎▄█·▄·▄▄·▄··▄ beats 7/7medium · avg 42%▇ 2.1%at its level
MMLU (simple-evals, zero-shot CoT) lab90.8% ✎▂▂·█▄▄▂·█▂·▂ beats 14/16easy · avg 80%▇ 1.9%typical
MMMLU lab86.7% ✎▂▁▁▂▃▂▄▄▅██▃ beats 39/46medium · avg 66%▇ 1.3%small run
SimpleQA (simple-evals) lab42.4% ✎▄·▁▁▁▁··█▁·▁ beats 11/14hard · avg 18%▇ 0.3%at its level · noisy
2 more tests
testresultvs the others in its rundifficultycountswhy
DROP F1, 3-shot (simple-evals) lab74.8% ✎▃▂▄▆▆█·▂···▄ beats 0/16easy · avg 78%▇ 0.3%blunt
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)96.7% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 98/140very easy · avg 94%▇ 0.1%blunt · maxed out · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · WildEval ZeroEval published r… 24%, EvalPlus / EvalPerf published… 16%, OpenAI simple-evals README be… 15% and 10 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
WildEval ZeroEval published results4▇▇▇▇▇▇▇ 24%Apache-2.0
EvalPlus / EvalPerf published evaluation runs3▇▇▇▇▇ 16%Apache-2.0
OpenAI simple-evals README benchmark results (OpenAI's own models) · lab-reported7▇▇▇▇ 15%MIT
Epoch AI — internal evaluations3▇▇▇▇ 14%CC BY 4.0
LiveCodeBench official website results (2024-12-26)1▇▇▇ 10%CC-BY-SA-4.0
Aider code editing benchmark runs (old edit leaderboard)1▇ 4%Apache-2.0
SciCode leaderboard (repository README)1▇ 4%Apache-2.0
LMArena leaderboard dataset1▇ 4%CC BY 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)1▇ 3%Apache-2.0
WebApp1K models leaderboard (onekq-ai)2▇ 3%MIT
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets1▇ 2%Apache-2.0 (stated in the LiveBench datasheet)
OpenAI — OpenAI o1 System Card (arXiv:2412.16720v2) · lab-reported1▇ 1%CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 62.8

not counted (1)

OpenAI — OpenAI o1 System Card (arXiv:2412.16720v2)SWE-bench Verified [correct]41.3%fewer than three models have this test yet, or this source ran it for this model alone

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json