whatmodel.sh / Codex code-davinci-002 (GPT-3.5 base)

Codex code-davinci-002 (GPT-3.5 base)

+ contribute

Codex code-davinci-002 (GPT-3.5 base) is a closed-weights language model from OpenAI, released March 2022. Licence: proprietary.

Catalogue identity notes

OpenAI API completions engine code-davinci-002. OpenAI's 'New Capabilities for GPT-3: Edit & Insert' post (March 15, 2022; archived copy) names 'the latest versions of GPT-3 and Codex, text-davinci-002 and code-davinci-002'. OpenAI deprecations page: Codex models shut down 2023-03-23 (base-model table also lists 2024-01-04). Parameter count never officially published. Comparison-only.

Model information sources

Release date ↗

47.3±0.4comparison only: no current run option verifiedfrom 64 benchmark results across 26 sources

Score within the LMArena (text) comparison set only.

$ how it compares

The LLMs ranked either side of Codex code-davinci-002 (GPT-3.5 base), out of 877 with a score, then more from OpenAI.

#	model	score	where	RAM	released
515	Granite 34B Code Instruct (8K) IBM	47±1	local	~22GB	2024-04
516	WizardCoder Python 34B V1.0 WizardLM	47±1	local	~22GB	2023-08
517	Codex code-davinci-002 (GPT-3.5 base) OpenAI	47±0	comparison only		2022-03
518	Granite 4.0 Micro 3B IBM	47±1	local	~3.5GB	2025-09
519	Cogito v1 Preview Llama 3B Deep Cogito	47±4	local	~3.4GB	2025-03
	more from OpenAI
1	GPT-6 Astra OpenAI	86±1	api $20.00/M		2026-09
3	GPT-6.1 Sol OpenAI	84±1	api $4.00/M		2026-09
8	GPT-6 Sol OpenAI	81±1	api $4.00/M		2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▅▇█▇▅▄▂▁ #442 of 529 * · 1.0% of its score · 1 test

Beats LFM2.5 8B-A1B, DeepSeek R1 Distill Qwen 7B, Qwen3.5 2B here, though they rank higher overall. Trails Qwen2.5 1.5B Instruct, WizardLM 13B V1.2, Falcon3 3B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval	45.9% ✎	▁▁▁▁▂▂▆▆█▅▂▁ beats 83/205	easy · avg 68%	▇ 1.0%	sharp · at its level · small run

math▁▂▃▆█▇▅▇▃▁▁▁ #253 of 523 · 28% of its score · 2 tests

Beats MiniMax-M2.1, DeepSeek V3 (Dec '24), GPT-4o (2024-08-06) here, though they rank higher overall. Trails Gemma 7B (base), Phi-2 (2.7B), Granite 8B Code Base (4K) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
GSM8K +12	63.1% ✎	█·▅▃▃▃·▃·▃▃▅ beats 12/12	easy · avg 67%	▇▇▇▇▇▇▇▇▇▇ 25%	sharp · at its level · several runs
MGSM	35.0% ✎	▆·▂·▄▂▂▂▂█▄▆ beats 12/18	medium · avg 47%	▇ 2.7%	sharp · at its level

other tests26 not in a task · 71% of its score

test	result	vs the others in its run	difficulty	counts	why
AQuA-RAT +5	45.3% ✎	█▁·▁▁·····▂▃ beats 12/12	medium · avg 48%	▇▇▇▇▇▇▇ 17%	sharp · several runs
SVAMP +7	76.4% ✎	▃▅█···▅·▃▃▃▅ beats 11/12	easy · avg 82%	▇▇▇▇▇▇ 14%	sharp · several runs
CommonsenseQA +5	82.3% ✎	█······▁·▁▁▃ beats 12/12	easy · avg 87%	▇▇▇ 8.3%	sharp · several runs
ASDiv +1	80.4% ✎	▅▃▃▅··▃▃·▃▃█ beats 12/12	easy · avg 83%	▇▇ 5.5%	sharp · several runs
MAWPS (CoT-paper union)	92.6% ✎	▅▃█···▅·▃▃▃▅ beats 11/12	very easy · avg 94%	▇▇ 4.2%	sharp
BIG-Bench Hard +1	52.8% ✎	▂▂▂▂·▅·▃▂█▃▃ beats 16/18	medium · avg 60%	▇▇ 3.8%	sharp · at its level · several runs
MMLU	68.2% ✎	▁···▁··▃▁▅▅█ beats 14/18	easy · avg 74%	▇ 2.7%	at its level
BIG-bench Sports Understanding	71.7% ✎	█·····▃·▁·▄▁ beats 10/12	medium · avg 81%	▇ 2.5%	sharp · at its level

18 more tests

test	result	vs the others in its run	difficulty	counts	why
BIG-bench Date Understanding +1	49.0% ✎	▃·▅·▅▃▅·▃▃·█ beats 12/12	medium · avg 52%	▇ 2.2%	at its level · several runs
StrategyQA +5	67.1% ✎	█·····▅▃▃▅▃█ beats 11/12	medium · avg 70%	▇ 2.0%	at its level · several runs
ARC Challenge	83.6% ✎	▄▁▁▂▁▄█▇▃▂▃▂ beats 128/140	easy · avg 86%	▇ 1.5%	sharp
MultiArith +1	96.2% ✎	█·········██ beats 2/2	very easy · avg 95%	▇ 1.0%	sharp · maxed out · small run · several runs
TyDiQA-GoldP (1-shot, EM, 8-language average)	48.1% ✎	▂··▂·▂▄▄·███ beats 12/18	medium · avg 61%	▇ 1.0%	at its level
CoNLL-2003 English NER, few-shot in-context prompting (entity F1) +1	82.3% ✎	▃···▅·▃▅··▃█ beats 8/9	easy · avg 83%	▇ 0.8%	small run · several runs
Natural Questions (closed book)	31.4% ✎	▂▁▁▁▂▄▄▃▅█▅▁ beats 63/68	medium · avg 38%	▇ 0.7%	at its level · small run
AddSub	89.4% ✎	█·········██ beats 1/2	very easy · avg 89%	▇ 0.5%	small run
ARCADE New Tasks pass@1 (notebook-context prompting)	23.4% ✎	█·█········█ beats 2/2	hard · avg 23%	▇ 0.5%	small run
TriviaQA	81.8% ✎	▁▁▂▂▄▄▃▆▅█▅▃ beats 77/80	easy · avg 80%	▇ 0.4%	small run
OpenBookQA	71.8% ✎	▅▁▁▁▂▂▃█▃▃▃▁ beats 93/107	medium · avg 61%	▇ 0.4%	at its level · noisy · small run
WinoGrande	60.7% ✎	▁▁▁▁▂▁▁▅▇█▃▁ beats 36/131	medium · avg 73%	▇ 0.4%	at its level
ACE05-E event argument extraction (gold triggers), few-shot prompting, Arg-C F1	58.5% ✎	█·····█····█ beats 1/2	medium · avg 65%	▇ 0.4%	at its level · small run
CSK-PN boolean commonsense QA (10-shot)	84.9% ✎	▃·▃·····▃▃▃█ beats 7/7	easy · avg 87%	▇ 0.3%	blunt · at its level
ODEX English pass@1 (zero-shot)	47.1% ✎	▄▄········▄█ beats 4/4	medium · avg 58%	▇ 0.3%	blunt · at its level
HellaSwag	65.2% ✎	▁▁▁▂▂▂▄▆█▇▄▂ beats 53/135	easy · avg 77%	▇ 0.3%	at its level
Zero-shot relation extraction, vanilla prompt (TACRED/RETACRED/TACREV/SemEval avg F1)	39.8% ✎	█····▄····█▄ beats 0/5	medium · avg 50%	▇ 0.2%	blunt · at its level
ARC Easy	94.0% ✎	▁▃▂▄▆█▅▃▁··▃ beats 99/100	easy · avg 87%	▇ 0.1%	blunt

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Chain-of-Thought Prompting El… 37%, Self-Consistency Improves Cha… 11%, Scaling Instruction-Finetuned… 9% and 23 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — authors' own runs	9	▇▇▇▇▇▇▇▇▇▇▇ 37%	CC BY 4.0
Self-Consistency Improves Chain of Thought Reasoning in Language Models — authors' own runs	10	▇▇▇ 11%	CC BY 4.0
Scaling Instruction-Finetuned Language Models — authors' own runs	4	▇▇▇ 9%	CC BY 4.0
Faithful Chain-of-Thought Reasoning — authors' own runs	3	▇▇ 8%	CC BY 4.0
Large Language Models are Better Reasoners with Self-Verification — authors' own runs	4	▇ 5%	CC BY 4.0
Batch Prompting: Efficient Inference with Large Language Model APIs — authors' own runs	3	▇ 4%	CC BY 4.0
Coverage-based Example Selection for In-Context Learning — authors' own runs	1	▇ 3%	CC BY 4.0
Making Language Models Better Reasoners with Step-Aware Verifier — authors' own runs	2	▇ 3%	CC BY 4.0
Tab-CoT: Zero-shot Tabular Chain of Thought — authors' own runs	3	▇ 3%	CC BY 4.0
Automatic Model Selection with Large Language Models for Reasoning — authors' own runs	2	▇ 3%	CC BY 4.0
Complexity-Based Prompting for Multi-Step Reasoning — authors' own runs	4	▇ 3%	CC BY 4.0
PAL: Program-aided Language Models — authors' own runs	1	▇ 2%	CC0 1.0
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting — authors' own runs	2	▇ 2%	CC BY 4.0
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them — authors' own runs	1	▇ 1%	CC BY 4.0
Symbolic Chain-of-Thought Distillation: Small Models Can Also ``Think'' Step-by-Step — authors' own runs	2	▇ 1%	CC BY 4.0
Recitation-Augmented Language Models — authors' own runs	2	▇ 1%	CC BY 4.0
StarCoder: may the source be with you! — authors' own runs	1	▇ 1%	CC BY-SA 4.0
Improving Mathematics Tutoring With A Code Scratchpad — authors' own runs	1	▇ 1%	CC BY 4.0
Exploring the Curious Case of Code Prompts — authors' own runs	2	▇ 1%	CC BY 4.0
Natural Language to Code Generation in Interactive Data Science Notebooks — authors' own runs	1	▇ 0%	CC BY 4.0
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples! — authors' own runs	1	▇ 0%	CC BY 4.0
Code4Struct: Code Generation for Few-Shot Event Structure Prediction — authors' own runs	1	▇ 0%	CC BY 4.0
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors — authors' own runs	1	▇ 0%	CC BY 4.0
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge — authors' own runs	1	▇ 0%	CC BY 4.0
Execution-Based Evaluation for Open-Domain Code Generation — authors' own runs	1	▇ 0%	CC BY 4.0
Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors — authors' own runs	1	▇ 0%	CC BY 4.0
prior · pull toward the middle		▇ 0%
weighted average		= 47.3

not counted (4)

Batch Prompting: Efficient Inference with Large Language Model APIs — authors' own runs	StrategyQA [correct]	73.3%	the only model in this source's run of the test, so there is nothing to compare it with there; another run of the test counts
Batch Prompting: Efficient Inference with Large Language Model APIs — authors' own runs	AQuA-RAT [correct]	46.1%	the only model in this source's run of the test, so there is nothing to compare it with there; another run of the test counts
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models — authors' own runs	GSM8K [correct]	60.3%	the only model in this source's run of the test, so there is nothing to compare it with there; another run of the test counts
Faithful Chain-of-Thought Reasoning — authors' own runs	StrategyQA [correct]	72.5%	the only model in this source's run of the test, so there is nothing to compare it with there; another run of the test counts

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-14 08:02 UTC · sources and licences · json