whatmodel.sh / Codex code-davinci-001
Codex code-davinci-001
↗ share
+ contribute
Codex code-davinci-001 is a closed-weights language model from OpenAI, released August 2021. Licence: proprietary.
Catalogue identity notes
Comparison-only. First-generation OpenAI Codex API model 'code-davinci-001' (the OpenAI deprecations page lists it among the 'Codex models' shut down 2023-03-23). Release month from OpenAI's 2021-08-10 'OpenAI Codex' post (archived): 'we are releasing it through our API in private beta starting today'. Not code-davinci-002. Parameter count never published.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Codex code-davinci-001, out of 877 with a score, then more from OpenAI.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 758 |  OPT-175B | 33±1 | comparison only | ||
| 759 |  BLOOMZ 176B | 33±2 | comparison only | ||
| 760 | Codex code-davinci-001 | 33±1 | comparison only | ||
| 761 |  StarCoderBase 7B | 33±1 | local | ||
| 762 | Llama 3.2 1B Instruct | 33±1 | api $0.020/M +local | ||
| more from OpenAI | |||||
| 1 | GPT-6 Astra | 86±1 | api $20.00/M | ||
| 3 | GPT-6.1 Sol | 84±1 | api $4.00/M | ||
| 8 | GPT-6 Sol | 81±1 | api $4.00/M | ||
$ what it's good at
math▁▂▃▆█▇▅▇▃▁▁▁ #443 of 523 * · 28% of its score · 1 test
Beats GPT-3.5 Turbo (0125), DBRX Instruct, Command R Plus (04-2024) here, though they rank higher overall. Trails OPT-1.3B, OPT-2.7B, Gemma 2B (base) here, though they rank lower overall.
other tests5 not in a task · 71% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| SVAMP +1 | 41.2% ✎ | █▆▆▅▂▅▂▅▂▂▅▃ beats 23/30 | 33% |
| AQuA-RAT +1 | 22.4% ✎ | █··▃▁▁··▁▁▁▁ beats 18/27 | 16% |
| CommonsenseQA | 46.8% ✎ | █·····▂▂▃▂▄▄ beats 21/38 | 13% |
| CoNLL-2003 English NER, few-shot in-context prompting (entity F1) | 61.9% ✎ | ▃···▅·▃▅··▃█ beats 5/9 | 5.2% |
| ODEX English pass@1 (zero-shot) | 33.6% ✎ | ▄▄········▄█ beats 1/4 | 3.2% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Faithful Chain-of-Thought Rea… 48%, Large Language Models are Bet… 43%, CodeIE: Large Code Generation… 5% and 1 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Faithful Chain-of-Thought Reasoning — authors' own runs | 3 | 48% |
| Large Language Models are Better Reasoners with Self-Verification — authors' own runs | 4 | 43% |
| CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors — authors' own runs | 1 | 5% |
| Execution-Based Evaluation for Open-Domain Code Generation — authors' own runs | 1 | 3% |
| prior · pull toward the middle | 1% | |
| weighted average | = 32.8 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json