whatmodel.sh / Codex code-cushman-001
Codex code-cushman-001
↗ share
+ contribute
Codex code-cushman-001 is a closed-weights language model from OpenAI, released August 2021. Licence: proprietary.
Catalogue identity notes
Comparison-only. Smaller first-generation OpenAI Codex API model 'code-cushman-001' (OpenAI deprecations page, Codex models, shut down 2023-03-23). Release month from the 2021-08-10 Codex API private-beta announcement (archived). Parameter count never officially published.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Codex code-cushman-001, out of 1182 with a score, then more from OpenAI.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 1025 |  Vicuna 7B v1.3 | 35±2 | local | ||
| 1026 |  Stable Code 3B | 35±1 | local | ||
| 1027 | Codex code-cushman-001 | 35±5 | comparison only | ||
| 1028 |  T0 11B | 35±3 | comparison only | ||
| 1029 |  Qwen1.5 4B Chat | 34±2 | local | ||
| more from OpenAI | |||||
| 1 | GPT-6 Astra | 86±1 | api $20.00/M | ||
| 3 | GPT-6.1 Sol | 84±1 | api $4.00/M | ||
| 8 | GPT-6 Sol | 81±1 | api $4.00/M | ||
$ what it's good at
other tests2 not in a task · 89% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| ARCADE New Tasks pass@1 (notebook-context prompting) | 8.9% ✎ | █·█········█ beats 1/2 | 47% |
| ODEX English pass@1 (zero-shot) | 31.9% ✎ | ▄▄········▄█ beats 0/4 | 42% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Natural Language to Code Gene… 47%, Execution-Based Evaluation fo… 42%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Natural Language to Code Generation in Interactive Data Science Notebooks — authors' own runs | 1 | 47% |
| Execution-Based Evaluation for Open-Domain Code Generation — authors' own runs | 1 | 42% |
| prior · pull toward the middle | 11% | |
| weighted average | = 34.7 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json