whatmodel.sh / GPT-3 text-babbage-001
GPT-3 text-babbage-001
↗ share
+ contribute
GPT-3 text-babbage-001 is a closed-weights language model from OpenAI, released January 2022. Licence: proprietary.
Catalogue identity notes
OpenAI API InstructGPT-series completion engine text-babbage-001. Identity from Wei et al. (arXiv:2201.11903v6): 'for which we use text-ada-001, text-babbage-001, text-curie-001, and text-davinci-002, which presumably correspond to InstructGPT models of 350M, 1.3B, 6.7B, and 175B parameters' (the 1.3B size is the authors' presumption, so params_b is omitted). OpenAI made InstructGPT models the API default on 2022-01-27 ('Aligning Language Models to Follow Instructions', archived). Retired with the other InstructGPT engines on 2024-01-04 (OpenAI deprecations page). Comparison-only.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of GPT-3 text-babbage-001, out of 877 with a score, then more from OpenAI.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 863 |  BLOOM 1.1B | 7±1 | local | ||
| 864 | GPT-2 XL (1.5B) | 7±1 | local | ||
| 865 | GPT-3 text-babbage-001 | 4±2 | comparison only | ||
| 866 | GPT-3 text-ada-001 | 1±3 | comparison only | ||
| 867 |  OPT-350M | 1±1 | local | ||
| more from OpenAI | |||||
| 1 | GPT-6 Astra | 86±1 | api $20.00/M | ||
| 3 | GPT-6.1 Sol | 84±1 | api $4.00/M | ||
| 8 | GPT-6 Sol | 81±1 | api $4.00/M | ||
$ what it's good at
math▃▂▃▆█▇▅▇▃▁▁▁ #520 of 523 * · 4.4% of its score · 1 test
Trails GPT-2 Small (124M), GPT-Neo 125M, OPT-125M here, though they rank lower overall.
other tests11 not in a task · 93% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MAWPS (CoT-paper union) | 1.7% ✎ | ▅▃█···▅·▃▃▃▅ beats 2/12 | 29% |
| ASDiv | 1.4% ✎ | ▅▃▃▅··▃▃·▃▃█ beats 2/12 | 20% |
| SVAMP | 1.7% ✎ | ▃▅█···▅·▃▃▃▅ beats 2/12 | 14% |
| BIG-bench Date Understanding | 4.0% ✎ | ▃·▅·▅▃▅·▃▃·█ beats 1/12 | 11% |
| New Yorker Caption Contest matching, from description, 5-shot (accuracy) | 19.0% ✎ | █·····▃··▅·▃ beats 1/6 | 4.1% |
| StrategyQA | 45.8% ✎ | █·····▅▃▃▅▃█ beats 1/12 | 4.0% |
| CommonsenseQA | 12.0% ✎ | █······▁·▁▁▃ beats 4/12 | 3.0% |
| LMentry accuracy (zero-shot) | 34.2% ✎ | ▂▆·▄▄█▂·▂··▂ beats 4/14 | 3.0% |
3 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BIG-bench Sports Understanding | 0.0% ✎ | █·····▃·▁·▄▁ beats 4/12 | 2.8% |
| ConvRe Re2Text, normal relation, zero-shot (prompt 1#) | 52.4% ✎ | ▂▂▄··▂▂█▄··▂ beats 2/12 | 1.9% |
| AQuA-RAT | 4.3% ✎ | █▁·▁▁·····▂▁ beats 5/12 | 0.3% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Chain-of-Thought Prompting El… 89%, Do Androids Laugh at Electric… 4%, LMentry: A Language Model Ben… 3% and 1 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — authors' own runs | 9 | 89% |
 Do Androids Laugh at Electric Sheep? Humor ``Understanding'' Benchmarks from The New Yorker Caption Contest — authors' own runs | 1 | 4% |
| LMentry: A Language Model Benchmark of Elementary Language Tasks — authors' own runs | 1 | 3% |
| An Investigation of LLMs' Inefficacy in Understanding Converse Relations — authors' own runs | 1 | 2% |
| prior · pull toward the middle | 3% | |
| weighted average | = 4.0 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json