whatmodel.sh / Chinchilla (70B)

Chinchilla (70B)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Chinchilla (70B) is a closed-weights language model from Google DeepMind, released March 2022. Its published size is 70B parameters. Licence: proprietary.

Catalogue identity notes

DeepMind's Chinchilla 70B, trained on 1.4T tokens (Hoffmann et al., arXiv:2203.15556, Table 1). Never released. Comparison-only.

Model information sources

Release date ↗

73.5±2.5comparison only: no current run option verifiedfrom 9 benchmark results across 1 source

Score within the LAMBADA (OpenAI version), last-word accuracy comparison set only.

$ how it compares

Scored LLMs released closest to Chinchilla (70B), out of 1182 with a score.

#modelscorewhereRAMreleased
719Codex code-davinci-002 (GPT-3.5 base) OpenAI47±0comparison only2022-03
744PaLM 540B Google47±0never released2022-04
816GPT-3.5 text-davinci-002 OpenAI45±0comparison only2022-03
943PaLM 62B Google40±1never released2022-04
1099GPT-NeoX-20B EleutherAI28±1local~14GB2022-02

see all LLMs · find the ones that fit your machine

$ what it's good at

other tests9 not in a task · 97% of its score
testresultvs the others in its rundifficultycountswhy
MMLU lab67.6% ✎▁▁▁▁▁▂▂▅██▆▄ beats 121/214very hard · avg 31%▇▇▇▇▇▇▇▇▇▇▇▇ 31%sharp · at its level · small run
WinoGrande lab74.9% ✎▁▁▁▁▂▁▁▅▇█▃▁ beats 112/131hard · avg 58%▇▇▇ 8.7%at its level · small run
BoolQ lab83.7% ✎▁····▁▂█▆▅▆▄ beats 86/103hard · avg 64%▇▇▇ 8.7%at its level · small run
TriviaQA lab67.0% ✎▁▁▂▂▄▄▃▆▅█▅▂ beats 63/80hard · avg 25%▇▇▇ 8.5%at its level · small run
Natural Questions (closed book) lab16.6% ✎▂▁▁▁▂▄▄▃▅█▅▁ beats 36/68very hard · avg 3.3%▇▇▇ 8.5%small run
HellaSwag lab80.8% ✎▁▁▁▂▂▂▄▆█▇▄▂ beats 114/135medium · avg 55%▇▇▇ 8.2%small run
Social IQA lab51.3% ✎▁·▁██▄▅▁▂▃▇▁ beats 37/59very hard · avg 39%▇▇▇ 8.1%at its level · small run
LAMBADA (OpenAI version), last-word accuracy lab77.4% ✎▁·····▁▁▃▃██ beats 83/86medium · avg 55%▇▇▇ 7.9%small run
1 more test
testresultvs the others in its rundifficultycountswhy
PIQA lab81.8% ✎▁▁▁▂▂▃▅█▅▃▁▁ beats 107/123hard · avg 65%▇▇▇ 7.9%at its level · small run

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Training Compute-Optimal Larg… 97%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Training Compute-Optimal Large Language Models — maker's own results · lab-reported9Factual vendor-reported score (not licensed content)
prior · pull toward the middle
weighted average= 73.5

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json