whatmodel.sh / Gopher (280B)

Gopher (280B)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Gopher (280B) is a closed-weights language model from Google DeepMind, released December 2021. Its published size is 280B parameters. Licence: proprietary.

Catalogue identity notes

DeepMind's Gopher 280B (Rae et al., arXiv:2112.11446, Dec 2021). Model card: 'We will not make this model available publicly.' Scores here come from DeepMind's own Chinchilla paper, where Gopher is the same-lab comparison column. Comparison-only.

Model information sources

Release date ↗

70.1±1.9comparison only: no current run option verifiedfrom 11 benchmark results across 2 sources

Score within the LAMBADA (OpenAI version), last-word accuracy comparison set only.

$ how it compares

Scored LLMs released closest to Gopher (280B), out of 1182 with a score.

#modelscorewhereRAMreleased
1034LaMDA 137B (pre-trained, LaMDA-PT) Google34±1never released2022-01
1035GPT-3 text-davinci-001 OpenAI34±1comparison only2022-01
1106LaMDA 68B (pre-trained) Google27±1never released2022-01
1156LaMDA 8B (pre-trained) Google12±2never released2022-01
1158GPT-3 text-curie-001 OpenAI12±1comparison only2022-01

see all LLMs · find the ones that fit your machine

$ what it's good at

other tests9 not in a task · 98% of its score
testresultvs the others in its rundifficultycountswhy
MMLU lab +160.0% ✎▁▁▁▁▁▂▂▅██▆▄ beats 87/214very hard · avg 30%▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 53%sharp · at its level · several runs
LAMBADA (OpenAI version), last-word accuracy lab +174.5% ✎█··█·█··█·██ beats 5/5medium · avg 55%▇▇▇▇ 10%small run · several runs
WinoGrande lab70.1% ✎▁▁▁▁▂▁▁▅▇█▃▁ beats 92/131hard · avg 58%▇▇ 5.2%at its level · small run
BoolQ lab79.3% ✎▁····▁▂█▆▅▆▄ beats 73/103hard · avg 64%▇▇ 5.2%at its level · small run
TriviaQA lab52.8% ✎▁▁▂▂▄▄▃▆▅█▅▂ beats 47/80hard · avg 25%▇▇ 5.1%at its level · small run
Natural Questions (closed book) lab10.1% ✎▂▁▁▁▂▄▄▃▅█▅▁ beats 26/68very hard · avg 3.3%▇▇ 5.1%small run
HellaSwag lab79.2% ✎▁▁▁▂▂▂▄▆█▇▄▂ beats 106/135medium · avg 55%▇▇ 4.9%at its level · small run
Social IQA lab50.6% ✎▁·▁██▄▅▁▂▃▇▁ beats 36/59very hard · avg 39%▇▇ 4.8%small run
1 more test
testresultvs the others in its rundifficultycountswhy
PIQA lab81.8% ✎▁▁▁▂▂▃▅█▅▃▁▁ beats 106/123hard · avg 65%▇▇ 4.7%at its level · small run

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Training Compute-Optimal Larg… 59%, Scaling Language Models: Meth… 40%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Training Compute-Optimal Large Language Models — maker's own results · lab-reported9Factual vendor-reported score (not licensed content)
Scaling Language Models: Methods, Analysis & Insights from Training Gopher — maker's own results · lab-reported2Factual vendor-reported score (not licensed content)
prior · pull toward the middle
weighted average= 70.1

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json