whatmodel.sh / PaLM 8B

PaLM 8B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

PaLM 8B is a closed-weights language model from Google, released April 2022. Its published size is 8.63B parameters. Licence: proprietary.

Catalogue identity notes

Google's original pre-trained PaLM 8B (Chowdhery et al., arXiv:2204.02311; Table 1: 8.63B parameters; 780B training tokens for 8B/540B, 795B for 62B). Not the 62B checkpoint trained to 1.3T tokens ('cont-PaLM'), not PaLM 2. Never released as weights or a public API. Comparison-only.

Model information sources

Release date ↗

25.8±0.7comparison only: no current run option verifiedfrom 51 benchmark results across 6 sources

Score within the LMArena (text) comparison set only.

$ how it compares

The LLMs ranked either side of PaLM 8B, out of 877 with a score, then more from Google.

#modelscorewhereRAMreleased
811Falcon3 1B Instruct TII26±4local~2.4GB2024-12
812OPT-13B (13 billion parameters) Meta26±1local~29GB2022-05
813PaLM 8B Google26±1never released2022-04
814Falcon 7B Instruct TII26±2local~5.9GB2023-04
815OpenAssistant SFT-4 Pythia 12B LAION (OpenAssistant)25±1local~9.5GB2023-04
more from Google
11Gemini 4 Argon Google80±1api $4.00/M2026-09
26Gemini 3.8 Flash Google76±1api $1.50/M2026-09
34Gemini 3.7 Flash Google75±1api $1.50/M2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

math▁▂▃▆█▇▅▇▃▁▁▁ #491 of 523 · 9.4% of its score · 3 tests

Beats Command R (03-2024), Mistral 7B Instruct v0.3, Gemma 2 2B IT here, though they rank higher overall. Trails GPT-Neo 125M, OPT-125M, GPT-2 XL (1.5B) here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
GSM8K lab +14.1% ✎█·▅▃▃▃·▃·▃▃▅ beats 6/12easy · avg 67%▇▇▇ 8.1%sharp · too hard for it · several runs
MGSM lab3.4% ✎▆·▂·▄▂▂▂▂█▄▆ beats 4/18medium · avg 47%▇ 0.9%sharp · too hard for it
Minerva MATH lab1.5% ✎▄··▄··▄·▄··█ beats 0/5hard · avg 32%▇ 0.4%too hard for it
other tests45 not in a task · 90% of its score
testresultvs the others in its rundifficultycountswhy
MAWPS (CoT-paper union) lab30.5% ✎▅▃█···▅·▃▃▃▅ beats 6/12very easy · avg 94%▇▇▇▇▇▇▇▇▇▇ 24%sharp · at its level
SVAMP lab16.8% ✎▃▅█···▅·▃▃▃▅ beats 6/12easy · avg 82%▇▇▇▇▇ 11%sharp
ASDiv lab25.2% ✎▅▃▃▅··▃▃·▃▃█ beats 7/12easy · avg 83%▇▇▇▇ 11%sharp
CommonsenseQA lab19.8% ✎█······▁·▁▁▃ beats 6/12easy · avg 87%▇▇ 4.2%sharp
MMLU lab +124.3% ✎▃···▁··▃▁▅▅█ beats 0/18easy · avg 74%▇▇ 4.1%several runs
WinoGrande lab +169.3% ✎▄····▄··▄·██ beats 3/6easy · avg 85%▇ 2.7%at its level · several runs
BIG-bench Date Understanding lab12.9% ✎▃·▅·▅▃▅·▃▃·█ beats 6/12medium · avg 52%▇ 2.7%typical
BIG-bench Sports Understanding lab55.1% ✎█·····▃·▁·▄▁ beats 6/12medium · avg 81%▇ 2.2%sharp
37 more tests
testresultvs the others in its rundifficultycountswhy
TriviaQA lab39.5% ✎▁▁▂▂▄▄▃▆▅█▅▂ beats 36/80easy · avg 79%▇ 1.7%at its level
HellaSwag lab68.7% ✎▁▁▁▂▂▂▄▆█▇▄▂ beats 65/135easy · avg 88%▇ 1.5%at its level
BoolQ lab68.3% ✎▁····▁▂█▆▅▆▄ beats 48/103easy · avg 89%▇ 1.5%at its level
MultiRC (SuperGLUE, prompted, F1a) lab47.5% ✎▃·▃██···▅··▃ beats 1/10very easy · avg 85%▇ 1.4%at its level · small run
COPA lab86.0% ✎▁·▁▁▁▅▅█▆▂▁▁ beats 49/67very easy · avg 96%▇ 1.2%at its level · small run
BIG-Bench Hard lab30.8% ✎▂▃▂▂·▅·▃▂█▃▃ beats 1/18medium · avg 60%▇ 1.1%sharp · too hard for it
TyDiQA-GoldP (1-shot, EM, 8-language average) lab25.0% ✎▂··▂·▂▄▄·███ beats 6/18medium · avg 61%▇ 1.1%typical
MMLU-STEM (Minerva subset) lab22.0% ✎▄······▄▄·▄█ beats 0/5medium · avg 62%▇ 1.1%sharp · too hard for it
AQuA-RAT lab21.7% ✎█▁·▃▁·····▂▁ beats 8/12medium · avg 48%▇ 1.0%sharp · too hard for it
StrategyQA lab55.6% ✎█·····▅▃▃▅▃█ beats 6/12medium · avg 70%▇ 1.0%typical
SQuAD 2.0 (prompted, F1) lab56.4% ✎▃▃···▃▃█▃▃▃▃ beats 5/10very easy · avg 85%▇ 1.0%at its level · small run
MedMCQA (few-shot prompted) lab26.7% ✎▄·····▄··█·█ beats 0/5medium · avg 55%▇ 1.0%too hard for it
MedQA USMLE 4-option (few-shot prompted) lab25.7% ✎▄······▄▄▄·█ beats 0/5medium · avg 58%▇ 1.0%too hard for it
CoQA lab66.0% ✎▂···▃▂▂·▆▅▂█ beats 8/17very easy · avg 88%▇ 0.9%at its level · small run
Natural Questions (closed book) lab8.4% ✎▂▁▁▁▂▄▄▃▅█▅▁ beats 21/68hard · avg 32%▇ 0.8%too hard for it
RTE (prompted) lab54.2% ✎▅▃·▃··▅·▅▅▃█ beats 5/13medium · avg 75%▇ 0.8%small run
Winograd Schema Challenge (WSC273) lab80.6% ✎█▃·▅▃▃█▅██▅▃ beats 9/21easy · avg 92%▇ 0.8%blunt · at its level · small run
ARC Challenge lab39.2% ✎▄▁▁▂▁▄█▇▃▂▃▂ beats 48/140medium · avg 72%▇ 0.7%sharp · noisy · small run
LAMBADA (OpenAI version), last-word accuracy lab69.5% ✎▁·····▁▁▃▃██ beats 52/86easy · avg 85%▇ 0.7%blunt · at its level
Story Cloze Test lab77.5% ✎▃▃▃▂▅█▅▂▁··▂ beats 30/50easy · avg 89%▇ 0.7%blunt · at its level · small run
ANLI Round 1 (prompted) lab34.9% ✎█▁▃·▃·▄·▃··▁ beats 10/16hard · avg 51%▇ 0.7%sharp · too hard for it · small run
ARC Easy lab69.2% ✎▁▃▂▄▆█▅▃▁··▁ beats 63/100easy · avg 85%▇ 0.7%blunt · at its level · small run
ANLI Round 2 (prompted) lab35.8% ✎█▅▃▅▃▃▅▃█··▃ beats 11/16hard · avg 45%▇ 0.6%too hard for it · small run
SuperGLUE WSC (prompted) lab78.9% ✎█··▁·▃·▄▂▁▅▃ beats 19/26easy · avg 83%▇ 0.6%blunt · at its level · small run
ReCoRD (SuperGLUE, prompted) lab87.8% ✎▄·▄···▄███▄▄ beats 4/10very easy · avg 95%▇ 0.5%blunt · small run
PIQA lab77.1% ✎▁▁▁▂▂▃▅█▅▃▁▁ beats 63/123easy · avg 86%▇ 0.5%blunt · at its level
PubMedQA (few-shot prompted) lab34.0% ✎▄·······▄▄▄█ beats 0/5medium · avg 75%▇ 0.5%noisy
OCWCourses (Minerva) lab1.5% ✎▄·····▄█··▄▄ beats 0/5hard · avg 15%▇ 0.5%too hard for it
RACE-middle lab57.9% ✎▄▄····▄█·███ beats 4/10easy · avg 74%▇ 0.4%blunt · at its level · small run
DROP lab45.1% ✎▃▅█▅·▃·▆▄▂▁▃ beats 26/44medium · avg 62%▇ 0.4%blunt · at its level · small run
QuAC (prompted, F1) lab33.1% ✎▅··▃▃█··▃▅·▃ beats 4/10medium · avg 53%▇ 0.4%blunt · at its level · small run
WiC (prompted) lab47.0% ✎█·▂▁······▁▁ beats 6/13hard · avg 60%▇ 0.3%sharp · too hard for it · noisy · small run
RACE-high lab42.3% ✎▂▄·▂▂▂▂▂▆▄▂█ beats 11/17medium · avg 53%▇ 0.3%blunt · small run
ANLI Round 3 (prompted) lab34.5% ✎▂·▆▃██·▄··▂▂ beats 4/16hard · avg 44%▇ 0.3%too hard for it · small run
OpenBookQA lab48.4% ✎▅▁▁▁▂▂▃█▃▁▃▁ beats 63/107medium · avg 59%▇ 0.3%noisy · small run
CommitmentBank (prompted) lab41.1% ✎▂▃·▂█▂▂▂·▂·▂ beats 5/13medium · avg 60%▇ 0.2%blunt · at its level · small run
WebQuestions (closed book) lab6.7% ✎▅▅▃▅▃·▃▃▅█▅▃ beats 10/17very hard · avg 4.3%▇ 0.0%blunt · too hard for it · small run

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Chain-of-Thought Prompting El… 63%, PaLM: Scaling Language Modeli… 23%, Scaling Instruction-Finetuned… 6% and 3 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — authors' own runs · lab-reported9CC BY 4.0
PaLM: Scaling Language Modeling with Pathways — authors' own runs · lab-reported30CC BY 4.0
Scaling Instruction-Finetuned Language Models — authors' own runs · lab-reported4CC BY 4.0
Solving Quantitative Reasoning Problems with Language Models — authors' own runs · lab-reported4CC BY 4.0
Large Language Models Encode Clinical Knowledge — authors' own runs · lab-reported3CC BY 4.0
WinoDict: Probing language models for in-context word acquisition — authors' own runs · lab-reported1CC BY 4.0
prior · pull toward the middle
weighted average= 25.8

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json