whatmodel.sh / Flan-PaLM 8B
Flan-PaLM 8B
↗ share
+ contribute
Flan-PaLM 8B is a closed-weights language model from Google, released October 2022. Its published size is 8.63B parameters. Licence: proprietary.
Catalogue identity notes
Google's instruction-finetuned PaLM (Flan-PaLM 8B; Chung et al., arXiv:2210.11416, Table 2). Never released. Comparison-only.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Flan-PaLM 8B, out of 877 with a score, then more from Google.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 706 |  EXAONE 3.5 2.4B Instruct | 38±2 | local | ||
| 707 |  xDAN-L1-Chat-RL-v1 | 38±1 | local | ||
| 708 | Flan-PaLM 8B | 38±2 | never released | ||
| 709 |  LLaMA 33B | 37±1 | comparison only | ||
| 710 | Llama 3 8B (base) | 37±1 | local | ||
| more from Google | |||||
| 11 |  Gemini 4 Argon | 80±1 | api $4.00/M | ||
| 26 | Gemini 3.8 Flash | 76±1 | api $1.50/M | ||
| 34 | Gemini 3.7 Flash | 75±1 | api $1.50/M | ||
$ what it's good at
math▁▂▃▆█▇▅▇▃▁▁▁ #447 of 523 * · 17% of its score · 1 test
Beats GPT-3.5 Turbo (0125), DBRX Instruct, Command R Plus (04-2024) here, though they rank higher overall. Trails OPT-1.3B, OPT-2.7B, Gemma 2B (base) here, though they rank lower overall.
other tests6 not in a task · 82% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MMLU lab | 49.3% ✎ | ▁···▁··▃▁▅▅█ beats 5/18 | 30% |
| BIG-Bench Hard lab | 36.4% ✎ | ▂▂▂▂·▅·▃▂█▃▃ beats 4/18 | 17% |
| MedQA USMLE 4-option (few-shot prompted) lab | 35.4% ✎ | ▄······▄▄▄·█ beats 1/5 | 11% |
| TyDiQA-GoldP (1-shot, EM, 8-language average) lab | 47.5% ✎ | ▂··▂·▂▄▄·███ beats 11/18 | 10% |
| MedMCQA (few-shot prompted) lab | 34.5% ✎ | ▄·····▄··█·█ beats 1/5 | 10% |
| PubMedQA (few-shot prompted) lab | 67.6% ✎ | ▄·······▄▄▄█ beats 3/5 | 4.0% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Scaling Instruction-Finetuned… 74%, Large Language Models Encode… 25%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Scaling Instruction-Finetuned Language Models — authors' own runs · lab-reported | 4 | 74% |
| Large Language Models Encode Clinical Knowledge — authors' own runs · lab-reported | 3 | 25% |
| prior · pull toward the middle | 1% | |
| weighted average | = 37.5 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json