whatmodel.sh / Flan-PaLM 540B
Flan-PaLM 540B
↗ share
+ contribute
Flan-PaLM 540B is a closed-weights language model from Google, released October 2022. Its published size is 540B parameters. Licence: proprietary.
Catalogue identity notes
Google's instruction-finetuned PaLM 540B (Chung et al., arXiv:2210.11416, Flan-PaLM model card Table 25). Never released (unlike the Flan-T5 checkpoints from the same paper). Comparison-only.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Flan-PaLM 540B, out of 877 with a score, then more from Google.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 415 |  Bielik 11B v2.3 Instruct | 51±2 | local | ||
| 416 |  WizardCoder 33B V1.1 | 51±1 | local | ||
| 417 | Flan-PaLM 540B | 51±1 | never released | ||
| 418 |  Llama-3-8B-ProLong-512k-Instruct | 51±4 | local | ||
| 419 |  Nova Pro | 51±1 | api $1.40/M | ||
| more from Google | |||||
| 11 |  Gemini 4 Argon | 80±1 | api $4.00/M | ||
| 26 | Gemini 3.8 Flash | 76±1 | api $1.50/M | ||
| 34 | Gemini 3.7 Flash | 75±1 | api $1.50/M | ||
$ what it's good at
math▁▂▃▆█▇▅▇▃▁▁▁ #194 of 523 * · 24% of its score · 1 test
Beats MiniMax-M2.1, Mistral Medium 3.5 (128B), GPT-4.1 here, though they rank higher overall. Trails OpenThinker3 7B here, though they rank lower overall.
other tests6 not in a task · 76% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BIG-Bench Hard lab | 57.9% ✎ | ▂▂▂▂·▅·▃▂█▃▃ beats 17/18 | 22% |
| MMLU lab | 73.5% ✎ | ▁···▁··▃▁▅▅█ beats 17/18 | 21% |
| MedQA USMLE 4-option (few-shot prompted) lab | 60.3% ✎ | ▄······▄▄▄·█ beats 5/5 | 12% |
| MedMCQA (few-shot prompted) lab | 56.5% ✎ | ▄·····▄··█·█ beats 5/5 | 11% |
| TyDiQA-GoldP (1-shot, EM, 8-language average) lab | 67.8% ✎ | ▂··▂·▂▄▄·███ beats 17/18 | 7.8% |
| PubMedQA (few-shot prompted) lab | 79.0% ✎ | ▄·······▄▄▄█ beats 5/5 | 2.8% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Scaling Instruction-Finetuned… 74%, Large Language Models Encode… 25%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Scaling Instruction-Finetuned Language Models — authors' own runs · lab-reported | 4 | 74% |
| Large Language Models Encode Clinical Knowledge — authors' own runs · lab-reported | 3 | 25% |
| prior · pull toward the middle | 1% | |
| weighted average | = 51.0 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json