whatmodel.sh / OLMo 7B

OLMo 7B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

OLMo 7B is an open-weights language model from Allen Institute for AI, released February 2024. Its published size is 6.89B parameters and a 2k-token context window. Licence: apache-2.0.

Catalogue identity notes

OLMo 7B (Feb 2024), main (annealed, ~2.46T-token) checkpoint of allenai/OLMo-7B; the -hf repo is AI2's Transformers-format conversion of the same weights. Not OLMo-7B-0424/0724 or OLMo 2.

28.4±1.2#655 of 719 LLMsfrom 13 benchmark results across 1 source

$ how it compares

The LLMs ranked either side of OLMo 7B, out of 828 with a score, then more from Allen Institute for AI.

#modelscorewhereRAMreleased
752OPT-30B Meta29±1local~64GB2022-05
753NexusRaven V2 13B Nexusflow29±3local~9.3GB2023-12
754OLMo 7B Allen Institute for AI28±1local~30GB2024-02
755Falcon 40B Instruct TII28±2local~27GB2023-05
756GPT-NeoX-20B EleutherAI28±1local~14GB2022-02
more from Allen Institute for AI
493OLMo 2 13B Instruct (1124) Allen Institute for AI46±1local~9.6GB2024-12
516Tulu 2 70B Allen Institute for AI46±2local~28GB2023-11
557Code Tulu 2 34B Allen Institute for AI44±2local~72GB2023-11

see all LLMs · find the ones that fit your machine

$ what it's good at

coding█··········█ #2 of 2 * · 7.3% of its score · 1 test
testresultvs the others in its rundifficultycountswhy
Tulu 2: CodexEval pass@10 lab21.4% ✎▃▃▄▁▄▃▃█▃▁▁▄ beats 0/27easy · avg 83%▇▇▇ 7.3%sharp · small run
math█··········█ #2 of 2 * · 7.2% of its score · 1 test
testresultvs the others in its rundifficultycountswhy
GSM8K lab8.5% ✎▃▂▂▅▅▆▅▇█▇▄▃ beats 2/146easy · avg 68%▇▇▇ 7.2%sharp · small run
other tests11 not in a task · 85% of its score
testresultvs the others in its rundifficultycountswhy
ARC Challenge lab48.5% ✎▃▃······▃▅▃█ beats 7/8easy · avg 88%▇▇▇▇▇▇▇ 18%sharp · at its level
WinoGrande lab67.9% ✎▂····▂·▂·▂▃█ beats 4/8easy · avg 89%▇▇▇▇▇ 12%at its level
OpenBookQA lab50.4% ✎▄·····▄▄▄██▄ beats 6/8easy · avg 84%▇▇▇▇ 11%at its level
HellaSwag lab76.4% ✎▂·····▂▂·▂·█ beats 6/8very easy · avg 92%▇▇▇ 7.7%at its level
BoolQ lab73.4% ✎█····▃··█▃·▃ beats 4/8easy · avg 92%▇▇▇ 7.1%at its level
BIG-Bench Hard lab31.7% ✎▁▁▁▁▃▄▅▅█▃▄▁ beats 9/129medium · avg 73%▇▇▇ 7.1%sharp · small run
ARC Easy lab65.4% ✎▃···▃·▃··▃▅█ beats 3/8easy · avg 88%▇▇▇ 6.9%at its level
SciQ lab93.8% ✎▂·▂···▂·▂·█▂ beats 5/8very easy · avg 98%▇▇ 6.2%typical
3 more tests
testresultvs the others in its rundifficultycountswhy
PIQA lab78.4% ✎▄····▄·▄▄▄██ beats 7/8easy · avg 88%▇▇ 3.8%blunt · at its level
MMLU lab28.3% ✎▁▁▃▁▁▂▃▅█▃▆▃ beats 4/183medium · avg 73%▇ 3.2%noisy · small run
Tulu 2: TyDiQA gold passage, 1-shot F1 lab32.3% ✎▃··▃▂▃▂▂█▅▁▂ beats 4/27medium · avg 57%▇ 1.4%blunt · at its level · small run

$ where to run it

Other ways to run it

  1. Run it as published · needs ~27.55 GB · from huggingface.co/allenai
    uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='allenai/OLMo-7B-hf', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"

Weights and documentation: huggingface.co/allenai ↗

$ how the score was worked out · OLMo: Accelerating the Scienc… 99%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
OLMo: Accelerating the Science of Language Models — authors' own runs · lab-reported13▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99%CC BY 4.0
prior · pull toward the middle▇ 1%
weighted average= 28.4

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json