whatmodel.sh / OLMo 7B Instruct

OLMo 7B Instruct

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

OLMo 7B Instruct is an open-weights language model from Ai2, released February 2024. Its published size is 6.89B parameters. Licence: apache-2.0.

Catalogue identity notes

Comparison-only. Arena entry 'olmo-7b-instruct' (text). LMArena's leaderboard table (arena-leaderboard space @6f2eeeb) labels 'olmo-7b-instruct' as 'OLMo-7B-instruct' (Allen AI, Apache-2.0), linking https://huggingface.co/allenai/OLMo-7B-Instruct. Maker repo allenai/OLMo-7B-Instruct @5e3ff29: licence tag apache-2.0. 6,888,095,744 parameters from the Hugging Face safetensors metadata (dense, so active = total). Released 2024-02: month the maker's Hugging Face repo was created (createdAt 2024-02-23).

29.0±1.6#646 of 719 LLMsfrom 7 benchmark results across 3 sources

$ how it compares

The LLMs ranked either side of OLMo 7B Instruct, out of 828 with a score, then more from Ai2.

#modelscorewhereRAMreleased
743LLaMA 7B Meta30±1comparison only2023-02
744MPT-7B (base) MosaicML30±1comparison only2023-05
745OLMo 7B Instruct Ai229±2local~16GB2024-02
746StableLM-3B-4E1T Stability AI29±1local~3.3GB2023-09
747Gemma 2B (base) Google29±1local~3.2GB2024-02
more from Ai2
259Olmo 3.1 32B Think Ai258±1local~21GB2025-12
262Olmo 3 32B Think Ai258±1local~21GB2025-11
314Molmo 2 8B Ai255±2local~6.8GB2025-12

see all LLMs · find the ones that fit your machine

$ what it's good at

coding█··········█ #1 of 2 * · 14% of its score · 1 test
testresultvs the others in its rundifficultycountswhy
Tulu 2: CodexEval pass@10 lab27.8% ✎▃▃▄▁▄▃▃█▃▁▁▄ beats 3/27easy · avg 83%▇▇▇▇▇▇ 14%sharp · at its level · small run
chat & writing▁▁▃▃▅▄▅█▆▆▇▃ #382 of 393 * · 47% of its score · 2 tests

Beats Phi-2 (2.7B) here, though they rank higher overall. Trails Gemma 3 270M IT, SmolLM2 360M Instruct, Granite 4.0 350M here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)995.0 ✎▂▁▂▁▃▇█▆▇▄▂▂ beats 13/95avg 1201▇▇▇▇▇▇▇▇▇ 23%blunt
LMArena (text)1073.6 ✎▁▁▃▂▃▃▃▆▆▇█▃ beats 13/379avg 1268▇▇▇▇▇▇▇▇▇ 23%blunt
math█··········█ #1 of 2 * · 14% of its score · 1 test
testresultvs the others in its rundifficultycountswhy
GSM8K lab11.0% ✎▃▂▂▅▅▆▅▇█▇▄▃ beats 3/146easy · avg 68%▇▇▇▇▇▇ 14%sharp · small run
other tests3 not in a task · 23% of its score
testresultvs the others in its rundifficultycountswhy
BIG-Bench Hard lab35.8% ✎▁▁▁▁▂▄▅▅█▃▄▁ beats 16/129medium · avg 73%▇▇▇▇▇▇ 14%sharp · small run
MMLU lab46.1% ✎▁▁▁▁▁▂▃▅█▃▆▃ beats 30/183medium · avg 73%▇▇▇ 6.3%noisy · small run
Tulu 2: TyDiQA gold passage, 1-shot F1 lab21.7% ✎▃··▂▂▃▂▂█▅▁▂ beats 0/27medium · avg 57%▇ 2.8%blunt · at its level · small run

$ where to run it

Other ways to run it

  1. Run it as published · needs ~13.78 GB · from huggingface.co/allenai
    pip install ai2-olmo && python -c "from hf_olmo import OLMoForCausalLM, OLMoTokenizerFast; o=OLMoForCausalLM.from_pretrained('allenai/OLMo-7B-Instruct'); t=OLMoTokenizerFast.from_pretrained('allenai/OLMo-7B-Instruct'); p=t.apply_chat_template([{'role': 'user', 'content': 'What is language modeling?'}], tokenize=False, add_generation_prompt=True); i=t.encode(p, add_special_tokens=False, return_tensors='pt'); print(t.batch_decode(o.generate(input_ids=i.to(o.device), max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95), skip_special_tokens=True)[0])"

Weights and documentation: huggingface.co/allenai ↗

$ how the score was worked out · OLMo: Accelerating the Scienc… 52%, Polyrating: A Cost-Effective… 23%, LMArena leaderboard dataset 23%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
OLMo: Accelerating the Science of Language Models — authors' own runs · lab-reported5▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 52%CC BY 4.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)1▇▇▇▇▇▇▇ 23%CC-BY-4.0
LMArena leaderboard dataset1▇▇▇▇▇▇▇ 23%CC BY 4.0
prior · pull toward the middle▇ 1%
weighted average= 29.0

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json