whatmodel.sh / Olmo 3 7B Instruct

Olmo 3 7B Instruct

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Olmo 3 7B Instruct is an open-weights language model from Ai2, released November 2025. Its published size is 7.3B parameters and a 66k-token context window. Licence: apache-2.0.

45.5±0.8#434 of 719 LLMsfrom 29 benchmark results across 5 sources

$ how it compares

The LLMs ranked either side of Olmo 3 7B Instruct, out of 828 with a score, then more from Ai2.

#modelscorewhereRAMreleased
517LFM2 8B-A1B Liquid AI46±1comparison only2025-10
518OpenChat 3.5 0106 Gemma OpenChat46±3local~6.9GB2024-03
519Olmo 3 7B Instruct Ai246±1local~5.8GB2025-11
520InternLM2.5 20B Chat Shanghai AI Laboratory (InternLM)46±2local~14GB2024-08
521StarCoder2-15B-Instruct-v0.1 BigCode46±1local~11GB2024-04
more from Ai2
259Olmo 3.1 32B Think Ai258±1local~21GB2025-12
262Olmo 3 32B Think Ai258±1local~21GB2025-11
314Molmo 2 8B Ai255±2local~6.8GB2025-12

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #288 of 300 · 3.4% of its score · 3 tests

Trails Granite 4.0 350M, Granite 4.0 H 350M, ChatGLM3-6B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BFCL (version not stated) lab49.8% ✎▄▆▄▂·█▆·▄··▂ beats 0/17easy · avg 66%▇ 1.3%blunt · at its level
BFCL V4 (December 2025 archive)19.8% ✎▁▁▁▃▇▆▆▆█▂▄▂ beats 6/111medium · avg 36%▇ 1.3%at its level
BFCL v341.9% ✎▁▁▁▁▁▃▅█▆▅▄▁ beats 21/170medium · avg 56%▇ 0.8%blunt · at its level
coding▁▁▁▃▆██▇▆▄▃▁ #340 of 458 · 14% of its score · 5 tests

Beats LFM2.5 2.6B, LFM2.5 8B-A1B, Qwen3-VL 2B Thinking here, though they rank higher overall. Trails Phi-3-mini-128k-instruct (Apr 2024), DeepSeek LLM 67B Chat, WizardLM 70B V1.0 here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HumanEval+ lab77.2% ✎▂·······▃·▂█ beats 1/6easy · avg 80%▇▇ 4.1%sharp · at its level
LiveCodeBench v6 (window unspecified)28.2% ✎▂▃▄▄▅██▅▄▇▃▆ beats 2/65hard · avg 34%▇ 3.7%at its level
MBPP+ lab60.2% ✎▄▄····▄·▄··█ beats 0/5medium · avg 62%▇ 2.5%at its level
LiveCodeBench v3 lab29.5% ✎▃▃·····▃·▃·█ beats 0/6easy · avg 65%▇ 2.3%at its level
MultiPL-E36.1% ✎▃··▄·▅▅▇▄█▃▄ beats 3/31medium · avg 55%▇ 1.1%at its level
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #218 of 393 · 4.4% of its score · 2 tests

Beats Granite 4.2 8B, Mistral Small 4 (119B-A6.5B), Olmo 3.1 32B Think here, though they rank higher overall. Trails LFM2.5 1.2B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AlpacaEval 2 length-controlled lab40.9% ✎█···█·█·█·██ beats 1/5medium · avg 44%▇ 2.9%at its level
IFEval (aggregation unspecified) lab +185.6% ✎▄▄·····█▄▄·▄ beats 2/6easy · avg 76%▇ 1.5%at its level · several runs
math▁▂▃▅▆▅▄█▄▁▁▁ #200 of 398 · 20% of its score · 4 tests

Beats MiniMax-M2.1, Qwen3 235B-A22B (Apr 2025, hybrid), DeepSeek V3 (Dec '24) here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
AIME 2024 lab44.3% ✎▄·····▄··▄██ beats 1/6medium · avg 59%▇▇▇▇ 10%sharp · at its level
AIME 2025 lab32.5% ✎▄··▄····█▄▄▄ beats 1/6medium · avg 51%▇▇▇ 8.0%sharp · at its level
OMEGA (math generalisation) lab28.9% ✎█···▄·▄·▄··█ beats 5/6hard · avg 20%▇ 1.4%typical
MATH (subset unspecified) lab87.3% ✎██····██·███ beats 5/6easy · avg 71%▇ 1.3%at its level · noisy
other tests14 not in a task · 58% of its score
testresultvs the others in its rundifficultycountswhy
BIG-Bench Hard lab71.2% ✎▄·····▄·▄·██ beats 1/6easy · avg 78%▇▇▇▇▇ 12%sharp · at its level
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)0.3503 ✎▁▄▃▃▇▄█▃▅▁▁▁ beats 1/30avg 0.423▇▇▇ 8.5%sharp
GPQA (subset unspecified) lab40.4% ✎▃······▃·▅·█ beats 1/6hard · avg 43%▇▇ 5.9%sharp
MMLU lab69.1% ✎▄···▄··█··▄█ beats 1/6easy · avg 77%▇▇ 5.8%at its level
GPQA Diamond40.9% ✎▁▁▁▁▂▄▅▅███▄ beats 52/310hard · avg 47%▇▇ 5.5%sharp
MMLU-Redux71.8% ✎▁▁▃▁▂▃▄▅▇▅█▄ beats 4/63easy · avg 78%▇▇ 4.7%at its level
AGIEval English lab64.4% ✎▄····▄·▄▄··█ beats 0/5easy · avg 76%▇▇ 4.3%at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)35.0% ✎▂▃▅▂▆▃█▅█▅▃▆ beats 1/34medium · avg 42%▇▇ 3.9%blunt · at its level
6 more tests
testresultvs the others in its rundifficultycountswhy
ZebraLogic lab32.9% ✎█▄····▄·▄·▄▄ beats 1/6medium · avg 53%▇ 2.3%at its level
PopQA lab14.1% ✎▄··█···▄···█ beats 0/5hard · avg 22%▇ 1.3%blunt
IFBench (aggregation unspecified) lab32.3% ✎█·██·██····█ beats 1/5medium · avg 36%▇ 1.2%at its level
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy61.3% ✎▃·▁▁▂▁▂▅█▇▄▁ beats 36/100medium · avg 68%▇ 1.0%at its level · noisy
GSM-Plus85.8% ✎▁··▁▁▁·▃▁▄█▁ beats 7/16easy · avg 82%▇ 0.9%blunt
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy75.1% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 49/146easy · avg 83%▇ 0.8%at its level · noisy

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Olmo 3 7B Instruct · needs ~4.5 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/Olmo-3-7B-Instruct-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/Olmo-3-7B-Instruct-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Olmo 3 7B Instruct · needs ~4.5 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/Olmo-3-7B-Instruct-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Olmo 3 7B Instruct · needs ~4.88 GB · from ollama.com
    chat in the terminal
    ollama run olmo-3:7b-instruct

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Olmo 3 7B Instruct · needs ~4.1 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/Olmo-3-7B-Instruct-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/Olmo-3-7B-Instruct-MLX-4bit

Weights and documentation: huggingface.co/allenai ↗

$ how the score was worked out · Olmo 3 63%, JetBrains/Mellum2-12B-A2.5B-I… 19%, Medmarks: A Comprehensive Ope… 12% and 2 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Olmo 3 · lab-reported14▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 63%CC BY 4.0
JetBrains/Mellum2-12B-A2.5B-Instruct model card8▇▇▇▇▇▇ 19%Apache-2.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models2▇▇▇▇ 12%CC BY 4.0
allenai/Olmo-3-7B-Instruct model card · lab-reported3▇ 4%Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 2%MIT
prior · pull toward the middle▇ 0%
weighted average= 45.5

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json