whatmodel.sh / Molmo 72B (0924)

Molmo 72B (0924)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Molmo 72B (0924) is an open-weights language model from Ai2, released September 2024. Its published size is 73.3B parameters. Licence: apache-2.0.

Catalogue identity notes

Comparison-only. Arena entry 'molmo-72b-0924' (vision). LMArena's leaderboard table (arena-leaderboard space @6f2eeeb) labels 'molmo-72b-0924' as 'Molmo-72B-0924' (AI2, Apache 2.0), linking https://huggingface.co/allenai/Molmo-72B-0924. Maker repo allenai/Molmo-72B-0924 @ab586dd: licence tag apache-2.0. 73,308,285,952 parameters from the Hugging Face safetensors metadata (dense, so active = total). Released 2024-09: month the maker's Hugging Face repo was created (createdAt 2024-09-25).

49.7±1.8#348 of 719 LLMsfrom 17 benchmark results across 4 sources

$ how it compares

The LLMs ranked either side of Molmo 72B (0924), out of 828 with a score, then more from Ai2.

#modelscorewhereRAMreleased
416Bielik 11B v2.3 Instruct SpeakLeash50±2local~8.6GB2024-08
417Qwen2.5-VL 7B Instruct Alibaba50±1local~6.4GB2025-01
418Molmo 72B (0924) Ai250±2local~309GB2024-09
419Codestral 22B v0.1 Mistral AI50±1local~15GB2024-05
420GLM-4 (0116) Z.ai50±1comparison only2024-01
more from Ai2
259Olmo 3.1 32B Think Ai258±1local~21GB2025-12
262Olmo 3 32B Think Ai258±1local~21GB2025-11
314Molmo 2 8B Ai255±2local~6.8GB2025-12

see all LLMs · find the ones that fit your machine

$ what it's good at

math▁▂▃▅▆▅▄█▄▁▁▁ #289 of 398 * · 15% of its score · 1 test

Beats GPT-4 (0613), Athene-70B, Gemma 2 27B IT here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Phi-2 (2.7B), StarCoder2 15B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MathVista (testmini) +152.2% ✎▂····▂···▂█▆ beats 4/9medium · avg 60%▇▇▇▇▇▇ 15%at its level · small run · several runs
vision & documents▁▁▃▂▃▃▄▂▆▇█▄ #122 of 147 · 99% of its score · 13 tests

Beats Gemma 3 12B, Llama 3.2 90B Vision Instruct here, though they rank higher overall. Trails Qwen3.5 2B, Qwen3-VL 2B Thinking here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Arena vision (style controlled)1044.9 ✎▁▁▂▂▃▂▄▅▅▆██ beats 21/145avg 1068▇▇▇▇▇▇▇▇ 19%blunt
MMMU (val) +152.7% ✎▂·▂······█▄▄ beats 5/9medium · avg 54%▇▇▇▇▇▇ 16%at its level · small run · several runs
MathVista (testmini) +152.2% ✎▂····▂···▂█▆ beats 4/9medium · avg 60%▇▇▇▇▇▇ 15%at its level · small run · several runs
MM-MT-Bench (Mistral, GPT-4o judge)3.5100 MM-MT-Bench score (1-10) ✎▅··▃·█·▃▃··▅ beats 0/9avg 5.740 MM-MT-Bench score (1-10)▇▇▇▇ 9.4%sharp · noisy
VQAv2 (val) +175.2% ✎▃···█▃··▅▃▃▃ beats 5/9easy · avg 74%▇▇▇▇ 9.3%blunt · at its level · several runs
InfographicVQA lab81.9% ✎▃·▃▅█▃▃▃▃▃▃▃ beats 9/13easy · avg 79%▇▇▇ 8.2%small run
CountBench lab91.2% ✎▂▂▃▂·▂▂▃▃·▅█ beats 7/18very easy · avg 91%▇▇▇ 6.7%small run
ChartQA +175.6% ✎▄·▄▄▄··▄▄▄▄█ beats 6/9easy · avg 83%▇ 3.5%blunt · small run · several runs
5 more tests
testresultvs the others in its rundifficultycountswhy
DocVQA lab93.5% ✎▁··▃▁▁▁▇█▇▄▅ beats 10/28very easy · avg 93%▇ 3.1%blunt · maxed out · small run
Ai2 Molmo human preference Elo (Sept 2024, 27 VLMs) lab1077.0 ✎▄▄·▄·▄▄▄·▄·█ beats 8/8avg 1037▇ 3.0%blunt
RealWorldQA lab75.2% ✎▂▃▂█▃██▆█▂▆▆ beats 20/38easy · avg 73%▇ 2.7%blunt · at its level · small run
AI2D test lab96.3% ✎▁▁▅▇▃▇·▄·█·▃ beats 28/28easy · avg 86%▇ 1.4%blunt · small run
TextVQA lab83.1% ✎▁▁▁▄▅▁▃·▃▃▃█ beats 17/24easy · avg 79%▇ 1.3%blunt · small run

$ where to run it

Other ways to run it

  1. Run it as published · needs ~293.23 GB, NVIDIA only · from github.com/vllm-project
    vllm serve allenai/Molmo-72B-0924 --trust-remote-code

Weights and documentation: huggingface.co/allenai ↗

$ how the score was worked out · Molmo2: Open Weights and Data… 48%, Pixtral 12B (Mistral AI, arXi… 28%, LMArena leaderboard dataset 19% and 1 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · lab-reported10▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 48%CC BY 4.0
Pixtral 12B (Mistral AI, arXiv 2410.07073v2)5▇▇▇▇▇▇▇▇▇ 28%CC BY 4.0
LMArena leaderboard dataset1▇▇▇▇▇▇ 19%CC BY 4.0
Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column · lab-reported1▇ 3%Apache-2.0
prior · pull toward the middle▇ 1%
weighted average= 49.7

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json