whatmodel.sh / Molmo 7B-D (0924)

Molmo 7B-D (0924)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Molmo 7B-D (0924) is an open-weights language model from Ai2, released September 2024. Its published size is 8.02B parameters. Licence: apache-2.0.

Catalogue identity notes

Comparison-only. Arena entry 'molmo-7b-d-0924' (vision). LMArena's leaderboard table (arena-leaderboard space @6f2eeeb) labels 'molmo-7b-d-0924' as 'Molmo-7B-D-0924' (AI2, Apache 2.0), linking https://huggingface.co/allenai/Molmo-7B-D-0924. Maker repo allenai/Molmo-7B-D-0924 @cab33fb: licence tag apache-2.0. 8,021,025,280 parameters from the Hugging Face safetensors metadata (dense, so active = total). Released 2024-09: month the maker's Hugging Face repo was created (createdAt 2024-09-25).

40.5±1.4#525 of 719 LLMsfrom 21 benchmark results across 5 sources

$ how it compares

The LLMs ranked either side of Molmo 7B-D (0924), out of 828 with a score, then more from Ai2.

#modelscorewhereRAMreleased
615Granite 4.0 1B IBM41±1local~2.4GB2025-10
616Mixtral 8x7B Instruct v0.1 Mistral41±1local~29GB2023-12
617Molmo 7B-D (0924) Ai241±1local~35GB2024-09
618OpenHermes-2.5-Code-290k-13B ajibawa-202341±1local~9.8GB2024-03
619Granite 4.0 H 1B IBM40±1local~2.3GB2025-10
more from Ai2
259Olmo 3.1 32B Think Ai258±1local~21GB2025-12
262Olmo 3 32B Think Ai258±1local~21GB2025-11
314Molmo 2 8B Ai255±2local~6.8GB2025-12

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▃▁▃▆██▇▆▄▃▁ #453 of 458 * · 5.6% of its score · 1 test

Trails Phi-1 (1.3B), OpenAssistant SFT-4 Pythia 12B, Falcon3 1B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HumanEval3.7% ✎▁▁▃▁▁▃▅▅█▃▂▁ beats 5/190easy · avg 72%▇▇ 5.6%sharp · at its level · noisy
math▁▃▃▅▆▅▄█▄▁▁▁ #383 of 398 · 14% of its score · 2 tests

Beats Command R (03-2024) here, though they rank higher overall. Trails Qwen2 0.5B Instruct, Granite 3.1 1B-A400M Instruct, Gemma 2B (base) here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MathVista (testmini) +112.3% ✎▃····▂···▂█▆ beats 0/9medium · avg 60%▇▇▇▇ 8.9%at its level · small run · several runs
MATH (subset unspecified)10.2% ✎▃▄▄▃█▇▅▆▄▃▄▁ beats 0/80medium · avg 55%▇▇ 5.4%at its level
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #137 of 147 · 75% of its score · 14 tests

Beats LLaVA-OneVision Qwen2 72B (ov-sft), Gemma 3 4B, CogVLM2 Llama3 Chat 19B here, though they rank higher overall. Trails Qwen3.5 0.8B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Arena vision (style controlled)993.8 ✎▁▁▂▃▂▂▄▅▅▆██ beats 9/145avg 1068▇▇▇▇▇ 13%blunt
MathVista (testmini) +112.3% ✎▃····▂···▂█▆ beats 0/9medium · avg 60%▇▇▇▇ 8.9%at its level · small run · several runs
MMMU (val) +124.3% ✎▂·▃······█▄▄ beats 1/9medium · avg 54%▇▇▇ 8.7%at its level · small run · several runs
VQAv2 (val) +157.1% ✎▃···█▃··▅▃▃▃ beats 0/9easy · avg 74%▇▇▇ 7.3%blunt · at its level · several runs
Kaleidoscope (in-language multimodal exams, 18 languages, overall accuracy)32.9% ✎█·█···█·█·██ beats 0/5medium · avg 47%▇▇▇ 6.8%at its level
InfographicVQA lab72.6% ✎▃·▃▅█▃▃▃▃▃▃▃ beats 6/13easy · avg 79%▇▇▇ 6.7%at its level · small run
MM-MT-Bench (Mistral, GPT-4o judge)3.7200 MM-MT-Bench score (1-10) ✎▅··▃·█·▃▃··▅ beats 1/9avg 5.740 MM-MT-Bench score (1-10)▇▇▇ 6.3%sharp · noisy
CountBench lab88.5% ✎▂▂▃▂·▂▃▃▃·▅█ beats 6/18very easy · avg 91%▇▇ 6.1%small run
6 more tests
testresultvs the others in its rundifficultycountswhy
ChartQA +127.0% ✎▄·▄▄▄··▄▄▄▄█ beats 1/9easy · avg 83%▇ 2.8%blunt · small run · several runs
DocVQA lab92.2% ✎▁··▃▁▁▃▇█▇▄▅ beats 5/28very easy · avg 93%▇ 2.8%blunt · small run
RealWorldQA lab70.7% ✎▂▃▂█▃██▆█▂▆▆ beats 13/38easy · avg 73%▇ 2.1%blunt · at its level · small run
Ai2 Molmo human preference Elo (Sept 2024, 27 VLMs) lab1056.0 ✎▄▄·▄·▄▄▄·▄·█ beats 6/8avg 1037▇ 2.1%blunt
AI2D test lab93.2% ✎▁▁▅▇▃▇·▄·█·▃ beats 24/28easy · avg 86%▇ 1.0%blunt · small run
TextVQA lab81.7% ✎▁▁▁▄▅▁▃·▃▃▃█ beats 15/24easy · avg 79%▇ 1.0%blunt · at its level · small run
other tests1 not in a task · 13% of its score
testresultvs the others in its rundifficultycountswhy
MMLU61.2% ✎▁▁▁▁▁▂▃▅█▃▆▃ beats 73/183easy · avg 76%▇▇▇▇▇ 13%at its level

$ where to run it

Other ways to run it

  1. Run it as published · needs ~32.08 GB, NVIDIA only · from github.com/vllm-project
    vllm serve allenai/Molmo-7B-D-0924 --trust-remote-code

Weights and documentation: huggingface.co/allenai ↗

$ how the score was worked out · Pixtral 12B (Mistral AI, arXi… 42%, Molmo2: Open Weights and Data… 36%, LMArena leaderboard dataset 13% and 2 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Pixtral 12B (Mistral AI, arXiv 2410.07073v2)8▇▇▇▇▇▇▇▇▇▇▇▇▇ 42%CC BY 4.0
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Ai2, arXiv 2601.10611v4) · lab-reported10▇▇▇▇▇▇▇▇▇▇▇ 36%CC BY 4.0
LMArena leaderboard dataset1▇▇▇▇ 13%CC BY 4.0
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (arXiv:2504.07072v2)1▇▇ 7%CC-BY-4.0
Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column · lab-reported1▇ 2%Apache-2.0
prior · pull toward the middle▇ 1%
weighted average= 40.5

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json