whatmodel.sh / Qwen3.5 9B

Qwen3.5 9B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Qwen3.5 9B is an open-weights language model from Alibaba, released March 2026. Its published size is 9.65B parameters and a 262k-token context window. Licence: apache-2.0.

62.6±0.5#168 of 719 LLMsfrom 135 benchmark results across 23 sources

$ how it compares

The LLMs ranked either side of Qwen3.5 9B, out of 828 with a score, then more from Alibaba.

#modelscorewhereRAMreleased
194Gemini 2.5 Flash Preview (04-17) Google63±1comparison only2025-04
195GLM-4.5-Air Z.ai63±1api $0.42/M +local~35GB2025-07
196Qwen3.5 9B Alibaba63±1api $0.11/M +local~7.4GB2026-03
197Seed-OSS 36B Instruct ByteDance Seed63±1local~23GB2025-08
198GPT-4.5 Preview OpenAI63±1comparison only2025-02
more from Alibaba
18Qwen3.8 Max (2.4T-A95B) Alibaba78±1api $3.00/M +local~380GB2026-08
34Qwen3.8 Max (0902) Alibaba76±1api $3.00/M2026-09
37Qwen3.8-Flash-Next Alibaba75±1local~59GB2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #142 of 300 · 16% of its score · 15 tests

Beats Qwen3 235B-A22B Thinking 2507, o4-mini, o1 (2024-12-17) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BFCL V4 (December 2025 archive) +460.6% ✎▁▁▁▂▇▆▆▆█▂▄▂ beats 98/111medium · avg 42%▇ 2.5%at its level · several runs
Terminal-Bench 2.1 +121.7% ✎▁·▁·▁▁▄▃▄█▃▇ beats 6/56very hard · avg 6.5%▇ 2.2%several runs
SWE-bench Pro (split unspecified) +138.0% ✎▃·▁▁▁▂▂▃█▇▃▁ beats 17/55very hard · avg 9.6%▇ 2.1%sharp · at its level · small run · several runs
SWE-bench Verified +150.0% ✎▁▁▁▁▁▁▂▄█▆▁▁ beats 54/131hard · avg 23%▇ 2.0%sharp · at its level · small run · several runs
ContractBench success rate (33 observation-contract tool tasks)56.6% ✎▆··▆▄▂▄▂▆▆█▄ beats 15/23very hard · avg 14%▇ 1.8%sharp · at its level
SWE-bench Multilingual43.3% ✎▁··▁▃▃▅▇█▃▃▁ beats 9/60hard · avg 18%▇ 1.0%at its level
OSWorld-Verified lab41.8% ✎█▄·▂▆▂▄▂▆▄▆█ beats 4/25hard · avg 20%▇ 0.8%at its level · small run
MCPMark13.4% ✎▃·····█··▃█▃ beats 2/8very hard · avg 5.4%▇ 0.7%typical
7 more tests
testresultvs the others in its rundifficultycountswhy
τ³-bench Banking5.1% ✎▂▂·▅█▂▂▃▂▅▂▃ beats 5/20very hard · avg 4.0%▇ 0.6%blunt · too hard for it
Thinkingbox-bench pass@1 (507 stateful business-workflow agent tasks, N=20)5.7% ✎▅··▃·▃·█▃▅██ beats 1/15very hard · avg 4.9%▇ 0.5%typical
τ²-bench aggregate lab79.1% ✎▁▁▁·▂▂▄▃▃▅█▂ beats 21/37medium · avg 37%▇ 0.5%at its level · noisy
BrowseComp8.3% ✎▅···█·▃···▃▃ beats 3/7very hard · avg 10%▇ 0.4%at its level · noisy
τ³-bench9.3% ✎▃·▃·█·▃··▃·▃ beats 3/7very hard · avg 14%▇ 0.4%blunt · at its level
BFCL v3 +170.5% ✎▁▁▁▁▁▃▅█▆▅▄▁ beats 152/170medium · avg 56%▇ 0.4%blunt · at its level · several runs
Toolathlon-Verified25.9% ✎▃·▂▂▅▃···▂██ beats 0/18hard · avg 24%▇ 0.3%blunt · at its level · small run
coding▁▁▁▃▆██▇▆▄▃▁ #181 of 458 · 13% of its score · 9 tests

Trails QwQ-32B-Preview, OpenThinker 32B, Ministral 3 8B Reasoning 2512 here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench v6 (window unspecified) +363.7% ✎▂▃▄▄▅██▅▄▇▃▆ beats 40/65hard · avg 34%▇ 3.6%at its level · several runs
SWE-bench Pro (split unspecified) +138.0% ✎▃·▁▁▁▂▂▃█▇▃▁ beats 17/55very hard · avg 9.6%▇ 2.1%sharp · at its level · small run · several runs
SWE-bench Verified +150.0% ✎▁▁▁▁▁▁▂▄█▆▁▁ beats 54/131hard · avg 23%▇ 2.0%sharp · at its level · small run · several runs
Luau Leetcode fully solved (Torpedo Software)19.2% ✎▁▁▃▁·▁▂▃▅▃█▇ beats 4/49hard · avg 18%▇ 1.7%at its level
LiveCodeBench v6 (Feb–May 2025) lab +165.6% ✎▁▁▃▁█▅▄▇▅▅▃█ beats 16/38medium · avg 40%▇ 1.6%at its level · several runs
SWE-bench Multilingual43.3% ✎▁··▁▃▃▅▇█▃▃▁ beats 9/60hard · avg 18%▇ 1.0%at its level
HumanEval+89.6% ✎▁·▁▁▂▃▄▄█▅▄▃ beats 130/145easy · avg 80%▇ 0.6%sharp · maxed out
MBPP+69.8% ✎▁··▁▁▃▄█▆▄▁▁ beats 84/120easy · avg 66%▇ 0.4%typical
1 more test
testresultvs the others in its rundifficultycountswhy
MultiPL-E67.1% ✎▃··▄·▅▅▇▄█▃▄ beats 16/31medium · avg 55%▇ 0.3%typical
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #144 of 393 · 1.0% of its score · 1 test

Beats INTELLECT-3, Nemotron 3 Super 120B-A12B, MiniMax-M2 here, though they rank higher overall. Trails LFM2.5 1.2B Thinking, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
IFEval (aggregation unspecified) +383.9% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 126/141easy · avg 80%▇ 1.0%small run · several runs
math▁▂▃▅▆▅▄█▄▁▁▁ #144 of 398 · 19% of its score · 15 tests

Beats Claude Sonnet 4.6, MiMo-V2.5 (310B-A15B), MiniMax-M2.7 here, though they rank higher overall. Trails Nemotron 3 Nano 30B-A3B, Olmo 3 32B Think, Olmo 3.1 32B Think here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HMMT February 202670.8% ✎█·▄··▄··▄█·▄ beats 6/7hard · avg 32%▇ 3.0%sharp · at its level
OTIS mock AIME 2024–202542.8% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 72/193very hard · avg 8.5%▇ 1.9%sharp · at its level · noisy
MathArena ArXivMath February 2026 (32 problems)27.3% ✎▁▁▁▃█▇▄▃▁▃·▃ beats 3/24very hard · avg 9.1%▇ 1.9%typical
MathArena ArXivMath January 2026 (23 problems)44.6% ✎▂··▄▃▄█▆▆▄▄█ beats 3/23hard · avg 22%▇ 1.8%at its level
IMO-AnswerBench69.8% ✎▄▄···▄·▄··█▄ beats 5/6hard · avg 32%▇ 1.6%at its level
MATH-Vision lab +178.9% ✎▁▁·▁··▄▁▃▇▇█ beats 14/24medium · avg 52%▇ 1.4%at its level · small run · several runs
AIME 202688.2% ✎▁▁▁▂▁▂▆█▆▂·▁ beats 16/49medium · avg 53%▇ 1.3%sharp
AIME 202556.1% ✎▁·▁▁▁▂▅▆█▇▅▁ beats 42/133hard · avg 31%▇ 1.2%sharp · at its level
7 more tests
testresultvs the others in its rundifficultycountswhy
MathArena ArXivMath December 2025 (17 problems)39.7% ✎▃·▃▃█▁·▄·▁▁▁ beats 6/18hard · avg 19%▇ 1.1%at its level
HMMT February 2025 lab83.2% ✎▁▁▁▃▁▅▅▃██▁▄ beats 14/31medium · avg 48%▇ 1.0%typical
HMMT November 2025 lab82.9% ✎▁▁····▁▅▃█▇▁ beats 7/23medium · avg 61%▇ 0.9%typical
PolyMATH lab57.3% ✎▁···▃▁·▇█▄▄▄ beats 16/23hard · avg 33%▇ 0.7%at its level · small run
MathVista (testmini) lab85.7% ✎▁··▁▄▄▇█▅▆█▄ beats 39/47medium · avg 67%▇ 0.6%small run
DynaMath lab83.6% ✎▂·····▂··█▄▆ beats 5/10easy · avg 67%▇ 0.5%small run
We-Math lab75.2% ✎█··········▃ beats 1/3medium · avg 57%▇ 0.4%small run
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #69 of 147 · 9.0% of its score · 21 tests

Beats DeepSeek V4 Flash Vision (Exp), Inkling Small (266B-A12B), GPT-5.4 nano here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
MATH-Vision lab +178.9% ✎▁▁·▁··▄▁▃▇▇█ beats 14/24medium · avg 52%▇ 1.4%at its level · small run · several runs
ChartArena chart parsing (English, mAP_high average)40.6% ✎▁·▁·▃▃▁▇▃█·▄ beats 13/22hard · avg 27%▇ 1.0%blunt · at its level
BEAR-Bench (bilingual RU/EN enterprise and academic document images, overall accuracy)49.6% ✎▃▃·▅·▃·█·█▅▅ beats 7/14hard · avg 24%▇ 0.7%at its level
CharXiv (reasoning questions) lab73.0% ✎▂▅▃▅▅▆▆▅▆█▆▂ beats 62/100medium · avg 45%▇ 0.7%at its level · small run
MathVista (testmini) lab85.7% ✎▁··▁▄▄▇█▅▆█▄ beats 39/47medium · avg 67%▇ 0.6%small run
MMMU (val) lab78.4% ✎▁····▁▁▄█▇▆▆ beats 44/52medium · avg 60%▇ 0.6%at its level · small run
MMMU Pro lab70.1% ✎▂▃▁▃▆▇▃▇▇██▅ beats 34/52medium · avg 48%▇ 0.6%at its level · small run
OmniDocBench 1.5 overall (higher is better) lab87.7% ✎▂··▂▂▂··▂▃█▅ beats 5/13easy · avg 72%▇ 0.5%small run
13 more tests
testresultvs the others in its rundifficultycountswhy
MMStar lab79.7% ✎▃▅▂▃▂▂▆█▆▆█▅ beats 28/34medium · avg 65%▇ 0.4%at its level · small run
CountBench lab97.2% ✎▂▂▃▂·▂▂▃▃·▅█ beats 14/18very easy · avg 92%▇ 0.4%maxed out · small run
PMC-VQA lab57.9% ✎▃···▃··▃▃·▃█ beats 3/7medium · avg 41%▇ 0.4%at its level · small run
SimpleVQA lab51.2% ✎▃·▃·▃·▃·█▅▃▃ beats 3/10hard · avg 34%▇ 0.4%at its level · small run
MMBench-EN dev v1.1 lab90.1% ✎█▄···▂··▄▂▄▆ beats 8/14easy · avg 82%▇ 0.3%blunt · small run
ScreenSpot-Pro lab65.2% ✎▂▃▅▆██▅▂···▂ beats 15/24medium · avg 48%▇ 0.3%blunt · at its level · small run
HallusionBench lab69.3% ✎▃▁·▃▃▃▄▅▄█▅▅ beats 29/32medium · avg 57%▇ 0.2%blunt · at its level · small run
RealWorldQA lab80.3% ✎▂▃▂█▃██▆█▂▆▆ beats 29/38easy · avg 72%▇ 0.2%blunt · small run
SpaceCast-Bench predictive spatial reasoning (overall, 16 task types)34.6% ✎▃·▅▃▅▃··▅··█ beats 2/11very hard · avg 32%▇ 0.2%blunt · noisy
CC-OCR lab79.3% ✎▁·▂··▂▂▃▃▃▆█ beats 13/26easy · avg 72%▇ 0.1%blunt · small run
OCRBench lab89.2% ✎▁·▃▁▆▅▂█▅█▁▃ beats 27/37easy · avg 83%▇ 0.1%blunt · small run
AI2D test lab90.2% ✎▁▁▅▇▃▇·▄·█·▃ beats 20/28easy · avg 85%▇ 0.1%blunt · small run
PureDocBench document parsing (Avg3 over Clean/Digital/Real)70.9% ✎▂·▁·▂▂▅█·▁▂▂ beats 17/22medium · avg 62%▇ 0.0%blunt · at its level
web search█·█▄██·▄▄·▄▄ #7 of 13 * · 0.4% of its score · 1 test

Trails Spark-X2.5 1.7B, Granite 4.2 3B, Qwen3.5 4B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BrowseComp8.3% ✎▅···█·▃···▃▃ beats 3/7very hard · avg 10%▇ 0.4%at its level · noisy
other tests50 not in a task · 49% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond +579.0% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 124/208hard · avg 40%▇▇▇▇▇ 13%sharp · at its level · several runs
MMLU-Redux +391.1% ✎▁▁▃▁▂▃▄▅▇▅█▄ beats 43/63easy · avg 78%▇ 2.8%several runs
Humanity's Last Exam14.3% ✎▄▄·▄··▄·▄·▄█ beats 7/7very hard · avg 3.8%▇ 2.3%typical
MMLU-Pro lab +182.5% ✎▁▁·▁▁▁▃▂▄▅█▁ beats 147/188medium · avg 61%▇ 1.9%several runs
IFBench (aggregation unspecified) +264.5% ✎▃▇▂▄▅▅▂▄▄▆█▇ beats 47/80medium · avg 39%▇ 1.7%at its level · several runs
GPQA (subset unspecified)77.2% ✎█···▃···▅·▃▃ beats 7/7hard · avg 45%▇ 1.6%sharp · at its level
Polish medical board exams, bias-mitigated MCQ protocol (7-exam average)63.1% ✎▄·▂▂▂··█▂▄·▄ beats 10/13hard · avg 48%▇ 1.3%at its level
LongBench v2 lab55.2% ✎▂·▁·▂▂▁▃▄▁█▄ beats 16/28very hard · avg 35%▇ 1.2%at its level
42 more tests
testresultvs the others in its rundifficultycountswhy
C-Eval lab88.2% ✎▁···▁▁▂▁▂▂▃█ beats 19/33medium · avg 73%▇ 1.2%typical
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items30.1% ✎▁▁▂▁▃▄█▆▄▅▂▃ beats 22/39very hard · avg 22%▇ 1.1%blunt
MedXPertQA MM lab49.9% ✎▂···▂·▃··▆▂█ beats 7/13very hard · avg 30%▇ 1.1%sharp · at its level · small run
MMMLU lab81.2% ✎▂▁▁▂▃▂▄▄▅██▃ beats 30/46medium · avg 63%▇ 1.0%at its level
INCLUDE lab75.6% ✎▂····▂▂▄▄▄█▄ beats 22/36medium · avg 56%▇ 0.9%at its level · small run
CPTUB average (SpeakLeash, 0-5 judge scale)65.5% ✎▁▂▂▂▃█▃▆▅▄▂▃ beats 45/95easy · avg 66%▇ 0.9%typical
Global PIQA lab83.2% ✎▃▃··▃··▃▃▅█▅ beats 5/11medium · avg 71%▇ 0.8%at its level
RobloxQA v2.0 multiple choice (Torpedo Software)74.5% ✎▃▁▃▁█▄▃▃▆▂▂▂ beats 0/49easy · avg 78%▇ 0.8%typical
VitaBench lab29.8% ✎▄··▂▄▆▆█▄▄▄▂ beats 14/21very hard · avg 10%▇ 0.8%typical
EuroEval Life in the UK (English knowledge MC), validation split, 5-shot, accuracy89.8% ✎▂·▂▂·▂·▃▃·█▃ beats 7/14easy · avg 80%▇ 0.8%typical
MultiChallenge lab54.5% ✎▁·▁·▁▃▆▅▃▁█▅ beats 17/29hard · avg 30%▇ 0.8%at its level
MAXIFE lab83.4% ✎▂·▂·▂···▆·▄█ beats 6/11medium · avg 63%▇ 0.7%typical
GSM-Plus +287.9% ✎▁··▁▁▁·▃▁▄█▃ beats 16/16easy · avg 83%▇ 0.7%blunt · several runs
MMLU-ProX lab76.3% ✎▁·▁▁▁▂▁▃▃█▅▄ beats 13/35medium · avg 56%▇ 0.7%at its level
EuroEval HellaSwag (English), validation split, 5-shot generative MC, accuracy82.1% ✎▂··▂▂·▂▆▆█·▄ beats 12/15medium · avg 63%▇ 0.7%at its level
OJBench lab29.2% ✎▁···▁▃·▅▄▅██ beats 15/26very hard · avg 13%▇ 0.6%at its level · small run
NOVA-63 lab55.9% ✎▃▃·▃···▃█▃▃█ beats 7/11medium · avg 38%▇ 0.5%blunt · at its level
SuperGPQA lab58.2% ✎▂▁▂▂▅▆▃▇▇▇█▃ beats 31/46medium · avg 37%▇ 0.5%at its level · small run
VideoMMMU lab78.9% ✎▂···▂···▂▃█▆ beats 3/10medium · avg 61%▇ 0.5%at its level · small run
VLMs Are Blind lab93.7% ✎▂·▂·····▃··█ beats 3/8very easy · avg 86%▇ 0.4%maxed out · small run
Chess Puzzles (Epoch)2.0% ✎█▁▁▁▁▂▁▄▂▂▁▁ beats 22/135very hard · avg 5.9%▇ 0.4%sharp · too hard for it · noisy
LVBench lab70.0% ✎▇·█▅▅▄▄··▃▁▁ beats 19/28medium · avg 53%▇ 0.4%at its level · small run
Video-MME (with subtitles) lab84.5% ✎▁▁▁▁▁·▁▃▁▁▃█ beats 10/16medium · avg 72%▇ 0.4%small run
SLAKE lab79.0% ✎▄·····▄▄·██▄ beats 4/7medium · avg 63%▇ 0.4%small run
MMVU lab67.8% ✎▃···▃···▃▃▅█ beats 3/8medium · avg 49%▇ 0.4%at its level · small run
ERQA lab55.5% ✎▃··▃··█·▃█▅▅ beats 4/12medium · avg 37%▇ 0.4%at its level · small run
RefSpatial-Bench lab58.5% ✎▃·▃····▃▃▅▅█ beats 3/10medium · avg 40%▇ 0.4%at its level · small run
LingoQA lab80.4% ✎▄······▄·█·█ beats 2/5medium · avg 59%▇ 0.4%small run
ZEROBench subquestions lab31.1% ✎▄·▄···▄·▄██▄ beats 3/8hard · avg 17%▇ 0.4%at its level · small run
Claw-Eval average score66.5% ✎▄·▄··█▄▄▄▄▄▄ beats 6/9medium · avg 53%▇ 0.4%blunt · at its level
MLVU lab84.4% ✎▂···▂···▂▄▄█ beats 4/10easy · avg 71%▇ 0.4%small run
MMLongBench-Doc lab57.7% ✎▃····▃··▃▃▅█ beats 4/8medium · avg 38%▇ 0.4%at its level · small run
AutomationBench v1.0.65.0% ✎▃····▃·▃▃··█ beats 0/6very hard · avg 5.5%▇ 0.4%noisy · small run
EmbSpatial-Bench lab83.0% ✎▁·····▁·▁·▃█ beats 3/10easy · avg 70%▇ 0.4%small run
AndroidWorld lab57.8% ✎▁·▁▁▂▃█▅▂▂·▂ beats 13/24medium · avg 41%▇ 0.4%blunt · at its level · small run
NTCIR SciClaimEval Subtask 1 pair-accuracy (dev, single model)48.3% ✎▃▃···▃·▃▃█·▃ beats 0/8medium · avg 45%▇ 0.3%blunt · at its level
Video-MME (without subtitles) lab78.4% ✎▁▃▁▃▁·█▄▄▅·▇ beats 20/27medium · avg 67%▇ 0.3%blunt · at its level · small run
DeepPlanning lab18.0% ✎▂·····▂··▄█▄ beats 3/9very hard · avg 7.1%▇ 0.2%noisy
MVBench lab74.4% ✎▁··▃▄▁▃▄▄▄█▄ beats 18/26medium · avg 66%▇ 0.1%blunt · at its level · small run
Multi-IF (aggregation unspecified)62.5% ✎▁▁▁▁▁▃▃▄█▄▁▂ beats 7/41medium · avg 63%▇ 0.1%blunt · noisy
LIBRA Mini (Russian long context, May 2026 version)41.5% ✎▄···▄█·██··▄ beats 4/8medium · avg 38%▇ 0.1%blunt · at its level
WMT24++ lab72.6% ✎▁·▁▁·▁·▁▃▆█▃ beats 6/27easy · avg 68%▇ 0.1%blunt · at its level

$ where to run it

Through an API

api · Deep Infra · $0.11/M · $0.1 in / $0.15 out per 1M tokens · 262k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Qwen3.5 9B · needs ~5.6 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Qwen3.5 9B · needs ~5.6 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Qwen3.5 9B · needs ~6.29 GB · from ollama.com
    chat in the terminal
    ollama run qwen3.5:9b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Qwen3.5 9B · needs ~6 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/Qwen3.5-9B-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/Qwen3.5-9B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3.5-9B model card 33%, XHToken/Spark-X2.5-4B model c… 13%, JetBrains/Mellum2.1-12B-A2.5B… 9% and 20 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Qwen/Qwen3.5-9B model card · lab-reported62▇▇▇▇▇▇▇▇▇▇ 33%apache-2.0
XHToken/Spark-X2.5-4B model card10▇▇▇▇ 13%Apache-2.0
JetBrains/Mellum2.1-12B-A2.5B-Thinking model card11▇▇▇ 9%Apache-2.0
JetBrains/Mellum2-12B-A2.5B-Instruct model card8▇▇ 6%Apache-2.0
Epoch AI — internal evaluations3▇▇ 6%CC BY 4.0
Qwen/Qwen3.5-4B model card · lab-reported4▇▇ 5%apache-2.0
MathArena ArXivMath / BrokenArXiv / ArXivLean model-output datasets3▇ 5%CC BY-SA 4.0
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B model card5▇ 4%MIT
LiquidAI/LFM2.5-2.6B model card7▇ 4%LFM Open License v1.0
JetBrains/Mellum2-12B-A2.5B-Thinking model card7▇ 3%Apache-2.0
Roblox LLM Leaderboard (Torpedo Software): RobloxQA v2.0 and Luau Leetcode2▇ 2%MIT
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)1▇ 2%CC BY 4.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 1%MIT
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? (arXiv:2606.12250v1)1▇ 1%CC BY-SA 4.0
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)1▇ 1%CC BY 4.0
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats (arXiv:2606.01348v3)1▇ 1%CC BY 4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 1%MIT
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models (arXiv:2608.17895v1)1▇ 1%CC BY 4.0
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741v4)1▇ 1%CC BY 4.0
SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task — authors' own single-model dev evaluation1▇ 0%CC BY 4.0
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models (arXiv:2610.12402v1)1▇ 0%CC BY 4.0
ai-forever LIBRA leaderboard (Russian long-context benchmark)1▇ 0%MIT
How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings (arXiv:2605.07492v2)1▇ 0%CC BY 4.0
prior · pull toward the middle▇ 0%
weighted average= 62.6
$ quantization evidence · 18 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.825 → 0.82499.9%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.947 → 0.94599.8%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.84 → 0.83899.8%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.85 → 0.84799.6%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.883 → 0.87999.5%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.903 → 0.88998.4%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.944 → 0.946100.2%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.824 → 0.81799.2%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.852 → 0.83898.4%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.895 → 0.87798.0%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.822 → 0.77994.8%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.854 → 0.68880.6%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.822 → 0.823100.1%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.852 → 0.853100.1%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.895 → 0.896100.1%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.854 → 0.854100.0%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.944 → 0.94399.9%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.824 → 0.8299.5%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json