whatmodel.sh / Qwen3.5 4B

Qwen3.5 4B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Qwen3.5 4B is an open-weights language model from Alibaba, released March 2026. Its published size is 4.66B parameters and a 262k-token context window. Licence: apache-2.0.

60.5±0.5#188 of 719 LLMsfrom 128 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of Qwen3.5 4B, out of 828 with a score, then more from Alibaba.

#modelscorewhereRAMreleased
224Gemini 2.0 Pro Experimental (02-05) Google61±1comparison only2025-02
225Qwen3 30B-A3B Thinking 2507 Alibaba61±1api $0.75/M +local~20GB2025-07
226Qwen3.5 4B Alibaba61±1local~4.3GB2026-03
227Nemotron-Cascade-14B-Thinking NVIDIA61±1local~10GB2025-12
228Mellum2.1 12B-A2.5B Thinking JetBrains60±2local~8.3GB2026-09
more from Alibaba
18Qwen3.8 Max (2.4T-A95B) Alibaba78±1api $3.00/M +local~380GB2026-08
34Qwen3.8 Max (0902) Alibaba76±1api $3.00/M2026-09
37Qwen3.8-Flash-Next Alibaba75±1local~59GB2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #158 of 300 · 11% of its score · 13 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BFCL V4 (December 2025 archive) +456.8% ✎▄·█·▄▄▄▄·▄·▄ beats 6/8medium · avg 43%▇ 2.9%at its level · several runs
SWE-bench Multilingual27.7% ✎▁··▁▃▂▅▇█▃▃▁ beats 3/60hard · avg 18%▇ 1.0%at its level
SWE-bench Verified33.6% ✎▄··▄▄█··▄·▄█ beats 6/8very hard · avg 12%▇ 1.0%sharp · at its level · noisy
BrowseComp-ZH39.6% ✎▄▄·▄▄▄·█···█ beats 7/8very hard · avg 13%▇ 1.0%at its level
SWE-bench Pro (split unspecified)28.2% ✎█▂···▂···▄·▃ beats 8/8very hard · avg 4.3%▇ 0.8%sharp · noisy
OSWorld-Verified lab35.6% ✎█▄·▂▆▂▄▂▆▄▆█ beats 0/25hard · avg 20%▇ 0.8%at its level · small run
MCPMark12.5% ✎▃·····█··▃█▃ beats 1/8very hard · avg 5.4%▇ 0.7%typical
τ³-bench Banking5.5% ✎▂▂·▅█▂▂▃▂▅▂▃ beats 6/20very hard · avg 4.0%▇ 0.5%blunt · too hard for it
5 more tests
testresultvs the others in its rundifficultycountswhy
τ²-bench aggregate lab79.9% ✎▁▁▁·▂▂▄▃▃▅█▂ beats 24/37medium · avg 37%▇ 0.5%at its level · noisy
BFCL v3 +164.1% ✎▁▁▁▁▁▃▅█▆▅▄▁ beats 164/170medium · avg 56%▇ 0.5%blunt · at its level · several runs
BrowseComp14.3% ✎▅···█·▃···▃▃ beats 5/7very hard · avg 10%▇ 0.4%noisy
τ³-bench6.7% ✎▃·▃·█·▃··▃·▃ beats 1/7very hard · avg 14%▇ 0.4%blunt
τ²-bench Telecom87.7% ✎▄█▄········█ beats 4/5hard · avg 24%▇ 0.4%sharp · at its level · noisy
coding▁▁▁▃▆██▇▆▄▃▁ #233 of 458 · 13% of its score · 9 tests

Trails QwQ-32B-Preview, InternLM2.5 20B Chat, Gemma 2 27B IT here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench v6 (window unspecified) +356.4% ✎▃····█·▃▅▃·▃ beats 6/8hard · avg 34%▇▇ 4.5%at its level · several runs
LiveCodeBench Pro 2025Q2 (Medium)7.0% ✎█···▂··▄▃··▂ beats 7/8very hard · avg 2.0%▇ 1.8%sharp · too hard for it · noisy
LiveCodeBench v6 (Feb–May 2025) lab +155.8% ✎▁▁▃▁█▅▄▇▅▅▃█ beats 10/38medium · avg 40%▇ 1.7%at its level · several runs
LiveCodeBench Pro 2025Q2 (Easy)58.3% ✎▄···█▄··▄█▄▄ beats 7/8hard · avg 33%▇ 1.3%at its level
SWE-bench Multilingual27.7% ✎▁··▁▃▂▅▇█▃▃▁ beats 3/60hard · avg 18%▇ 1.0%at its level
SWE-bench Verified33.6% ✎▄··▄▄█··▄·▄█ beats 6/8very hard · avg 12%▇ 1.0%sharp · at its level · noisy
SWE-bench Pro (split unspecified)28.2% ✎█▂···▂···▄·▃ beats 8/8very hard · avg 4.3%▇ 0.8%sharp · noisy
SciCode (level unspecified)24.0% ✎█·····██████ beats 3/6hard · avg 17%▇ 0.6%at its level
1 more test
testresultvs the others in its rundifficultycountswhy
MultiPL-E51.0% ✎▃··▄·▅▅▇▄█▃▄ beats 8/31medium · avg 55%▇ 0.3%at its level
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #159 of 393 · 0.9% of its score · 1 test

Beats MiniMax-M2, MiniCPM5 2B, Granite 4.2 30B here, though they rank higher overall. Trails LFM2.5 1.2B Thinking, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
IFEval (aggregation unspecified) +390.2% ✎▄▄····▄··███ beats 5/8easy · avg 80%▇ 0.9%small run · several runs
math▁▂▃▅▆▅▄█▄▁▁▁ #149 of 398 · 23% of its score · 13 tests

Beats MiMo-V2.5 (310B-A15B), MiniMax-M2.7, MiniMax-M2.1 here, though they rank higher overall. Trails Nemotron 3 Nano 30B-A3B, Qwen3-VL 8B Thinking, Olmo 3 32B Think here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HMMT February 2026 +164.0% ✎▅··▃▃·▃··▃·█ beats 8/8hard · avg 32%▇▇ 6.2%sharp · at its level · several runs
AIME 2025 +178.8% ✎█▄▄▄▄····█·▄ beats 6/8medium · avg 42%▇ 3.2%sharp · at its level · several runs
AIME 2026 +182.7% ✎▄·▄█·▄▄···█▄ beats 6/8medium · avg 53%▇ 3.1%sharp · several runs
OTIS mock AIME 2024–202555.8% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 84/193very hard · avg 8.5%▇ 2.1%sharp · at its level · noisy
IMO-AnswerBench68.5% ✎▄▄···▄·▄··█▄ beats 4/6hard · avg 32%▇ 1.8%at its level
MATH-Vision lab +174.6% ✎▁▁·▁··▄▁▃▇▇█ beats 12/24medium · avg 52%▇ 1.5%at its level · small run · several runs
HMMT February 2025 lab74.0% ✎▁▁▁▃▁▅▅▃██▁▄ beats 12/31medium · avg 48%▇ 1.2%at its level
HMMT November 2025 lab76.8% ✎▁▁····▁▅▃█▇▁ beats 4/23medium · avg 61%▇ 1.0%typical
5 more tests
testresultvs the others in its rundifficultycountswhy
MATH-50099.0% ✎██▄▄▄··▄···▄ beats 8/8very easy · avg 89%▇ 0.7%maxed out
PolyMATH lab51.1% ✎▁···▃▁·▇█▄▄▄ beats 13/23hard · avg 33%▇ 0.7%at its level · small run
MathVista (testmini) lab85.1% ✎▁··▁▄▄▇█▅▆█▄ beats 38/47medium · avg 67%▇ 0.7%small run
DynaMath lab83.3% ✎▂·····▂··█▄▆ beats 4/10easy · avg 67%▇ 0.5%small run
We-Math lab75.4% ✎█··········▃ beats 2/3medium · avg 57%▇ 0.5%at its level · small run
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #84 of 147 · 10% of its score · 21 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-5 mini, GLM-4.6V here, though they rank higher overall. Trails GPT-4.1 mini here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MATH-Vision lab +174.6% ✎▁▁·▁··▄▁▃▇▇█ beats 12/24medium · avg 52%▇ 1.5%at its level · small run · several runs
ChartArena chart parsing (English, mAP_high average)33.5% ✎▁·▁·▃▃▁▇▃█·▄ beats 8/22hard · avg 27%▇ 1.1%blunt · at its level
BEAR-Bench (bilingual RU/EN enterprise and academic document images, overall accuracy)48.6% ✎▃▃·▅·▃·█·█▅▅ beats 5/14hard · avg 24%▇ 0.7%at its level
CharXiv (reasoning questions) lab70.8% ✎▂▅▃▅▅▆▆▅▆█▆▂ beats 57/100medium · avg 45%▇ 0.7%at its level · small run
MathVista (testmini) lab85.1% ✎▁··▁▄▄▇█▅▆█▄ beats 38/47medium · avg 67%▇ 0.7%small run
MMMU (val) lab77.6% ✎▁····▁▁▄█▇▆▆ beats 42/52medium · avg 60%▇ 0.7%at its level · small run
MMMU Pro lab66.3% ✎▂▃▁▃▆▇▃▇▇██▅ beats 28/52medium · avg 48%▇ 0.6%at its level · small run
OmniDocBench 1.5 overall (higher is better) lab86.2% ✎▂··▂▂▂··▃▂█▅ beats 4/13easy · avg 72%▇ 0.5%small run
13 more tests
testresultvs the others in its rundifficultycountswhy
MMStar lab78.3% ✎▃▅▂▃▂▂▆█▆▆█▅ beats 24/34medium · avg 65%▇ 0.5%at its level · small run
CountBench lab96.3% ✎▂▂▃▂·▂▂▃▃·▅█ beats 12/18very easy · avg 92%▇ 0.5%maxed out · small run
PMC-VQA lab55.5% ✎▃···▃··▃▃·▃█ beats 2/7medium · avg 41%▇ 0.4%at its level · small run
SimpleVQA lab43.4% ✎▃·▃·▃·▃·█▅▃▃ beats 2/10hard · avg 34%▇ 0.4%at its level · small run
MMBench-EN dev v1.1 lab89.4% ✎█▄···▂··▄▂▄▆ beats 7/14easy · avg 82%▇ 0.3%blunt · small run
ScreenSpot-Pro lab60.3% ✎▂▃▅▆██▅▂···▂ beats 11/24medium · avg 48%▇ 0.3%blunt · at its level · small run
HallusionBench lab65.0% ✎▃▁·▃▃▃▄▅▄█▅▅ beats 19/32medium · avg 57%▇ 0.2%blunt · at its level · small run
RealWorldQA lab79.5% ✎▂▃▂█▃██▆█▂▆▆ beats 28/38easy · avg 72%▇ 0.2%blunt · small run
CC-OCR lab76.7% ✎▁·▂··▂▂▃▃▂▆█ beats 9/26easy · avg 72%▇ 0.2%blunt · small run
SpaceCast-Bench predictive spatial reasoning (overall, 16 task types)33.7% ✎▃·▅▃▅▃··▅··█ beats 1/11very hard · avg 32%▇ 0.2%blunt · noisy
OCRBench lab85.0% ✎▁·▃▁▆▅▃█▅█▁▃ beats 15/37easy · avg 83%▇ 0.1%blunt · small run
AI2D test lab89.6% ✎▁▁▅▇▃▇·▄·█·▃ beats 19/28easy · avg 85%▇ 0.1%blunt · small run
PureDocBench document parsing (Avg3 over Clean/Digital/Real)69.8% ✎▂·▁·▂▂▅█·▁▂▂ beats 14/22medium · avg 62%▇ 0.0%blunt · at its level
web search█·█▄██·▄▄·▄▄ #4 of 13 * · 1.4% of its score · 2 tests

Beats Qwen3.5 9B, Gemma 4 12B here, though they rank higher overall. Trails Spark-X2.5 1.7B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BrowseComp-ZH39.6% ✎▄▄·▄▄▄·█···█ beats 7/8very hard · avg 13%▇ 1.0%at its level
BrowseComp14.3% ✎▅···█·▃···▃▃ beats 5/7very hard · avg 10%▇ 0.4%noisy
other tests44 not in a task · 47% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond +376.8% ✎▁▁▁▁▂▄▅▅███▄ beats 184/310hard · avg 48%▇▇▇ 8.3%sharp · at its level · several runs
MMLU-Redux +388.7% ✎▂···█··▂▂··▃ beats 8/8easy · avg 78%▇ 3.3%several runs
MMLU-Pro +278.0% ✎▂··▂█·▂▂···▃ beats 8/8medium · avg 61%▇ 3.2%several runs
IFBench (aggregation unspecified) +359.0% ✎▅····▃▃·█·▃▃ beats 5/8medium · avg 39%▇ 2.4%at its level · several runs
LongBench v2 +147.3% ✎▃······▅▅▃·█ beats 8/8very hard · avg 35%▇ 2.2%at its level · several runs
Humanity's Last Exam8.6% ✎▄▄·▄··▄·▄·▄█ beats 4/7very hard · avg 3.8%▇ 2.2%typical
GPQA (subset unspecified)67.2% ✎█···▃···▅·▃▃ beats 4/7hard · avg 45%▇ 1.8%sharp · at its level
OJBench +124.8% ✎▃····▃▅··█▃▃ beats 7/8very hard · avg 12%▇ 1.8%several runs
36 more tests
testresultvs the others in its rundifficultycountswhy
C-Eval lab85.1% ✎▁···▁▁▂▁▂▂▃█ beats 14/33medium · avg 73%▇ 1.4%typical
SuperGPQA +152.8% ✎▃·▃·▃▅█····▃ beats 8/8hard · avg 32%▇ 1.3%at its level · several runs
MedXPertQA MM lab42.9% ✎▂···▂·▃··▆▂█ beats 4/13very hard · avg 30%▇ 1.2%sharp · at its level · small run
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items28.5% ✎▁▁▂▁▃▄█▆▄▅▂▃ beats 16/39very hard · avg 22%▇ 1.1%blunt
MMMLU lab76.1% ✎▂▁▁▂▃▂▄▄▅██▃ beats 23/46medium · avg 63%▇ 1.1%at its level
INCLUDE lab71.0% ✎▂····▂▂▄▄▄█▄ beats 17/36medium · avg 56%▇ 1.0%at its level · small run
EuroEval Life in the UK (English knowledge MC), validation split, 5-shot, accuracy86.1% ✎▂·▂▂·▂·▃▃·█▃ beats 4/14easy · avg 80%▇ 1.0%typical
Global PIQA lab78.9% ✎▃▃··▃··▃▃▅█▅ beats 3/11medium · avg 71%▇ 0.9%at its level
MAXIFE lab78.0% ✎▂·▂·▂···▆·▄█ beats 4/11medium · avg 63%▇ 0.8%typical
MultiChallenge lab49.0% ✎▁·▁·▁▃▆▅▃▁█▅ beats 13/29hard · avg 30%▇ 0.8%at its level
VitaBench lab22.0% ✎▄··▂▄▆▆█▄▄▄▂ beats 7/21very hard · avg 10%▇ 0.8%typical
MMLU-ProX lab71.5% ✎▁·▁▁▁▂▁▃▂█▅▄ beats 8/35medium · avg 56%▇ 0.8%at its level
EuroEval HellaSwag (English), validation split, 5-shot generative MC, accuracy76.9% ✎▂··▂▂·▂▆▆█·▄ beats 9/15medium · avg 63%▇ 0.8%at its level
GSM-Plus +185.2% ✎▁··▁▁▁·▃▁▄█▁ beats 15/16easy · avg 83%▇ 0.5%blunt · several runs
NOVA-63 lab54.3% ✎▃▃·▃···▃█▃▃█ beats 6/11medium · avg 38%▇ 0.5%blunt · at its level
VideoMMMU lab74.1% ✎▂···▂···▃▂█▆ beats 2/10medium · avg 61%▇ 0.5%at its level · small run
VLMs Are Blind lab92.6% ✎▂·▂·····▃··█ beats 2/8very easy · avg 86%▇ 0.5%maxed out · small run
Multi-IF (aggregation unspecified) +173.6% ✎▄▄···▄·▄·▄██ beats 6/8medium · avg 60%▇ 0.5%blunt · at its level · several runs
Video-MME (with subtitles) lab83.5% ✎▁▁▁▁▁·▁▃▁▃▁█ beats 9/16medium · avg 72%▇ 0.5%small run
SLAKE lab76.1% ✎▄·····▄▄·██▄ beats 2/7medium · avg 63%▇ 0.5%small run
LVBench lab66.4% ✎▇·█▅▅▄▄··▃▁▁ beats 17/28medium · avg 53%▇ 0.4%at its level · small run
LingoQA lab74.4% ✎▄······▄·█·█ beats 0/5medium · avg 59%▇ 0.4%at its level · small run
MMVU lab64.9% ✎▃···▃···▃▃▅█ beats 2/8medium · avg 49%▇ 0.4%at its level · small run
MLVU lab82.8% ✎▂···▂···▃▄▄█ beats 2/10easy · avg 71%▇ 0.4%small run
RefSpatial-Bench lab54.6% ✎▃·▃····▃▃▅▅█ beats 2/10medium · avg 40%▇ 0.4%at its level · small run
ERQA lab54.0% ✎▃··▃··█·▃█▅▅ beats 3/12medium · avg 37%▇ 0.4%at its level · small run
Claw-Eval average score62.3% ✎▄·▄··█▄▄▄▄▄▄ beats 2/9medium · avg 53%▇ 0.4%blunt · at its level
MMLongBench-Doc lab54.2% ✎▃····▃··▃▃▅█ beats 2/8medium · avg 38%▇ 0.4%at its level · small run
ZEROBench subquestions lab26.3% ✎▄·▄···▄·▄██▄ beats 2/8hard · avg 17%▇ 0.4%at its level · small run
EmbSpatial-Bench lab81.3% ✎▁·····▁·▃·▃█ beats 2/10easy · avg 70%▇ 0.4%small run
AndroidWorld lab58.6% ✎▁·▁▁▂▃█▅▂▂·▂ beats 14/24medium · avg 41%▇ 0.4%blunt · at its level · small run
Video-MME (without subtitles) lab76.9% ✎▁▃▁▃▁·█▄▄▅·▇ beats 18/27medium · avg 67%▇ 0.3%blunt · at its level · small run
DeepPlanning lab17.6% ✎▂·····▂··▄█▄ beats 2/9very hard · avg 7.1%▇ 0.2%noisy
MVBench lab71.2% ✎▁··▃▄▁▃▄▄▄█▄ beats 12/26medium · avg 66%▇ 0.2%blunt · at its level · small run
LIBRA Mini (Russian long context, May 2026 version)39.1% ✎▄···▄█·██··▄ beats 3/8medium · avg 38%▇ 0.1%blunt · at its level
WMT24++ lab66.6% ✎▁·▁▁·▁·▃▂▆█▃ beats 4/27easy · avg 68%▇ 0.1%blunt · at its level

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Qwen3.5 4B · needs ~2.7 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/Qwen3.5-4B-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/Qwen3.5-4B-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Qwen3.5 4B · needs ~2.7 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/Qwen3.5-4B-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Qwen3.5 4B · needs ~3.3 GB · from ollama.com
    chat in the terminal
    ollama run qwen3.5:4b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Qwen3.5 4B · needs ~3 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/Qwen3.5-4B-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/Qwen3.5-4B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3.5-9B model card 37%, openbmb/MiniCPM5-2B model card 20%, XHToken/Spark-X2.5-4B model c… 14% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Qwen/Qwen3.5-9B model card · lab-reported62▇▇▇▇▇▇▇▇▇▇▇ 37%apache-2.0
openbmb/MiniCPM5-2B model card19▇▇▇▇▇▇ 20%Apache-2.0
XHToken/Spark-X2.5-4B model card11▇▇▇▇ 14%Apache-2.0
JetBrains/Mellum2-12B-A2.5B-Instruct model card8▇▇ 7%Apache-2.0
Qwen/Qwen3.5-4B model card · lab-reported4▇▇ 6%apache-2.0
LiquidAI/LFM2.5-2.6B model card7▇ 4%LFM Open License v1.0
JetBrains/Mellum2-12B-A2.5B-Thinking model card7▇ 3%Apache-2.0
Epoch AI — internal evaluations1▇ 2%CC BY 4.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 2%MIT
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)1▇ 1%CC BY 4.0
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats (arXiv:2606.01348v3)1▇ 1%CC BY 4.0
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models (arXiv:2608.17895v1)1▇ 1%CC BY 4.0
LiquidAI/LFM2.5-8B-A1B model card1▇ 0%LFM Open License v1.0
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models (arXiv:2610.12402v1)1▇ 0%CC BY 4.0
ai-forever LIBRA leaderboard (Russian long-context benchmark)1▇ 0%MIT
How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings (arXiv:2605.07492v2)1▇ 0%CC BY 4.0
prior · pull toward the middle▇ 0%
weighted average= 60.5
$ quantization evidence · 40 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
AutoRound AutoScheme mixed-precision GGUF '2.5 bits mixed[q2,q3,q4,q6] tuning iters 200' (actual 3.9 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.75495.4%source ↗
AutoRound AutoScheme mixed-precision GGUF '2.5 bits mixed[q2,q3,q4,q6] tuning iters 200' (actual 3.9 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.38481.7%source ↗
AutoRound AutoScheme mixed-precision GGUF '2.5 bits mixed[q2,q3,q4,q6]' (actual 3.9 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.7291.1%source ↗
AutoRound AutoScheme mixed-precision GGUF '2.5 bits mixed[q2,q3,q4,q6]' (actual 3.9 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.42490.3%source ↗
AutoRound AutoScheme mixed-precision GGUF '3.5 bits mixed[q2,q3,q4,q6] tuning iters 200' (actual 4.86 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.78699.5%source ↗
AutoRound AutoScheme mixed-precision GGUF '3.5 bits mixed[q2,q3,q4,q6] tuning iters 200' (actual 4.86 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.41989.2%source ↗
AutoRound AutoScheme mixed-precision GGUF '3.5 bits mixed[q2,q3,q4,q6]' (actual 4.86 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.804101.8%source ↗
AutoRound AutoScheme mixed-precision GGUF '3.5 bits mixed[q2,q3,q4,q6]' (actual 4.86 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.41989.2%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.88 → 0.884100.5%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.942 → 0.945100.3%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.846 → 0.847100.1%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.85 → 0.85100.0%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.793 → 0.79199.7%source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.768 → 0.76399.3%source ↗
GGUF Q3_K_M (AutoRound-generated; actual 4.17 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.77297.7%source ↗
GGUF Q3_K_M (AutoRound-generated; actual 4.17 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.40486.1%source ↗
GGUF Q3_K_M (unsloth file as labelled; actual 4.34 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.77698.2%source ↗
GGUF Q3_K_M (unsloth file as labelled; actual 4.34 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.40486.0%source ↗
GGUF Q4_K_M (AutoRound-generated; actual 5.12 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.792100.3%source ↗
GGUF Q4_K_M (AutoRound-generated; actual 5.12 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.45596.8%source ↗
GGUF Q4_K_M (repeat 5) (AutoRound-generated; actual 5.12 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.796100.7%source ↗
GGUF Q4_K_M (repeat 5) (AutoRound-generated; actual 5.12 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.43292.0%source ↗
GGUF Q4_K_M (repeat 5) (unsloth file as labelled; actual 5.19 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.791100.1%source ↗
GGUF Q4_K_M (repeat 5) (unsloth file as labelled; actual 5.19 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.44294.2%source ↗
GGUF Q4_K_M (unsloth file as labelled; actual 5.19 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.794100.5%source ↗
GGUF Q4_K_M (unsloth file as labelled; actual 5.19 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.38481.7%source ↗
GGUF UD-IQ2_M (unsloth file as labelled; actual 3.33 bits/weight)MATH-500 (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.79 → 0.59475.2%source ↗
GGUF UD-IQ2_M (unsloth file as labelled; actual 3.33 bits/weight)GPQA Diamond (EvalScope via llama-server, --reasoning off, --predict 2048, ctx 65536, --parallel 16)0.47 → 0.22748.4%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.945 → 0.945100.0%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.845 → 0.83799.1%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.793 → 0.77597.7%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.883 → 0.86197.5%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.796 → 0.73191.8%source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.822 → 0.73389.2%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGSM8K-Platinum 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.945 → 0.94299.7%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMMLU-Pro 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.793 → 0.7999.6%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedIFEval prompt-level strict 0-shot (lm-evaluation-harness, vLLM; mean of 3 seeds)0.883 → 0.87799.3%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedMATH-500 0-shot (lighteval, vLLM; mean of 3 seeds)0.845 → 0.83999.3%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedAIME 2025 0-shot (lighteval, vLLM; mean of 8 seeds)0.822 → 0.81298.8%source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (GPTQModifier W8A8); lm_head, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantizedGPQA Diamond 0-shot (lighteval, vLLM; mean of 3 seeds)0.796 → 0.78598.6%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json