whatmodel.sh / Qwen3 32B

Qwen3 32B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Qwen3 32B is an open-weights language model from Alibaba, released April 2025. Its published size is 32.8B parameters and a 131k-token context window. Licence: apache-2.0.

57.6±0.4#223 of 719 LLMsfrom 103 benchmark results across 23 sources

$ how it compares

The LLMs ranked either side of Qwen3 32B, out of 828 with a score, then more from Alibaba.

#modelscorewhereRAMreleased
265Qwen3-VL 30B-A3B Instruct Alibaba58±1api $0.35/M +local~21GB2025-09
266Qwen3-VL 8B Thinking Alibaba58±1api $0.66/M +local~6.8GB2025-10
267Qwen3 32B Alibaba58±0api $0.13/M +local~20GB2025-04
268Qwen3 Coder 30B-A3B Instruct Alibaba58±1api $0.12/M +local~20GB2025-07
269Nova 2 Lite Amazon58±2api $0.94/M2025-10
more from Alibaba
18Qwen3.8 Max (2.4T-A95B) Alibaba78±1api $3.00/M +local~380GB2026-08
34Qwen3.8 Max (0902) Alibaba76±1api $3.00/M2026-09
37Qwen3.8-Flash-Next Alibaba75±1local~59GB2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #184 of 300 · 22% of its score · 10 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Mellum2 12B-A2.5B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
τ²-bench Telecom lab +426.3% ✎▃·█········█ beats 2/6very hard · avg 10%▇▇▇▇ 10%sharp · at its level · several runs
τ-bench Retail lab +252.8% ✎▁····▃·▃▇█▂▆ beats 5/28medium · avg 36%▇ 3.0%at its level · several runs
τ-bench Airline lab +229.0% ✎▁▁▁·▅▄▁█▇▁▁▇ beats 9/28hard · avg 25%▇ 1.9%at its level · small run · several runs
τ²-bench Retail lab +149.7% ✎▁····▂▃▃▆█▃▁ beats 5/44medium · avg 37%▇ 1.7%at its level · several runs
Terminal-Bench Hard2.8% ✎▂▅▆▂▂▆▂▂█▅▅▂ beats 3/27very hard · avg 1.4%▇ 1.3%sharp · too hard for it
τ²-bench Airline lab +145.5% ✎▁▁·▂▁▃▂▁█▅▂▃ beats 8/42hard · avg 32%▇ 1.2%at its level · several runs
BFCL v3 +269.3% ✎▁▁▁▁▁▃▅█▅▄▃▁ beats 108/114medium · avg 52%▇ 0.9%blunt · at its level · several runs
SWE-bench Verified23.4% ✎▁▁▁▁▁▃▂▄█▆▁▁ beats 14/131hard · avg 17%▇ 0.8%sharp · at its level · small run
2 more tests
testresultvs the others in its rundifficultycountswhy
BFCL V4 (December 2025 archive)48.7% ✎▁▁▁▂██▅▅▆▂▂▂ beats 56/77hard · avg 32%▇ 0.8%at its level
Multi-SWE-bench (scaffold per note)7.7% ✎█··█·······█ beats 0/2very hard · avg 5.7%▇ 0.4%small run
coding▁▁▁▃▆██▇▆▄▃▁ #189 of 458 · 13% of its score · 8 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B, Qwen3 30B-A3B Instruct 2507 here, though they rank higher overall. Trails QwQ-32B-Preview, DeepSeek-V2.5, OpenThinker 32B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Aider polyglot (225 exercises, two attempts) +140.0% ✎▂·▂▃▅▂▃█▆▄▂▂ beats 20/49very hard · avg 8.5%▇▇▇ 6.8%sharp · at its level · several runs
LiveCodeBench v6 (Feb–May 2025) +253.4% ✎▁▁▃▁█▅▄▇▅▅▃█ beats 14/38medium · avg 41%▇ 2.1%at its level · several runs
LiveCodeBench v6 (window unspecified)61.0% ✎█·▄·▄····▄·▄ beats 3/5medium · avg 42%▇ 1.2%at its level
WebApp1K-Duo-React pass@1 (onekq-ai)24.3% ✎▁··▃·▁▁▁▆█▇█ beats 2/45hard · avg 32%▇ 0.8%at its level
SWE-bench Verified23.4% ✎▁▁▁▁▁▃▂▄█▆▁▁ beats 14/131hard · avg 17%▇ 0.8%sharp · at its level · small run
MultiPL-E lab76.9% ✎▃··▄·▅▅▇▄█▃▄ beats 24/31easy · avg 68%▇ 0.7%typical
Multi-SWE-bench (scaffold per note)7.7% ✎█··█·······█ beats 0/2very hard · avg 5.7%▇ 0.4%small run
SciCode main problems9.2% ✎▄··▄····█▄·▄ beats 5/5very hard · avg 3.8%▇ 0.3%too hard for it
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #162 of 393 · 8.2% of its score · 5 tests

Beats MiniMax-M2, MiniCPM5 2B, Granite 4.2 30B here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, LFM2.5 8B-A1B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Arena-Hard v2 aggregate lab +148.4% ✎▁▁··▄▃▃▃█▁▁▇ beats 7/22hard · avg 28%▇ 2.6%sharp · at its level · several runs
Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge)44.5% ✎▂·▃▅▅·▃█▆▂▂▂ beats 12/22very hard · avg 12%▇ 2.2%sharp · at its level
LMArena (text)1346.9 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 188/379avg 1268▇ 1.6%blunt
IFEval (aggregation unspecified) lab +285.0% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 99/141easy · avg 77%▇ 1.4%several runs
Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges)53.3% ✎▆▂▂▆▂▂▃█▂▃·▂ beats 13/22hard · avg 19%▇ 0.4%sharp · at its level · noisy
math▁▂▃▅▆▅▄█▄▁▁▁ #169 of 398 · 18% of its score · 8 tests

Beats MiniMax-M2.1, Mistral Medium 3.5 (128B), GPT-4.1 here, though they rank higher overall. Trails OpenThinker3 7B, Olmo 3 7B Think, OpenThinker2 32B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AIME 2025 +371.0% ✎▄·█··▄····▄▄ beats 1/5medium · avg 53%▇▇ 5.1%sharp · at its level · several runs
HMMT 2025 (edition unspecified) +249.8% ✎▄··▄▄·▄····█ beats 2/5hard · avg 33%▇▇ 5.1%sharp · at its level · several runs
AIME 2024 +179.4% ✎█▄···█·····▄ beats 2/5medium · avg 62%▇ 3.7%sharp · several runs
OTIS mock AIME 2024–202566.9% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 96/193very hard · avg 8.5%▇ 1.4%sharp · at its level · noisy
PolyMATH lab +147.4% ✎▁···▃▁·▇█▄▄▄ beats 9/23hard · avg 32%▇ 1.2%at its level · several runs
MATH-50096.8% ✎▄·▄·····█··█ beats 2/5very easy · avg 92%▇ 0.9%maxed out
OMEGA (math generalisation)47.7% ✎▅···▃▃···▃▃█ beats 5/8hard · avg 24%▇ 0.5%at its level
MATH (subset unspecified)95.4% ✎▅····▃▃··▃▃█ beats 5/8easy · avg 74%▇ 0.2%noisy
other tests34 not in a task · 40% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond +265.7% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 97/208hard · avg 40%▇▇▇ 7.2%sharp · at its level · several runs
MMLU-Pro +373.9% ✎█··█··███··█ beats 3/5medium · avg 58%▇▇ 4.0%at its level · several runs
GPQA (subset unspecified) lab +168.4% ✎▂▁▂▂▄▃▆▃▅█▄▃ beats 73/104hard · avg 49%▇ 3.6%sharp · at its level · several runs
MMLU-Redux lab +190.9% ✎▁▁▃▁▂▃▄▅▇▅█▄ beats 40/63easy · avg 82%▇ 2.1%several runs
INCLUDE lab +173.7% ✎▂····▂▂▄▄▄█▄ beats 20/36medium · avg 58%▇ 2.1%at its level · several runs
SuperGPQA lab +254.1% ✎▂▁▂▂▅▆▃▇▇▇█▃ beats 25/46medium · avg 39%▇ 1.8%at its level · several runs
Humanity's Last Exam +18.3% ✎█··██···█·██ beats 3/5very hard · avg 3.8%▇ 1.8%too hard for it · several runs
MMLU-ProX lab +174.6% ✎▁·▁▁▁▂▁▃▃█▅▄ beats 12/35medium · avg 60%▇ 1.3%at its level · several runs
26 more tests
testresultvs the others in its rundifficultycountswhy
MMLU86.2% ✎▁▁▁▁▁▂▃▅█▃▆▃ beats 163/183medium · avg 72%▇ 1.2%typical
WritingBench lab +179.0% ✎▁·▁▃▁··▅█▂▂▁ beats 3/20easy · avg 70%▇ 1.1%several runs
LiveBench 2024-11-25 lab +174.9% ✎▄··▂·▂·▂█▆▂▂ beats 6/13medium · avg 60%▇ 1.0%at its level · several runs
KCL-Essay vanilla29.2% ✎▁···▁▃▂█▂▃▃▂ beats 7/23hard · avg 18%▇ 1.0%at its level
Multi-IF (aggregation unspecified) lab +173.0% ✎▁▁▁▁▁▁▃▄█▄▁▂ beats 16/41medium · avg 63%▇ 1.0%blunt · at its level · several runs
EQ-Bench 3 (emotional intelligence, judge Elo)1141.5 ✎▁·▂▁▄▅▅▃█▄▁▁ beats 18/62avg 1044▇ 1.0%typical
CPTUB average (SpeakLeash, 0-5 judge scale)74.2% ✎▁▂▂▂▃█▃▆▅▄▂▃ beats 67/95easy · avg 66%▇ 0.9%typical
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)35.2% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 34/66hard · avg 31%▇ 0.9%at its level
BRIDGE clinical text, zero-shot average (87 tasks)41.0% ✎▁▁▁▁▁▂▂▃▄▆█▃ beats 82/97hard · avg 32%▇ 0.9%blunt · at its level
MMMLU79.0% ✎▂▁▁▂▃▂▄▄▅██▃ beats 28/46medium · avg 64%▇ 0.8%at its level
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items27.4% ✎▁▁▂▁▃▄█▆▄▅▂▃ beats 14/39very hard · avg 22%▇ 0.8%blunt
Creative Writing v3 lab78.3% ✎▂····▂▆··▆█▆ beats 2/14easy · avg 69%▇ 0.8%typical
OJBench lab24.1% ✎▁···▁▃·▅▄▅██ beats 11/26very hard · avg 13%▇ 0.7%typical
Open Persian LLM Leaderboard v2 average (21 tasks)62.8% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 58/81medium · avg 58%▇ 0.6%blunt · at its level
KOR-Bench65.4% ✎▄·······█··▄ beats 0/3medium · avg 55%▇ 0.5%at its level
KCL-MCQA vanilla35.0% ✎▂···█▂▆▆██▆▂ beats 6/23very hard · avg 30%▇ 0.5%typical
LongSocialBench (full-context, 4-option MCQ over discussion threads)41.9% ✎▆▂·▄█·▂▂·▄▄▂ beats 8/16very hard · avg 36%▇ 0.4%blunt
SimpleQA8.6% ✎▄▄▄▆█▆███·▂█ beats 9/32very hard · avg 6.4%▇ 0.4%typical
RULER 128k77.5% ✎▃··▅▄▇▅█▇·▃▃ beats 22/32easy · avg 72%▇ 0.4%typical
IFBench (aggregation unspecified)35.4% ✎███···██···█ beats 2/5hard · avg 35%▇ 0.3%at its level
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy84.0% ✎▃·▁▁▂▁▂▅█▇▄▁ beats 81/100medium · avg 68%▇ 0.2%at its level · noisy
Chess Puzzles (Epoch)5.0% ✎█▁▁▁▁▂▁▄▂▂▁▁ beats 34/135very hard · avg 5.9%▇ 0.2%sharp · too hard for it · noisy
BeyondAIME29.0% ✎▄·▃·▂▄··▄▄▄█ beats 2/15medium · avg 36%▇ 0.2%at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy94.1% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 134/146easy · avg 83%▇ 0.1%noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)97.2% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 108/140very easy · avg 94%▇ 0.1%blunt · maxed out · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)94.1% ✎▁▂▁▂▅█▂▃▄▃▁▁ beats 78/98very easy · avg 90%▇ 0.0%blunt

$ where to run it

Through an API

api · Deep Infra · $0.13/M · $0.08 in / $0.28 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Qwen3 32B · needs ~19.8 GB · from huggingface.co/Qwen (official)
    chat in the terminal
    llama cli -hf Qwen/Qwen3-32B-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf Qwen/Qwen3-32B-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Qwen3 32B · needs ~19.8 GB · from huggingface.co/Qwen (official)

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/Qwen/Qwen3-32B-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Qwen3 32B · needs ~20.18 GB · from ollama.com
    chat in the terminal
    ollama run qwen3:32b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Qwen3 32B · needs ~17.4 GB, Mac only · from huggingface.co/Qwen (official)
    chat in the terminal
    mlx_lm.chat --model Qwen/Qwen3-32B-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model Qwen/Qwen3-32B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3-Next-80B-A3B-Think… 21%, Qwen/Qwen3-Next-80B-A3B-Instr… 20%, Falcon-H1R: Pushing the Reaso… 16% and 20 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Qwen/Qwen3-Next-80B-A3B-Thinking model card · lab-reported22▇▇▇▇▇▇ 21%apache-2.0
Qwen/Qwen3-Next-80B-A3B-Instruct model card · lab-reported24▇▇▇▇▇▇ 20%apache-2.0
Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling12▇▇▇▇▇ 16%CC BY 4.0
ByteDance-Seed/Seed-OSS-36B-Instruct model card18▇▇▇▇ 14%apache-2.0
Epoch AI — internal evaluations3▇ 4%CC BY 4.0
Aider polyglot benchmark runs1▇ 4%Apache-2.0
MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use (arXiv:2508.18669v1)1▇ 3%CC-BY-4.0
Arena-Hard-Auto v2.0-Preview leaderboard (repository README)2▇ 3%Apache-2.0
Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · lab-reported1▇ 2%Factual vendor-reported score (not licensed content)
LMArena leaderboard dataset1▇ 2%CC BY 4.0
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities2▇ 1%CC BY 4.0
Berkeley Function Calling Leaderboard result archive2▇ 1%Apache-2.0
EQ-Bench 3 canonical leaderboard results1▇ 1%MIT
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 1%MIT
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)1▇ 1%Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 1%MIT
WebApp1K models leaderboard (onekq-ai)1▇ 1%MIT
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)1▇ 1%CC BY 4.0
allenai/Olmo-3.1-32B-Think model card2▇ 1%Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 1%Apache-2.0
LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? (arXiv:2610.04118v1)1▇ 0%CC BY 4.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 0%MIT
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)2▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 57.6
$ quantization evidence · 55 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.75894.7%source ↗
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.59278.7%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.899.8%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.74699.3%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.80199.9%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.74799.4%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)GPQA (subset not stated in card), thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.684 → 0.69100.9%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)LiveBench 2024-11-25, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.598 → 0.598100.0%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)MMLU-Redux, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.909 → 0.90899.9%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)MMLU-Redux, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.857 → 0.85699.9%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)LiveBench 2024-11-25, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.749 → 0.73197.6%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)AIME 2024, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.814 → 0.79497.5%source ↗
AWQ INT4 (w4, group_size 128, zero_point, GEMM; checkpoint Qwen/Qwen3-32B-AWQ)GPQA (subset not stated in card), non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.546 → 0.53197.3%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.79999.8%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.74398.8%source ↗
FP8 block (weights FP8 128x128 blocks, activations FP8 dynamic per-group 128; llm-compressor QuantizationModifier)GSM8K — published row 'GSM8K (Strict-Match, 5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, add_bos_token)0.742 → 0.745100.5%source ↗
FP8 block (weights FP8 128x128 blocks, activations FP8 dynamic per-group 128; llm-compressor QuantizationModifier)MMLU — published row 'MMLU (Acc, 5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, add_bos_token)0.82 → 0.82100.0%source ↗
FP8 block (weights FP8 128x128 blocks, activations FP8 dynamic per-group 128; llm-compressor QuantizationModifier)OpenLLM v1 average of 6 tasks (ARC-Challenge (Acc-Norm, 25-shot); GSM8K (Strict-Match, 5-shot); HellaSwag (Acc-Norm, 10-shot); MMLU (Acc, 5-shot); TruthfulQA (MC2, 0-shot); Winogrande (Acc, 5-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, add_bos_token)0.747 → 0.74599.7%source ↗
FP8 block (weights FP8 128x128 blocks, activations FP8 dynamic per-group 128; llm-compressor QuantizationModifier)OpenLLM v2 average of 6 tasks (IFEval (Inst Level Strict Acc, 0-shot); BBH (Acc-Norm, 3-shot); Math-Hard (Exact-Match, 4-shot); GPQA (Acc-Norm, 0-shot); MUSR (Acc-Norm, 0-shot); MMLU-Pro (Acc, 5-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.305 → 0.29697.1%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.667 → 0.687103.0%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)LiveCodeBench — published row 'LiveCodeBench' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.625 → 0.633101.4%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)OpenLLM v2 average of 6 tasks (MMLU-Pro (5-shot); IFEval (0-shot); BBH (3-shot); Math-lvl-5 (4-shot); GPQA (0-shot); MuSR (0-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.421 → 0.427101.4%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)GSM8K — published row 'GSM-8K (5-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.876 → 0.883100.8%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.729 → 0.729100.1%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)AIME 2024 — published row 'AIME 2024' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.794 → 0.794100.0%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.81 → 0.80999.9%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)AIME 2025 — published row 'AIME 2025' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.718 → 0.70498.1%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.76695.5%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.69993.0%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.801 → 0.899.8%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.751 → 0.7599.8%source ↗
GPTQ-style W2A16 (weight-only 2-bit; group size not stated)GSM8K accuracy, thinking mode, 32K reasoning budget, temperature 0.60.958 → 0.93397.4%source ↗
GPTQ-style W2A16 (weight-only 2-bit; group size not stated)MATH-500 accuracy, thinking mode, 32K reasoning budget, temperature 0.60.93 → 0.6569.9%source ↗
GPTQ-style W2A16 (weight-only 2-bit; group size not stated)GPQA-Diamond accuracy, thinking mode, 32K reasoning budget, temperature 0.60.662 → 0.42964.9%source ↗
GPTQ-style W2A16 (weight-only 2-bit; group size not stated)AIME 2026 accuracy, thinking mode, 32K reasoning budget, temperature 0.60.7 → 0.16723.8%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.667 → 0.667100.0%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.729 → 0.72599.5%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.81 → 0.80499.3%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)GSM8K — published row 'GSM-8K (5-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.876 → 0.8698.1%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)OpenLLM v2 average of 6 tasks (MMLU-Pro (5-shot); IFEval (0-shot); BBH (3-shot); Math-lvl-5 (4-shot); GPQA (0-shot); MuSR (0-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.421 → 0.41297.9%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)AIME 2024 — published row 'AIME 2024' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.794 → 0.77397.4%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)LiveCodeBench — published row 'LiveCodeBench' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.625 → 0.59695.5%source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)AIME 2025 — published row 'AIME 2025' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.718 → 0.64389.5%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)GPQA Diamond — published row 'GPQA (Diamond, 0-shot)' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768, seed 420.629 → 0.645102.4%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)GSM8K — published row 'gsm8k' (OpenLLM v1 section); lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.742 → 0.747100.7%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)OpenLLM v1 average of 6 tasks (arc_challenge; gsm8k; hellaswag; mmlu; truthfulqa_mc2; winogrande) as published; lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.743 → 0.74299.8%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)MMLU — published row 'mmlu' (OpenLLM v1 section); lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.818 → 0.81299.3%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)OpenLLM v2 average of 6 tasks (BBH (3-shot); MMLU-Pro (5-shot); MuSR (0-shot); IFEval (0-shot); GPQA (0-shot); Math-|v|-5 (4-shot)) as published; lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.557 → 0.54497.8%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)Reasoning average of 3 tasks (AIME24 (0-shot); AIME25 (0-shot); GPQA (Diamond, 0-shot)) as published; lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768, seed 420.704 → 0.66394.2%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)AIME 2024 — published row 'AIME24 (0-shot)' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768, seed 420.759 → 0.6990.9%source ↗
NVFP4 W4A4 (FP4 weights and activations, group 16; llm-compressor QuantizationModifier)AIME 2025 — published row 'AIME25 (0-shot)' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768, seed 420.724 → 0.65590.5%source ↗
NVFP4A16 (FP4 weights, group 16, 16-bit activations; llm-compressor QuantizationModifier)GSM8K — published row 'GSM8K (8-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.873 → 0.877100.4%source ↗
NVFP4A16 (FP4 weights, group 16, 16-bit activations; llm-compressor QuantizationModifier)OpenLLM v1 average of 6 tasks (MMLU; ARC Challenge (0-shot); GSM8K (8-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.727 → 0.727100.0%source ↗
NVFP4A16 (FP4 weights, group 16, 16-bit activations; llm-compressor QuantizationModifier)MMLU — published row 'MMLU' (OpenLLM v1 section); lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn)0.809 → 0.80699.5%source ↗
NVFP4A16 (FP4 weights, group 16, 16-bit activations; llm-compressor QuantizationModifier)OpenLLM v2 average of 6 tasks (MMLU-Pro (5-shot); IFEval (0-shot); BBH (3-shot); Math-|v|-5 (4-shot); GPQA (0-shot); MuSR (0-shot)) as published; lm-evaluation-harness (neuralmagic fork) + vLLM (backend vllm, max_model_len 4096, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.554 → 0.54999.1%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json