whatmodel.sh / Gemma 4 26B-A4B

Gemma 4 26B-A4B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Gemma 4 26B-A4B is an open-weights language model from Google, released April 2026. Its published size is 25.2B parameters (3.8B active per token) and a 262k-token context window. Licence: apache-2.0.

63.9±0.6#158 of 719 LLMsfrom 62 benchmark results across 18 sources

$ how it compares

The LLMs ranked either side of Gemma 4 26B-A4B, out of 828 with a score, then more from Google.

#modelscorewhereRAMreleased
179Qwen3-Coder 480B-A35B Instruct Alibaba64±1api $0.67/M +local~78GB2025-07
180Mistral Medium 3.1 (2508) Mistral64±2comparison only2025-08
181Gemma 4 26B-A4B Google64±1api $0.14/M +local~18GB2026-04
182Kimi K2 Instruct (0905) Moonshot AI64±2local~298GB2025-09
183Qwen3-VL 235B-A22B Instruct Alibaba64±1local~134GB2025-09
more from Google
9Gemini 4 Argon (limited preview) Google82±1limited preview2026-09
23Gemini 3.8 Flash Google77±1api $1.50/M2026-09
32Gemini 3.7 Flash Google76±1api $1.50/M2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #153 of 300 · 26% of its score · 13 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
SWE-bench Verified35.8% ✎▄▄·····▄▄▄▄█ beats 1/7hard · avg 17%▇ 3.5%sharp · at its level
Terminal-Bench 2.127.3% ✎▄······█▄▄█▄ beats 2/7very hard · avg 7.1%▇ 3.3%at its level
Terminal-Bench Hard lab14.0% ✎██··█·█·█··█ beats 3/5very hard · avg 4.7%▇ 3.2%sharp
τ²-bench aggregate lab +168.2% ✎▄·▄··▄····█▄ beats 3/5hard · avg 32%▇ 2.9%at its level · several runs
ContractBench success rate (33 observation-contract tool tasks)38.4% ✎▆··▆▄▂▄▂▆▆█▄ beats 10/23very hard · avg 14%▇ 2.8%sharp · at its level
SWE-bench Pro (split unspecified)9.6% ✎▄···▄···▄▄██ beats 0/7very hard · avg 6.5%▇ 2.6%sharp · at its level
τ²-bench Telecom lab43.0% ✎▄····█··▄·▄▄ beats 3/5very hard · avg 14%▇ 2.3%sharp · at its level · noisy
SWE-bench Multilingual27.3% ✎▄·▄·····█▄█▄ beats 1/7hard · avg 18%▇ 1.7%at its level
5 more tests
testresultvs the others in its rundifficultycountswhy
Tau2 – airline lab76.0% ✎▃·▃··▃·····█ beats 5/5medium · avg 42%▇ 1.7%at its level
Humanity's Last Exam with tools lab17.2% ✎▃▂▂▁▄▂▃▆█▇▂▆ beats 1/54very hard · avg 11%▇ 0.8%at its level · small run
SWE-bench Pro public set, Qwen-refined13.8% ✎▃·····▂·▂█▃▅ beats 0/12hard · avg 15%▇ 0.8%at its level
Tau2 – retail lab85.5% ✎▄····▄··▄·▄█ beats 4/5medium · avg 40%▇ 0.5%sharp · noisy
T1-Bench multi-domain conversational agent pass^337.7% ✎▂▂··▂·▂·▂▂█▄ beats 10/11hard · avg 25%▇ 0.0%blunt · at its level · noisy
coding▁▁▁▃▆██▇▆▄▃▁ #120 of 458 · 21% of its score · 10 tests

Beats Qwen3.5 Plus, Gemini 2.5 Pro, Kimi K2 Thinking here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1 here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Arena WebDev (Code Arena)1358.5 ✎▁▂▄▃▅█▄▇▆▄▁▁ beats 32/119avg 981▇▇ 5.2%sharp
SWE-bench Verified35.8% ✎▄▄·····▄▄▄▄█ beats 1/7hard · avg 17%▇ 3.5%sharp · at its level
Luau Leetcode fully solved (Torpedo Software)38.9% ✎▁▁▁▁·▃▂▃▅▃█▇ beats 7/49hard · avg 18%▇ 2.6%at its level
SWE-bench Pro (split unspecified)9.6% ✎▄···▄···▄▄██ beats 0/7very hard · avg 6.5%▇ 2.6%sharp · at its level
SWE-bench Multilingual27.3% ✎▄·▄·····█▄█▄ beats 1/7hard · avg 18%▇ 1.7%at its level
SciCode lab40.0% ✎█▄·······▄▄▄ beats 4/5hard · avg 21%▇ 1.4%at its level
SciCode (level unspecified)30.8% ✎▄···▄▄█▄·▄·▄ beats 4/7very hard · avg 13%▇ 1.3%at its level
LiveCodeBench v6 (window unspecified) lab77.1% ✎▂▃▄▄▅██▅▄▇▃▆ beats 42/65medium · avg 47%▇ 1.3%small run
2 more tests
testresultvs the others in its rundifficultycountswhy
LiveCodeBench v6 (Feb–May 2025) lab77.1% ✎▄··▄·▄···▄·█ beats 4/5medium · avg 43%▇ 1.0%at its level
SWE-bench Pro public set, Qwen-refined13.8% ✎▃·····▂·▂█▃▅ beats 0/12hard · avg 15%▇ 0.8%at its level
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #69 of 393 * · 3.3% of its score · 2 tests

Beats GPT-5.2, LongCat 2.0 (1.6T-A48B), GPT-5 here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
LMArena (text)1437.5 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 295/379avg 1268▇ 2.9%blunt
IFEval lab98.5% ✎▄··▄··█···▄▄ beats 4/5very easy · avg 94%▇ 0.3%maxed out
math▁▂▃▅▆▅▄█▄▁▁▁ #98 of 398 · 7.9% of its score · 3 tests

Beats Claude Sonnet 4.6, MiniMax-M3, Nemotron 3 Ultra 550B-A55B here, though they rank higher overall. Trails Gemini 2.5 Flash, GLM-4.5-Air, Granite 4.2 30B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
OTIS mock AIME 2024–202582.2% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 118/193very hard · avg 8.5%▇ 2.8%sharp · at its level · noisy
AIME 2026 lab +188.3% ✎▄·▄▄····▄··█ beats 4/5medium · avg 51%▇ 2.7%sharp · small run · several runs
MATH-Vision lab +182.4% ✎██·█·····███ beats 4/5medium · avg 53%▇ 2.3%several runs
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #56 of 147 · 9.1% of its score · 5 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-5, Claude Opus 4.5 here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
MMMU Pro lab +173.8% ✎██·█····█·██ beats 4/5medium · avg 49%▇ 2.5%at its level · several runs
MATH-Vision lab +182.4% ✎██·█·····███ beats 4/5medium · avg 53%▇ 2.3%several runs
Arena vision (style controlled)1241.9 ✎▁▁▂▂▂▂▄▅▅▆██ beats 94/145avg 1068▇ 2.0%blunt
InfographicVQA lab89.3% ✎▄█·······█·▄ beats 4/5easy · avg 71%▇ 1.2%typical
OmniDocBench 1.5 ↓ lab +114.9% ✎█·█·····████ beats 4/5avg 31%▇ 1.1%blunt · several runs
web search▃▁▂▆▄█▃▃▁▁·▁ #77 of 80 * · 0.8% of its score · 1 test

Beats GLM-4.5, Claude Haiku 4.5 here, though they rank higher overall. Trails GPT-4.1 nano, GPT-4o mini (2024-07-18), Mistral Small 3.1 24B (2503) here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Humanity's Last Exam with tools lab17.2% ✎▃▂▂▁▄▂▃▆█▇▂▆ beats 1/54very hard · avg 11%▇ 0.8%at its level · small run
other tests23 not in a task · 43% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond +273.2% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 111/208hard · avg 40%▇▇▇▇ 11%sharp · at its level · several runs
MedXPertQA MM lab +158.1% ✎▂···▂·▃··▆▃█ beats 8/13very hard · avg 30%▇▇ 4.8%sharp · at its level · several runs
MRCR v2 8-needle, 128k lab +144.1% ✎▄▄·▄···█···▄ beats 4/5hard · avg 22%▇ 2.9%at its level · several runs
MMLU-Pro lab +182.6% ✎▄··█···▄··▄▄ beats 4/5medium · avg 61%▇ 2.8%several runs
MMMLU lab +186.3% ✎██··█···█·██ beats 4/5medium · avg 69%▇ 2.5%several runs
EQ-Bench 3 (emotional intelligence, judge Elo)1272.3 ✎▁·▂▁▄▅▅▃█▄▁▁ beats 40/62avg 1044▇ 1.8%typical
BRIDGE clinical text, zero-shot average (87 tasks)44.5% ✎▁▁▁▁▁▂▂▃▄▆█▃ beats 93/97hard · avg 32%▇ 1.7%blunt · at its level
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)67.3% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 53/66hard · avg 31%▇ 1.5%at its level
15 more tests
testresultvs the others in its rundifficultycountswhy
IFBench lab72.0% ✎▄▄·▄······▄█ beats 3/5medium · avg 40%▇ 1.4%at its level
Big Bench Extra Hard micro avg lab64.8% ✎█··▄···▄·▄·▄ beats 4/5hard · avg 28%▇ 1.3%at its level
CPTUB average (SpeakLeash, 0-5 judge scale)84.3% ✎▁▂▂▂▃█▃▆▅▄▃▃ beats 88/95easy · avg 66%▇ 1.3%typical
NL2Repo11.6% ✎▃▂·▂▂▄█▆▆▂▄▆ beats 0/21very hard · avg 6.6%▇ 1.2%typical
RobloxQA v2.0 multiple choice (Torpedo Software)90.9% ✎▁▁▃▁█▄▃▃▆▂▂▂ beats 19/49easy · avg 78%▇ 1.2%typical
QwenClawBench (Qwen internal)38.7% ✎▄·▄····▄···█ beats 0/4hard · avg 22%▇ 1.2%at its level
Humanity's Last Exam lab8.7% ✎▁▁▁▃▆▆▇██▆▁▃ beats 14/104very hard · avg 4.1%▇ 1.1%small run
EuroEval Life in the UK (English knowledge MC), validation split, 5-shot, accuracy93.4% ✎▂·▂▂·▂·▃▃·█▃ beats 12/14easy · avg 80%▇ 1.1%typical
Open Persian LLM Leaderboard v2 average (21 tasks)70.7% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 75/81medium · avg 58%▇ 1.0%blunt · at its level
EuroEval HellaSwag (English), validation split, 5-shot generative MC, accuracy82.3% ✎▂··▂▂·▂▆▆█·▄ beats 13/15medium · avg 63%▇ 1.0%typical
Chess Puzzles (Epoch)6.0% ✎█▃▁▁▁▂▁▄▂▂▁▁ beats 48/135very hard · avg 5.9%▇ 0.8%sharp · too hard for it · noisy
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)63.2% ✎▁··▃▁▁▁▅█▅▁▅ beats 14/24medium · avg 52%▇ 0.6%blunt · at its level
HLE lab8.7% ✎█···█······█ beats 1/2very hard · avg 3.9%▇ 0.6%small run
SkillsBench 78-task subset (OpenCode, avg of 5)12.3% ✎▄·····▄···▄█ beats 1/4very hard · avg 7.1%▇ 0.6%typical
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)94.8% ✎▁▂▁▂▅█▂▃▄▃▁▁ beats 87/98very easy · avg 90%▇ 0.0%blunt

$ where to run it

Through an API

api · DeepInfra · $0.14/M · $0.07 in / $0.34 out per 1M tokens · 262k context · deepinfra.com ↗ · price sourceUSD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-4-26B-A4B-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD.

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Gemma 4 26B-A4B · needs ~16.8 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/gemma-4-26B-A4B-it-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/gemma-4-26B-A4B-it-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Gemma 4 26B-A4B · needs ~16.8 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Gemma 4 26B-A4B · needs ~15.62 GB · from ollama.com
    chat in the terminal
    ollama run gemma4:26b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Gemma 4 26B-A4B · needs ~15.6 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/gemma-4-26B-A4B-it-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/gemma-4-26B-A4B-it-MLX-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Gemma Team — Gemma 4 Technica… 31%, google/diffusiongemma-26B-A4B… 15%, Kwaipilot/KAT-Coder-V2.5-Dev… 12% and 15 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Gemma Team — Gemma 4 Technical Report · lab-reported20▇▇▇▇▇▇▇▇▇ 31%CC BY 4.0
google/diffusiongemma-26B-A4B-it model card · lab-reported12▇▇▇▇ 15%Apache-2.0
Kwaipilot/KAT-Coder-V2.5-Dev model card5▇▇▇▇ 12%Apache-2.0
LMArena leaderboard dataset3▇▇▇ 10%CC BY 4.0
Epoch AI — internal evaluations3▇▇▇ 9%CC BY 4.0
Roblox LLM Leaderboard (Torpedo Software): RobloxQA v2.0 and Luau Leetcode2▇ 4%MIT
Qwen/Qwen3.6-35B-A3B model card4▇ 4%apache-2.0
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)1▇ 3%CC BY 4.0
google/gemma-4-31B-it model card · lab-reported2▇ 2%Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 2%MIT
EQ-Bench 3 canonical leaderboard results1▇ 2%MIT
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 2%MIT
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)1▇ 1%Apache-2.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 1%MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 1%Apache-2.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)1▇ 1%CC BY 4.0
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains (arXiv:2606.11070v1)1▇ 0%CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 63.9
$ published speed measurements · 3

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GBUD-Q4_K_M55.45 tok/sllama.cpp 0eca4d490 (build_number 9851 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r5 repetitions

Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 21 (2026-06-30, measured-local): Official llama.cpp b9851 Ubuntu Vulkan release binary with explicit -dev Vulkan0; direct Gemma 4 26B-A4B control. Not a speed headline; Gemma MTP server route remains the stronger Gemma throughput path.

source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GBUD-Q4_K_M54.18 tok/sllama.cpp 4fc4ec554 (build_number 9859 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r5 repetitions

Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 24 (2026-07-02, measured-local): Official llama.cpp b9859 Ubuntu Vulkan release binary with explicit -dev Vulkan0 and b9851-matching -fa on -mmp 0 command shape; direct Gemma 4 control remains secondary to the Gemma QAT/MTP server route.

source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GBGemma QAT GGUF UD-Q4_K_XL74.8 tok/sllama.cpp ac4cddeb0 (build_number 9592 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r5 repetitions

Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 32 (2026-06-11, measured-local): Current Gemma 4 26B-A4B QAT direct route; strong direct baseline for the matched QAT MTP server route, but direct llama-bench remains separate from speculative serving.

source ↗
$ quantization evidence · 24 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)AIME 2025 0-shot pass@1, mean of 8 seeds (with thinking)0.887 → 0.917103.3%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)GPQA Diamond 0-shot pass@1 (without thinking)0.732 → 0.748102.1%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)MATH-500 0-shot pass@1 (without thinking)0.848 → 0.859101.3%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)IFEval 0-shot prompt-level strict (with thinking)0.944 → 0.951100.7%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)MATH-500 0-shot pass@1 (with thinking)0.859 → 0.859100.1%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)AIME 2025 0-shot pass@1, mean of 8 seeds (without thinking)0.8 → 0.8100.0%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)MMLU-Pro 0-shot custom-extract (with thinking)0.852 → 0.85199.9%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)MMLU-Pro 0-shot custom-extract (without thinking)0.835 → 0.83399.7%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)IFEval 0-shot prompt-level strict (without thinking)0.9 → 0.89399.3%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)LiveCodeBench v6 0-shot pass@1 (without thinking)0.745 → 0.73999.2%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)GPQA Diamond 0-shot pass@1 (with thinking)0.808 → 0.80199.2%source ↗
FP8 dynamic (data-free; FP8 weights per-channel, activations dynamic per-token; llm-compressor model_free_ptq FP8_DYNAMIC)LiveCodeBench v6 0-shot pass@1 (with thinking)0.779 → 0.7697.6%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)MATH-500 0-shot pass@1 (without thinking)0.848 → 0.876103.3%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)IFEval 0-shot prompt-level strict (with thinking)0.944 → 0.94299.7%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)MATH-500 0-shot pass@1 (with thinking)0.859 → 0.85199.1%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)GPQA Diamond 0-shot pass@1 (without thinking)0.732 → 0.72498.9%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)MMLU-Pro 0-shot custom-extract (without thinking)0.835 → 0.81797.8%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)MMLU-Pro 0-shot custom-extract (with thinking)0.852 → 0.83397.8%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)IFEval 0-shot prompt-level strict (without thinking)0.9 → 0.87297.0%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)GPQA Diamond 0-shot pass@1 (with thinking)0.808 → 0.77696.0%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)LiveCodeBench v6 0-shot pass@1 (with thinking)0.779 → 0.72893.4%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)LiveCodeBench v6 0-shot pass@1 (without thinking)0.745 → 0.68291.6%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)AIME 2025 0-shot pass@1, mean of 8 seeds (with thinking)0.887 → 0.81391.5%source ↗
NVFP4 (FP4 weights/activations with per-group scaling; llm-compressor NVFP4)AIME 2025 0-shot pass@1, mean of 8 seeds (without thinking)0.8 → 0.66282.8%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json