whatmodel.sh / Qwen3-Next 80B-A3B Instruct

Qwen3-Next 80B-A3B Instruct

+ contribute

Qwen3-Next 80B-A3B Instruct is an open-weights language model from Alibaba, released September 2025. Its published size is 80B parameters (3B active per token) and a 131k-token context window. Licence: apache-2.0.

63.1±0.8#165 of 719 LLMsfrom 31 benchmark results across 6 sources

$ how it compares

The LLMs ranked either side of Qwen3-Next 80B-A3B Instruct, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
189	DeepSeek R1 (Jan '25) DeepSeek	63±1	api $4.00/M +local	~110GB	2025-01
190	Gemini 2.5 Flash Preview (09-2025) Google	63±1	comparison only		2025-09
191	Qwen3-Next 80B-A3B Instruct Alibaba	63±1	api $0.34/M +local	~27GB	2025-09
192	Granite 4.2 30B IBM	63±1	api $0.28/M +local	~19GB	2026-08
193	o1-preview OpenAI	63±1	comparison only		2024-09
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #191 of 300 · 16% of its score · 6 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Mellum2 12B-A2.5B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
τ²-bench Telecom lab	13.2% ✎	▁·▅█▆▅▂▃▃▂▃▁ beats 3/89	very hard · avg 6.0%	▇▇ 4.4%	sharp · at its level
τ-bench Retail lab	60.9% ✎	▁····▃·▂▇█▂▆ beats 11/28	hard · avg 32%	▇ 3.6%	at its level
τ²-bench Retail lab	57.3% ✎	▁····▂▁▃▆█▃▁ beats 9/44	medium · avg 36%	▇ 2.7%	at its level
τ²-bench Airline lab	45.5% ✎	▁▁·▂▁▃▂▁█▅▂▃ beats 9/42	hard · avg 23%	▇ 2.2%	at its level
τ-bench Airline lab	44.0% ✎	▁▁▁·▅▄▁█▇▁▁▇ beats 12/28	hard · avg 20%	▇ 2.2%	at its level
BFCL v3 lab	70.3% ✎	▁▁▁▁▁▃▅█▆▅▄▁ beats 150/170	medium · avg 57%	▇ 1.0%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #168 of 458 · 17% of its score · 3 tests

Beats Qwen3-Coder-Next (80B-A3B), Qwen3-VL 235B-A22B Instruct here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 8B Reasoning 2512, Phi-4-reasoning 14B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) lab	49.8% ✎	▂▁▁▃▆▃▄▆█▃▄▂ beats 41/71	very hard · avg 9.0%	▇▇▇▇▇ 12%	sharp · at its level
LiveCodeBench v6 (Feb–May 2025) lab	56.6% ✎	▁▁▃▁█▅▄▇▅▅▃█ beats 11/38	hard · avg 28%	▇ 2.6%	at its level
MultiPL-E lab	87.8% ✎	▃··▄·▅▅▇▄█▃▄ beats 30/31	easy · avg 68%	▇ 2.2%	typical

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #111 of 393 · 10% of its score · 3 tests

Beats LongCat 2.0 (1.6T-A48B), Step 3.5 Flash (196B-A11B), GPT-5 mini here, though they rank higher overall. Trails Nemotron-Cascade-8B, Gemini 2.5 Flash, Mistral Large 3 (675B-A41B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1399.0 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 243/379	avg 1268	▇▇ 5.9%	blunt
Arena-Hard v2 aggregate lab	82.7% ✎	▁▁··▄▃▁▃█▁▁▇ beats 21/22	hard · avg 32%	▇ 2.6%	sharp · at its level
IFEval (aggregation unspecified) lab	87.6% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 89/141	easy · avg 77%	▇ 1.7%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #99 of 398 · 8.4% of its score · 3 tests

Beats Claude Sonnet 4.6, MiniMax-M3, Nemotron 3 Ultra 550B-A55B here, though they rank higher overall. Trails Gemini 2.5 Flash, GLM-4.5-Air, Granite 4.2 30B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HMMT 2025 (edition unspecified) lab	54.1% ✎	▃█▇▅▇▅▁·▁··▁ beats 10/32	hard · avg 15%	▇▇ 3.8%	sharp · at its level
AIME 2025 lab	69.5% ✎	▁·▁▁▁▂▅▆█▇▅▁ beats 63/133	hard · avg 28%	▇ 2.7%	sharp · at its level
PolyMATH lab	45.9% ✎	▁···▃▁·▇█▄▄▄ beats 6/23	hard · avg 25%	▇ 1.9%	at its level

other tests16 not in a task · 48% of its score

test	result	vs the others in its run	difficulty	counts	why
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)	0.5687 ✎	▁▄▃▃▇▄█▃▅▁▁▁ beats 25/30	avg 0.423	▇▇▇▇ 9.0%	sharp
GPQA (subset unspecified) lab	72.9% ✎	▂▁▂▂▄▃▆▃▅█▄▃ beats 80/104	hard · avg 45%	▇▇ 5.6%	sharp · at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)	56.9% ✎	▂▃▅▂▆▃█▅█▅▃▆ beats 26/34	medium · avg 42%	▇▇ 4.4%	blunt · at its level
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items	30.1% ✎	▁▁▂▁▃▄█▆▄▅▂▃ beats 21/39	very hard · avg 22%	▇ 3.6%	blunt
MMLU-Pro lab	80.6% ✎	▁▁·▁▁▁▃▂▄▅█▁ beats 129/188	medium · avg 58%	▇ 3.6%	typical
INCLUDE lab	78.9% ✎	▂····▂▂▄▄▄█▄ beats 25/36	medium · avg 58%	▇ 3.4%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	39.8% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 71/97	hard · avg 32%	▇ 3.3%	blunt · at its level
MMLU-Redux lab	90.9% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 41/63	easy · avg 79%	▇ 3.0%	typical

8 more tests

test	result	vs the others in its run	difficulty	counts	why
Creative Writing v3 lab	85.3% ✎	▂····▂▆··▆█▆ beats 8/14	easy · avg 69%	▇ 2.3%	typical
MMLU-ProX lab	76.7% ✎	▁·▁▁▁▂▁▃▃█▅▄ beats 14/35	medium · avg 58%	▇ 2.1%	typical
SuperGPQA lab	58.8% ✎	▂▁▂▂▅▆▃▇▇▇█▃ beats 32/46	medium · avg 36%	▇ 2.1%	at its level
LiveBench 2024-11-25 lab	75.8% ✎	▄··▂·▂·▂█▆▂▂ beats 8/13	medium · avg 55%	▇ 1.7%	at its level
WritingBench lab	87.3% ✎	▁·▁▂▁··▅█▂▃▁ beats 19/20	easy · avg 70%	▇ 1.6%	typical
Multi-IF (aggregation unspecified) lab	75.8% ✎	▁▁▁▁▁▁▃▄█▄▁▂ beats 19/41	medium · avg 60%	▇ 1.5%	blunt
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	90.5% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 93/100	medium · avg 68%	▇ 0.6%	noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	94.4% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 138/146	easy · avg 83%	▇ 0.3%	noisy

$ where to run it

Through an API

api · Deep Infra · $0.34/M · $0.09 in / $1.1 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3-Next 80B-A3B Instruct · needs ~24.5 GB · from huggingface.co/unsloth
chat in the terminal
llama cli -hf unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF:UD-Q2_K_XL
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF:UD-Q2_K_XL

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3-Next 80B-A3B Instruct · needs ~44.8 GB, Mac only · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Qwen3-Next-80B-A3B-Instruct-MLX-4bit -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Qwen3-Next 80B-A3B Instruct · needs ~48.5 GB · from ollama.com
chat in the terminal
ollama run qwen3-next:80b-a3b-instruct-q4_K_M

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3-Next 80B-A3B Instruct · needs ~44.8 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3-Next-80B-A3B-Instruct-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3-Next-80B-A3B-Instruct-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3-Next-80B-A3B-Instr… 73%, Medmarks: A Comprehensive Ope… 13%, LMArena leaderboard dataset 6% and 3 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Qwen/Qwen3-Next-80B-A3B-Instruct model card · lab-reported	24	▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 73%	apache-2.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models	2	▇▇▇▇ 13%	CC BY 4.0
LMArena leaderboard dataset	1	▇▇ 6%	CC BY 4.0
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)	1	▇ 4%	CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 3%	MIT
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
prior · pull toward the middle		▇ 0%
weighted average		= 63.1

$ published speed measurements · 8

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	UD-Q4_K_XL	62.09 tok/s	llama.cpp c841aee (build_number 1 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r20 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 3 (2026-08-30, measured-local): Current-stack sentinel; 20 repeats; kernel 7.0.0-30, Mesa 26.1.7, desktop performance profile, DPM auto and recorded CPU-only background workload. Not a strict-clean speed headline or a same-condition regression comparison with older kernels. Artifact SHA-256 and exact effective flags are preserved in the raw bundle.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	UD-Q4_K_XL	59.06 tok/s	llama.cpp 1348f67c5 (build_number 9172 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=1024; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r20 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 59 (2026-05-16, measured-local): Latest-stack r20 with b2048 ub1024; b9172 improved the 80B MoE path versus the older 54.92 t/s b8933 row.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	UD-Q4_K_XL	49.57 tok/s	llama.cpp e77056f (build_number 1 as recorded), ROCm/HIP · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=1024; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r3 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics. Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 60 (2026-05-16, measured-local): ROCm spotcheck r3; HIP won pp512 but Vulkan b9172 won tg128 on the same model.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	UD-Q4_K_XL	54.92 tok/s	llama.cpp b8933, Vulkan (RADV) · llama-bench pp512/tg128 as tabulated (tg_tps column = tg128 generation); exact flags not retained in a raw CSV for this historical row Historical row sourced by the guide to its own BENCHMARKS.md/README.md; no raw CSV retained. Row date 2026-04-26, status measured-local, kernel 6.19.4, driver stack 26.0.2-26.0.6. Author note: 80B MoE with 256K context capability Instruct checkpoint inferred from the same guide's raw Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf files; this historical row has no raw CSV.	source ↗
NVIDIA GeForce RTX 5090 (32GB; host CPU/RAM not stated)	GGUF Q2_K_XL (Unsloth repository)	76.1 tok/s	llama.cpp (GGUF; version not stated) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase Model fully resident in GPU/unified memory as published.	source ↗
NVIDIA GeForce RTX 5090 (32GB; host CPU/RAM not stated)	GGUF Q4_K_M (Unsloth repository) · layers offloaded	4.7 tok/s	llama.cpp (GGUF; version not stated) · Offload column: 28% (CPU); prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase Partial CPU offload as published (share of model on CPU/system RAM); host CPU/RAM not stated.	source ↗
Apple M4 Pro (14-core CPU / 20-core GPU, 64GB)	MLX 4-bit	52.3 tok/s	MLX v0.30.6 (macOS 26 Tahoe) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase Model fully resident in GPU/unified memory as published.	source ↗
Apple M3 Ultra (28-core CPU / 60-core GPU, 96GB)	MLX 4-bit	49.1 tok/s	MLX v0.30.6 (macOS 26 Tahoe) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase Model fully resident in GPU/unified memory as published.	source ↗

$ quantization evidence · 24 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	GSM8K 5-shot strict-match (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.31 → 0.32	103.2%	source ↗
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v2 average (6 tasks: IFEval inst-level strict, BBH 3-shot, Math-Hard 4-shot, GPQA 0-shot, MuSR 0-shot, MMLU-Pro 5-shot; lm-evaluation-harness leaderboard, vLLM)	0.57 → 0.575	100.8%	source ↗
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v1 average (6 tasks: ARC-Challenge 25-shot acc_norm, GSM8K 5-shot strict-match, HellaSwag 10-shot acc_norm, MMLU 5-shot, TruthfulQA mc2 0-shot, Winogrande 5-shot; lm-evaluation-harness, vLLM)	0.691 → 0.694	100.4%	source ↗
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	IFEval inst-level strict acc 0-shot (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.904 → 0.905	100.1%	source ↗
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU 5-shot acc (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.856 → 0.856	100.0%	source ↗
FP8 block (E4M3; weights static 128x128 blocks, activations dynamic per-group 128; llm-compressor FP8_BLOCK, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU-Pro 5-shot acc (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.597 → 0.597	99.9%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	GSM8K 5-shot strict-match (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.31 → 0.332	107.1%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v1 average (6 tasks: ARC-Challenge 25-shot acc_norm, GSM8K 5-shot strict-match, HellaSwag 10-shot acc_norm, MMLU 5-shot, TruthfulQA mc2 0-shot, Winogrande 5-shot; lm-evaluation-harness, vLLM)	0.691 → 0.696	100.7%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU-Pro 5-shot acc (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.597 → 0.6	100.5%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v2 average (6 tasks: IFEval inst-level strict, BBH 3-shot, Math-Hard 4-shot, GPQA 0-shot, MuSR 0-shot, MMLU-Pro 5-shot; lm-evaluation-harness leaderboard, vLLM)	0.57 → 0.573	100.4%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	IFEval inst-level strict acc 0-shot (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.904 → 0.903	99.9%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU 5-shot acc (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.856 → 0.854	99.8%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	IFEval inst-level strict acc 0-shot (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.775 → 0.807	104.2%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	GSM8K 5-shot strict-match (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.816 → 0.822	100.7%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v2 average (6 tasks: IFEval inst-level strict, BBH 3-shot, Math-Hard 4-shot, GPQA 0-shot, MuSR 0-shot, MMLU-Pro 5-shot; lm-evaluation-harness leaderboard, vLLM)	0.577 → 0.579	100.5%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	OpenLLM v1 average (6 tasks: ARC-Challenge 25-shot acc_norm, GSM8K 5-shot strict-match, HellaSwag 10-shot acc_norm, MMLU 5-shot, TruthfulQA mc2 0-shot, Winogrande 5-shot; lm-evaluation-harness, vLLM)	0.739 → 0.738	99.8%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU 5-shot acc (OpenLLM v1 configuration; lm-evaluation-harness, vLLM)	0.856 → 0.85	99.4%	source ↗
INT4 W4A16 (GPTQ, symmetric, group 128; llm-compressor, compressed-tensors); lm_head, MoE router/shared-expert gates and linear-attention (Gated DeltaNet) layers unquantized	MMLU-Pro 5-shot acc (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.763 → 0.755	98.9%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	AIME 2025 0-shot (lighteval aime25, litellm endpoint, vLLM; mean of 6 seeded repetitions)	0.628 → 0.65	103.5%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	MATH-500 0-shot (lighteval math_500; mean of 6 seeded repetitions)	0.897 → 0.906	100.9%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	GPQA Diamond 0-shot (lighteval gpqa:diamond; mean of 6 seeded repetitions)	0.746 → 0.752	100.8%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	MMLU-Pro 5-shot (lm-evaluation-harness mmlu_pro_chat, chat completions; mean of 6 seeded repetitions)	0.786 → 0.788	100.3%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	IFEval (lm-evaluation-harness ifeval, chat completions; metric variant not stated; mean of 6 seeded repetitions)	0.914 → 0.915	100.1%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; llm-compressor, compressed-tensors) (QuantizationModifier W8A8, Open-Platypus calibration); lm_head, MoE router/shared-expert gates and linear-attention layers unquantized	GSM8K 5-shot (lm-evaluation-harness gsm8k, chat completions, fewshot as multiturn; mean of 6 seeded repetitions)	0.697 → 0.697	100.0%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json