whatmodel.sh / Qwen3 30B-A3B Instruct 2507

Qwen3 30B-A3B Instruct 2507

+ contribute

Qwen3 30B-A3B Instruct 2507 is an open-weights language model from Alibaba, released July 2025. Its published size is 30.5B parameters (3.3B active per token) and a 262k-token context window. Licence: apache-2.0.

60.2±0.6#193 of 719 LLMsfrom 60 benchmark results across 10 sources

$ how it compares

The LLMs ranked either side of Qwen3 30B-A3B Instruct 2507, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
229	GPT-4.1 mini OpenAI	60±1	api $0.70/M		2025-04
230	DeepSeek V3 0324 DeepSeek	60±1	local	~108GB	2025-03
231	Qwen3 30B-A3B Instruct 2507 Alibaba	60±1	api $0.15/M +local	~20GB	2025-07
232	Qwen3-VL 32B Instruct Alibaba	60±1	api $0.28/M +local	~22GB	2025-10
233	Falcon-H1R 7B TII	60±1	local	~6GB	2026-01
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #200 of 300 · 19% of its score · 7 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 (Jan '25) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Mellum2 12B-A2.5B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
τ²-bench Telecom lab +1	12.3% ✎	▁·▅█▆▅▂▃▃▂▃▁ beats 2/89	very hard · avg 8.7%	▇▇ 4.9%	sharp · at its level · several runs
τ-bench Retail lab +1	59.1% ✎	▁····▃·▂▇█▂▆ beats 8/28	hard · avg 32%	▇▇ 4.1%	at its level · several runs
τ²-bench Retail lab +1	57.0% ✎	▁····▂▁▃▆█▃▁ beats 8/44	medium · avg 36%	▇ 2.6%	at its level · several runs
τ-bench Airline lab +1	40.0% ✎	▁▁▁·▅▄▃█▇▁▁▇ beats 10/28	hard · avg 20%	▇ 2.4%	at its level · several runs
τ²-bench Airline lab +1	38.0% ✎	▁▁·▃▁▃▂▁█▅▂▃ beats 4/42	hard · avg 23%	▇ 2.3%	at its level · several runs
BFCL V4 (December 2025 archive)	41.4% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 47/77	hard · avg 32%	▇ 1.7%	at its level
BFCL v3 lab +1	65.1% ✎	▁▁▁▁▁▃▅█▆▅▄▁ beats 126/170	medium · avg 57%	▇ 1.1%	blunt · at its level · several runs

coding▁▁▁▃▆██▇▆▄▃▁ #197 of 458 · 17% of its score · 3 tests

Beats Gemini 2.5 Flash, Qwen3.5 4B here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, DeepSeek-Coder-V2 Instruct (0724) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) lab +1	35.6% ✎	▂▁▁▃▆▃▄▆█▃▄▂ beats 30/71	very hard · avg 9.0%	▇▇▇▇▇ 11%	sharp · at its level · several runs
LiveCodeBench v6 (Feb–May 2025) lab +1	43.2% ✎	▁▁▃▁█▅▄▇▅▅▃█ beats 2/38	hard · avg 28%	▇ 3.0%	at its level · several runs
MultiPL-E lab +1	83.8% ✎	▃··▄·▅▅▇▄█▃▄ beats 28/31	easy · avg 68%	▇ 2.5%	several runs

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #127 of 393 · 8.6% of its score · 3 tests

Beats LongCat 2.0 (1.6T-A48B), o3-mini, K-EXAONE (236B-A23B) here, though they rank higher overall. Trails LFM2.5 8B-A1B, LFM2.5 2.6B, Nemotron-Cascade-8B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1382.7 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 222/379	avg 1268	▇ 3.4%	blunt
Arena-Hard v2 aggregate lab +1	69.0% ✎	▁▁··▄▃▁▃█▃▁▇ beats 16/22	hard · avg 26%	▇ 3.3%	sharp · at its level · small run · several runs
IFEval (aggregation unspecified) lab +1	84.7% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 71/141	easy · avg 77%	▇ 1.8%	several runs

math▁▂▃▅▆▅▄█▄▁▁▁ #141 of 398 · 12% of its score · 4 tests

Beats Claude Sonnet 4.6, MiMo-V2.5 (310B-A15B), MiniMax-M2.7 here, though they rank higher overall. Trails Olmo 3 32B Think, Olmo 3.1 32B Think, gpt-oss-20b here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HMMT 2025 (edition unspecified) lab +1	43.0% ✎	▃█▇▅▇▅▁·▁··▁ beats 6/32	hard · avg 15%	▇▇ 3.8%	sharp · at its level · several runs
OTIS mock AIME 2024–2025	62.2% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 90/193	very hard · avg 8.5%	▇ 3.4%	sharp · at its level · noisy
AIME 2025 lab +1	61.3% ✎	▁·▁▁▁▂▅▆█▇▅▁ beats 50/133	hard · avg 28%	▇ 3.2%	sharp · at its level · several runs
PolyMATH lab +1	43.1% ✎	▁···▃▁·▇█▄▄▄ beats 5/23	hard · avg 25%	▇ 2.0%	at its level · several runs

other tests19 not in a task · 43% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA (subset unspecified) lab +1	70.4% ✎	▂▁▂▂▄▃▆▃▅█▄▃ beats 77/104	hard · avg 45%	▇▇▇ 6.3%	sharp · at its level · several runs
GPQA Diamond	55.6% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 78/208	hard · avg 40%	▇▇ 6.2%	sharp · at its level
MMLU-Pro lab +1	78.4% ✎	▁▁·▁▁▁▃▂▄▅█▁ beats 118/188	medium · avg 58%	▇▇ 4.0%	at its level · several runs
INCLUDE lab +1	71.9% ✎	▂····▂▂▄▄▄█▄ beats 18/36	medium · avg 58%	▇ 3.7%	at its level · several runs
MMLU-Redux lab +1	89.3% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 33/63	easy · avg 79%	▇ 3.5%	several runs
Medmarks-V weighted mean win rate (61 models, 71 configurations)	53.2% ✎	▂▃▅▂▆▃█▅█▅▃▆ beats 20/34	medium · avg 42%	▇ 2.6%	blunt · at its level
Creative Writing v3 lab +1	86.0% ✎	▂····▂▆··▆█▆ beats 10/14	easy · avg 69%	▇ 2.4%	several runs
MMLU-ProX lab +1	72.0% ✎	▁·▁▁▁▂▁▃▂█▅▄ beats 9/35	medium · avg 58%	▇ 2.4%	at its level · several runs

11 more tests

test	result	vs the others in its run	difficulty	counts	why
SuperGPQA lab +1	53.4% ✎	▂▁▂▂▅▆▃▇▇▇█▃ beats 24/46	medium · avg 36%	▇ 2.3%	at its level · several runs
LiveBench 2024-11-25 lab +1	69.0% ✎	▄··▃·▂·▂█▆▂▂ beats 2/13	medium · avg 55%	▇ 1.9%	at its level · several runs
BRIDGE clinical text, zero-shot average (87 tasks)	36.9% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 53/97	hard · avg 32%	▇ 1.9%	blunt · at its level
WritingBench lab +1	85.5% ✎	▁·▁▂▁··▅█▃▂▁ beats 17/20	easy · avg 70%	▇ 1.8%	several runs
Multi-IF (aggregation unspecified) lab +1	67.9% ✎	▁▁▁▁▁▁▃▄█▄▁▂ beats 10/41	medium · avg 60%	▇ 1.7%	blunt · at its level · several runs
Chess Puzzles (Epoch)	2.0% ✎	█▁▁▁▁▂▁▄▂▂▁▁ beats 43/135	very hard · avg 5.9%	▇ 0.6%	sharp · too hard for it · noisy
Open PT-LLM Leaderboard average (9 Portuguese tasks)	78.9% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 248/276	medium · avg 73%	▇ 0.5%	noisy
ZebraLogic lab	90.0% ✎	█·▁▁▁▇·▄▄▁▄▁ beats 17/24	medium · avg 47%	▇ 0.4%	at its level · noisy · small run
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	88.3% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 89/100	medium · avg 68%	▇ 0.4%	noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	90.6% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 118/146	easy · avg 83%	▇ 0.2%	noisy
LIBRA Mini (Russian long context, May 2026 version)	51.5% ✎	▄···▄█·██··▄ beats 8/8	medium · avg 38%	▇ 0.2%	blunt · at its level

$ where to run it

Through an API

api · Nebius · $0.15/M · $0.1 in / $0.3 out per 1M tokens · 262k context · tokenfactory.nebius.com ↗ · price sourceUSD per 1M text tokens. Nebius standard synchronous API; model Qwen/Qwen3-30B-A3B-Instruct-2507. Uncached input; output includes generated reasoning where billed. Nebius FP8 serving variant; FI / eu-north1. This serving precision is not a claim about benchmark precision. Batch, cache storage, tools and other modalities excluded.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3 30B-A3B Instruct 2507 · needs ~18.6 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3 30B-A3B Instruct 2507 · needs ~18.6 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Qwen3-30B-A3B-Instruct-2507-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Qwen3 30B-A3B Instruct 2507 · needs ~18.8 GB · from ollama.com
chat in the terminal
ollama run qwen3:30b-instruct

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3 30B-A3B Instruct 2507 · needs ~17.2 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3-30B-A3B-Instruct-2507-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3-30B-A3B-Instruct-2507-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3-Next-80B-A3B-Instr… 43%, Qwen/Qwen3-30B-A3B-Instruct-2… 35%, Epoch AI — internal evaluatio… 10% and 7 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Qwen/Qwen3-Next-80B-A3B-Instruct model card · lab-reported	24	▇▇▇▇▇▇▇▇▇▇▇▇▇ 43%	apache-2.0
Qwen/Qwen3-30B-A3B-Instruct-2507 model card · lab-reported	25	▇▇▇▇▇▇▇▇▇▇▇ 35%	apache-2.0
Epoch AI — internal evaluations	3	▇▇▇ 10%	CC BY 4.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models	1	▇ 3%	CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 2%	MIT
Berkeley Function Calling Leaderboard result archive	1	▇ 2%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
ai-forever LIBRA leaderboard (Russian long-context benchmark)	1	▇ 0%	MIT
prior · pull toward the middle		▇ 0%
weighted average		= 60.2

$ published speed measurements · 8

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 8-bit affine quantization (q8, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-8bit)	83.16 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 6-bit affine quantization (q6, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-6bit)	94.14 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 5-bit affine quantization (q5, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-5bit)	101 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 4-bit affine quantization (q4, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit)	113.33 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	IQ4_XS-3.63bpw	100.38 tok/s	llama.cpp ac4cddeb0 (build_number 9592 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r10 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 28 (2026-06-11, measured-local): Latest upstream Vulkan/RADV control with explicit Vulkan0; keeps the direct 30B-class Qwen route above 100 t/s on ac4cddeb0. Separate general-instruct Qwen route, not the Qwen3-Coder headline.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	IQ4_XS-3.63bpw	98.32 tok/s	llama.cpp 2016bf2 (build_number 1 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=-1; type_k=f16; type_v=f16; n_gpu_layers=-1; threads=16; one sequence; file label r3 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 38 (2026-06-05, measured-local): Latest/int-dot regression check; generation-only p0/n128 r20 measured 99.10 t/s, below the earlier b9467 100.04 t/s r50 headline.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	IQ4_XS-3.63bpw	100.04 tok/s	llama.cpp 1fd5f4803 (build_number 134 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r50 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 43 (2026-06-02, measured-local): First local direct llama-bench row above 100 t/s on Strix Halo; r20 was 100.58 t/s and generation-only p0/n128 r20 was 100.40 t/s. Separate general-instruct Qwen route, not the Qwen3-Coder headline or balanced-default claim.	source ↗
AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	IQ4_XS-3.63bpw	103.18 tok/s	llama.cpp 98d5e8ba8 (build_number 9544 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r10 repetitions Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 98 (2026-06-07, measured-local): Latest llama.cpp b9544 Vulkan/RADV regression control with explicit -dev Vulkan0; r10; confirms the 30B-class Qwen 100 t/s route still holds on the current build.	source ↗

$ quantization evidence · 20 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
GGUF Q2_K_S produced by AutoRound 'Optimized RTN' (--iters 0, no tuning), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.639	94.7%	source ↗
GGUF Q2_K_S produced by AutoRound + alg_ext (tuned), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.646	95.8%	source ↗
GGUF Q3_K_S produced by AutoRound 'Optimized RTN' (--iters 0, no tuning), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.66	97.8%	source ↗
GGUF Q3_K_S produced by AutoRound + alg_ext (tuned), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.662	98.2%	source ↗
GGUF Q4_K_S produced by AutoRound 'Optimized RTN' (--iters 0, no tuning), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.676	100.2%	source ↗
GGUF Q4_K_S produced by AutoRound + alg_ext (tuned), auto-round <=0.12.3	Average accuracy of 10 lm-eval tasks (lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge, mmlu); lm-eval defaults	0.675 → 0.678	100.5%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MATH-500 — published row 'Math 500'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.899 → 0.922	102.5%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	IFEval — published row 'IfEval'; lm-evaluation-harness task ifeval via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.891 → 0.891	100.0%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	GSM8K-Platinum — published row 'GSM8k Platinum (5-shot)'; lm-evaluation-harness task gsm8k_platinum_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.961 → 0.958	99.7%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MMLU (CoT) — published row 'MMLU-Cot (5-shot)'; lm-evaluation-harness task mmlu_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.843 → 0.835	99.1%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MMLU-Pro — published row 'MMLU-Pro (5-shot)'; lm-evaluation-harness task mmlu_pro_chat via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.789 → 0.771	97.7%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	GSM8K-Platinum — published row 'GSM8k Platinum (5-shot)'; lm-evaluation-harness task gsm8k_platinum_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.961 → 0.976	101.5%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MATH-500 — published row 'Math 500'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.899 → 0.905	100.6%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MMLU (CoT) — published row 'MMLU-Cot (5-shot)'; lm-evaluation-harness task mmlu_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.843 → 0.843	100.0%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MMLU-Pro — published row 'MMLU-Pro (5-shot)'; lm-evaluation-harness task mmlu_pro_chat via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.789 → 0.788	99.9%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	IFEval — published row 'IfEval'; lm-evaluation-harness task ifeval via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.891 → 0.889	99.7%	source ↗
MLX 4-bit affine quantization (q4, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.726 → 0.707	97.4%	source ↗
MLX 5-bit affine quantization (q5, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-5bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.726 → 0.72	99.1%	source ↗
MLX 6-bit affine quantization (q6, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-6bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.726 → 0.724	99.7%	source ↗
MLX 8-bit affine quantization (q8, group size not stated (mlx-lm default); mlx-community/Qwen3-30B-A3B-Instruct-2507-8bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.726 → 0.725	99.8%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json