whatmodel.sh / Qwen3 4B Instruct 2507

Qwen3 4B Instruct 2507

+ contribute

Qwen3 4B Instruct 2507 is an open-weights language model from Alibaba, released August 2025. Its published size is 4.02B parameters and a 262k-token context window. Licence: apache-2.0.

55.1±0.6#263 of 719 LLMsfrom 55 benchmark results across 10 sources

$ how it compares

The LLMs ranked either side of Qwen3 4B Instruct 2507, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
315	Magistral Medium 1.1 (2507) Mistral	55±2	comparison only		2025-07
316	Gemini 1.5 Pro Experimental (0801) Google	55±1	comparison only		2024-08
317	Qwen3 4B Instruct 2507 Alibaba	55±1	local	~3.9GB	2025-08
318	Qwen2.5 Coder 14B Instruct Alibaba	55±1	local	~10GB	2024-11
319	Qwen3-VL 8B Instruct Alibaba	55±1	api $0.31/M +local	~6.8GB	2025-10
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #223 of 300 · 14% of its score · 7 tests

Beats o1 (2024-12-17), o3-mini, Llama 3.1 Nemotron Ultra 253B v1 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
τ²-bench Telecom +1	13.2% ✎	▁·▅█▆▅▂▃▃▂▃▁ beats 4/89	very hard · avg 12%	▇▇ 5.6%	sharp · several runs
BFCL v3 lab +1	61.9% ✎	█▄··▄█·█▄█▄█ beats 3/13	medium · avg 59%	▇ 1.6%	blunt · at its level · several runs
τ-bench Retail lab	48.7% ✎	▁····▃·▃▇█▂▆ beats 4/28	hard · avg 31%	▇ 1.6%	at its level · small run
BFCL V4 (December 2025 archive)	35.7% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 39/77	hard · avg 32%	▇ 1.4%	at its level
τ²-bench Retail lab	40.4% ✎	▁····▃▁▃▆█▃▁ beats 4/44	hard · avg 30%	▇ 1.3%	at its level · small run
τ²-bench Airline lab	24.0% ✎	▃▁·▂▁▃▂▁█▅▂▃ beats 0/42	hard · avg 18%	▇ 1.0%	small run
τ-bench Airline lab	32.0% ✎	▁▁▁·▅▄▁█▇▁▁▇ beats 5/28	hard · avg 19%	▇ 1.0%	at its level · small run

coding▁▁▁▃▆██▇▆▄▃▁ #314 of 458 · 12% of its score · 5 tests

Beats GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, WizardLM 70B V1.0, Phi-4-mini 3.8B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) lab	12.9% ✎	▂▁▁▃▆▃▄▆█▃▄▂ beats 10/71	very hard · avg 8.9%	▇▇ 4.7%	sharp · small run
LiveCodeBench v6 (window unspecified) lab	35.1% ✎	▃▅█▅▅▃▅···▃█ beats 3/16	medium · avg 35%	▇ 2.6%	at its level
Luau Leetcode fully solved (Torpedo Software)	15.4% ✎	▁▃▁▁·▁▂▃▅▃█▇ beats 2/49	hard · avg 18%	▇ 2.1%	at its level
LiveCodeBench v6 (Feb–May 2025) lab	35.1% ✎	▁▃▃▁█▅▄▇▅▅▃█ beats 1/38	hard · avg 28%	▇ 1.3%	at its level · small run
MultiPL-E lab	76.8% ✎	▃··▄·▅▅▇▄█▃▄ beats 23/31	easy · avg 66%	▇ 1.0%	at its level · small run

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #178 of 393 · 3.9% of its score · 2 tests

Beats Seed-OSS 36B Instruct, Nemotron-Cascade-14B-Thinking, Qwen3-VL 32B Instruct here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
IFEval (aggregation unspecified) lab +1	83.4% ✎	▃··▃···██▅▃▃ beats 6/11	easy · avg 77%	▇ 2.5%	several runs
Arena-Hard v2 aggregate lab	43.4% ✎	▁▁··▄▃▁▃█▁▁▇ beats 6/22	hard · avg 20%	▇ 1.5%	sharp · at its level · small run

math▁▂▃▅▆▅▄█▄▁▁▁ #177 of 398 · 16% of its score · 8 tests

Beats MiniMax-M2.1, GPT-4.1, DeepSeek V3 0324 here, though they rank higher overall. Trails OpenThinker3 7B, Olmo 3 7B Think, OpenThinker2 32B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AIME 2025 lab +1	47.4% ✎	▃·▃▃█··▃▅▃·█ beats 5/12	medium · avg 45%	▇▇ 4.6%	sharp · at its level · several runs
GSM8K	88.6% ✎	█····█··▄·▄█ beats 5/7	very easy · avg 86%	▇ 2.2%	sharp
OTIS mock AIME 2024–2025	52.2% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 79/193	very hard · avg 8.5%	▇ 2.1%	sharp · noisy
HMMT 2025 (edition unspecified) lab	31.0% ✎	▃█▇▅▇▅▁·▁··▁ beats 1/32	very hard · avg 14%	▇ 2.0%	sharp · small run
PolyMath (Marco evaluation)	47.2% ✎	▄··▄██··▄··▄ beats 7/7	hard · avg 30%	▇ 1.5%	at its level
MATH (subset unspecified)	93.4% ✎	█▄··▄█▄····▄ beats 7/7	easy · avg 81%	▇ 1.5%	typical
MGSM (Marco evaluation)	84.4% ✎	▄·▄·▄▄▄·▄··█ beats 7/7	easy · avg 70%	▇ 1.1%	typical
PolyMATH lab	31.1% ✎	▁···▃▃·▇█▄▄▄ beats 3/23	hard · avg 23%	▇ 1.0%	at its level · small run

other tests23 not in a task · 55% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +1	45.8% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 50/208	hard · avg 38%	▇▇▇▇ 11%	sharp · at its level · several runs
MMLU-Pro lab +2	69.6% ✎	▄·▄·▄▄▄▄█▄██ beats 4/12	medium · avg 63%	▇▇▇ 8.4%	at its level · several runs
GPQA (subset unspecified) lab +1	62.0% ✎	▂····▂·▂▂█▆█ beats 6/14	hard · avg 49%	▇▇▇ 7.4%	sharp · at its level · several runs
MMLU-Redux lab +1	84.2% ✎	▃··▃·▃·█▃▅▃▅ beats 3/11	easy · avg 79%	▇▇ 4.8%	several runs
SuperGPQA lab +1	42.8% ✎	▄··▄·▄▄▄█▄██ beats 3/11	medium · avg 37%	▇ 3.5%	at its level · several runs
MMLU	80.8% ✎	▄···▄▄·▄█··█ beats 7/7	easy · avg 75%	▇ 2.9%	typical
MMLU-ProX-Lite (Marco)	58.3% ✎	█··▄··█··█·▄ beats 7/7	medium · avg 43%	▇ 2.2%	at its level
RobloxQA v2.0 multiple choice (Torpedo Software)	82.1% ✎	▁▃▃▁█▄▃▃▆▂▂▂ beats 2/49	easy · avg 78%	▇ 1.8%	typical

15 more tests

test	result	vs the others in its run	difficulty	counts	why
BRIDGE clinical text, zero-shot average (87 tasks)	34.1% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 42/97	hard · avg 32%	▇ 1.6%	blunt · at its level
GlobalMMLU (Marco evaluation)	70.2% ✎	▃··▃▃··▅···█ beats 7/7	medium · avg 57%	▇ 1.6%	at its level
INCLUDE lab	60.1% ✎	▂····▂▂▄▄▄█▄ beats 12/36	medium · avg 55%	▇ 1.4%	at its level · small run
MMMLU (Marco evaluation)	71.3% ✎	▄···▄·█·▄·▄█ beats 7/7	medium · avg 60%	▇ 1.3%	at its level
AGIEval (Marco suite unspecified)	51.7% ✎	▄··▄█▄··▄··█ beats 6/7	medium · avg 47%	▇ 1.1%	blunt · at its level
MMLU-ProX lab	61.6% ✎	▁·▁▁▁▃▁▃▂█▅▄ beats 5/35	medium · avg 55%	▇ 0.9%	at its level · small run
LiveBench 2024-11-25 lab	63.0% ✎	▄··▂·▂·▂█▆▂▂ beats 0/13	medium · avg 54%	▇ 0.8%	at its level · small run
Multi-IF (aggregation unspecified) lab	69.0% ✎	▁▁▁▁▁▁▃▄█▄▁▂ beats 11/41	medium · avg 62%	▇ 0.7%	blunt · at its level · small run
WritingBench lab	83.4% ✎	▁·▁▂▁··▅█▂▂▁ beats 6/20	easy · avg 69%	▇ 0.7%	small run
Creative Writing v3 lab	83.5% ✎	▂····▂▆··▆█▆ beats 5/14	medium · avg 65%	▇ 0.6%	at its level · small run
Open PT-LLM Leaderboard average (9 Portuguese tasks)	74.7% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 221/276	medium · avg 73%	▇ 0.6%	at its level · noisy
ZebraLogic lab	80.2% ✎	█·▁▁▁▇·▄▄▁▄▁ beats 13/24	medium · avg 49%	▇ 0.5%	at its level · noisy · small run
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	77.3% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 65/100	medium · avg 68%	▇ 0.5%	at its level · noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	81.7% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 74/146	easy · avg 83%	▇ 0.3%	noisy
Chess Puzzles (Epoch)	4.0% ✎	█▁▁▁▁▂▁▄▂▂▁▁ beats 35/135	very hard · avg 5.9%	▇ 0.2%	sharp · too hard for it · noisy

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3 4B Instruct 2507 · needs ~2.5 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3 4B Instruct 2507 · needs ~2.5 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Qwen3-4B-Instruct-2507-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Qwen3 4B Instruct 2507 · needs ~2.91 GB · from ollama.com
chat in the terminal
ollama run qwen3:4b-instruct

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3 4B Instruct 2507 · needs ~2.3 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3-4B-Instruct-2507-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3-4B-Instruct-2507-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3-4B-Instruct-2507 m… 34%, Marco-MoE: Open Multilingual… 24%, Qwen (Alibaba) vendor-reporte… 24% and 7 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Qwen/Qwen3-4B-Instruct-2507 model card · lab-reported	25	▇▇▇▇▇▇▇▇▇▇ 34%	apache-2.0
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling	11	▇▇▇▇▇▇▇ 24%	CC BY 4.0
Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · lab-reported	8	▇▇▇▇▇▇▇ 24%	Factual vendor-reported score (not licensed content)
Epoch AI — internal evaluations	3	▇▇ 7%	CC BY 4.0
Roblox LLM Leaderboard (Torpedo Software): RobloxQA v2.0 and Luau Leetcode	2	▇ 4%	MIT
Synthesize and Reward: Reinforcement Learning for Multi-Step Tool Use in Live Environments (arXiv:2606.03892v2)	1	▇ 3%	CC-BY-4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 2%	MIT
Berkeley Function Calling Leaderboard result archive	1	▇ 1%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 55.1

$ published speed measurements · 7

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

Apple M4 Max (64GB; CPU/GPU core count not stated)	bf16 (unquantized Qwen/ repo)	52.47 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 8-bit affine quantization (q8, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-8bit)	86.907 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 6-bit affine quantization (q6, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-6bit)	104.68 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 5-bit affine quantization (q5 g32, group size 32; mlx-community/Qwen3-4B-Instruct-2507-5bit-g32)	110.29 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 5-bit affine quantization (q5, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-5bit)	116.39 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 4-bit affine quantization (q4 g32, group size 32; mlx-community/Qwen3-4B-Instruct-2507-4bit-g32)	126 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗
Apple M4 Max (64GB; CPU/GPU core count not stated)	MLX 4-bit affine quantization (q4, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-4bit)	134.52 tok/s	mlx-lm 0.28.2, mlx 0.29.2.dev20251008+85a8824a8, macOS 26.1 · mlx_lm.benchmark -p 2048 -g 128; 'Generation (128) tok/sec' column: decode throughput for 128 generated tokens after a 2048-token prompt; batch size not stated beyond the documented command Published generation column; the separate Prompt (2048) tok/sec column is prompt processing and is not recorded as generation speed. Memory GB column retained as peak_memory_gb as printed. M4 Max core count is not stated; only the 64GB configuration is.	source ↗

$ quantization evidence · 18 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	GSM8K-Platinum — published row 'GSM8k Platinum (5-shot)'; lm-evaluation-harness task gsm8k_platinum_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.956 → 0.961	100.5%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MATH-500 — published row 'Math 500'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.843 → 0.842	99.8%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	IFEval — published row 'IfEval'; lm-evaluation-harness task ifeval via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.89 → 0.886	99.6%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MMLU (CoT) — published row 'MMLU-CoT (5-shot)'; lm-evaluation-harness task mmlu_cot_llama via vLLM server (command not found in card)	0.775 → 0.768	99.1%	source ↗
INT4 W4A16 (weight-only INT4, symmetric, group 128; GPTQ, llm-compressor)	MMLU-Pro — published row 'MMLU-Pro (5-shot)'; lm-evaluation-harness task mmlu_pro_chat via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.701 → 0.689	98.3%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	GSM8K-Platinum — published row 'GSM8k Platinum (5-shot)'; lm-evaluation-harness task gsm8k_platinum_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.956 → 0.957	100.1%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	IFEval — published row 'IfEval'; lm-evaluation-harness task ifeval via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.89 → 0.89	100.0%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	GPQA Diamond — published row 'GPQA diamond'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.452 → 0.452	100.0%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MMLU-Pro — published row 'MMLU-Pro (5-shot)'; lm-evaluation-harness task mmlu_pro_chat via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.701 → 0.7	99.8%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MMLU (CoT) — published row 'MMLU-CoT (5-shot)'; lm-evaluation-harness task mmlu_cot_llama via vLLM OpenAI-compatible server (backend local-chat-completions, chat template, --num_fewshot 0, gen_kwargs do_sample=True,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000)	0.775 → 0.77	99.3%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	MATH-500 — published row 'Math 500'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.843 → 0.836	99.2%	source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; SmoothQuant+GPTQ, llm-compressor)	AIME 2025 — published row 'AIME 2025'; lighteval (litellm endpoint to vLLM server), temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32000	0.476 → 0.467	98.0%	source ↗
MLX 4-bit affine quantization (q4 g32, group size 32; mlx-community/Qwen3-4B-Instruct-2507-4bit-g32)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.615	96.0%	source ↗
MLX 4-bit affine quantization (q4, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-4bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.607	94.8%	source ↗
MLX 5-bit affine quantization (q5 g32, group size 32; mlx-community/Qwen3-4B-Instruct-2507-5bit-g32)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.632	98.6%	source ↗
MLX 5-bit affine quantization (q5, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-5bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.624	97.4%	source ↗
MLX 6-bit affine quantization (q6, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-6bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.635	99.2%	source ↗
MLX 8-bit affine quantization (q8, group size not stated (mlx-lm default); mlx-community/Qwen3-4B-Instruct-2507-8bit)	MMLU-Pro (mlx_lm.evaluate --task mmlu_pro; lm-evaluation-harness task defaults, shots not stated in document)	0.64 → 0.638	99.7%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json