whatmodel.sh / Qwen3 235B-A22B Instruct 2507

Qwen3 235B-A22B Instruct 2507

+ contribute

Qwen3 235B-A22B Instruct 2507 is an open-weights language model from Alibaba, released July 2025. Its published size is 235B parameters (22B active per token) and a 262k-token context window. Licence: apache-2.0.

64.5±0.5#151 of 719 LLMsfrom 81 benchmark results across 9 sources

$ how it compares

The LLMs ranked either side of Qwen3 235B-A22B Instruct 2507, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
171	MiniMax-M2 MiniMax	65±1	api $0.53/M +local	~75GB	2025-10
172	Laguna XS.2 Poolside	65±2	local	~21GB	2026-04
173	Qwen3 235B-A22B Instruct 2507 Alibaba	65±1	api $0.20/M +local	~76GB	2025-07
174	Mistral Medium 3.5 (128B) Mistral	65±1	api $3.00/M +local	~42GB	2026-04
175	Command A+ (05-2026) Cohere	64±1	api price unknown +local	~71GB	2026-05
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #159 of 300 · 24% of its score · 10 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
τ²-bench Telecom +3	22.5% ✎	██··███···██ beats 1/6	very hard · avg 9.2%	▇▇▇ 6.9%	sharp · at its level · small run · several runs
τ²-bench Retail +2	70.5% ✎	▅···█·▃····▃ beats 2/6	medium · avg 41%	▇▇ 3.9%	at its level · several runs
τ-bench Retail lab +1	71.3% ✎	▁····▃·▂▇█▃▆ beats 22/28	hard · avg 32%	▇ 3.1%	at its level · several runs
τ²-bench Airline +2	36.0% ✎	▄·▄··▄···█▄▄ beats 0/6	hard · avg 23%	▇ 2.9%	at its level · several runs
SWE-bench Verified	42.0% ✎	▁▁▁▁▁▁▂▄█▆▁▁ beats 30/131	hard · avg 18%	▇ 2.3%	sharp · at its level
τ-bench Airline lab +1	44.0% ✎	▁▁▁·▅▄▁█▇▁▁▇ beats 13/28	hard · avg 20%	▇ 2.0%	at its level · several runs
BFCL V4 (December 2025 archive)	48.0% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 54/77	hard · avg 32%	▇ 1.3%	at its level
BFCL v3 lab +1	70.9% ✎	▁▁▁▁▁▃▅█▆▅▄▁ beats 154/170	medium · avg 57%	▇ 0.8%	blunt · at its level · several runs

2 more tests

test	result	vs the others in its run	difficulty	counts	why
Terminal-Bench (pre-2.0 release)	17.3% ✎	▄·▄···▄▄···█ beats 1/5	very hard · avg 12%	▇ 0.7%	at its level
Galileo Agent Leaderboard v2 (action completion)	53.0% ✎	▅▃·▃▃▅▅█▃█▅▅ beats 14/19	hard · avg 27%	▇ 0.6%	at its level

coding▁▁▁▃▆██▇▆▄▃▁ #158 of 458 · 18% of its score · 7 tests

Beats Gemini 2.5 Pro, Qwen3-Coder-Next (80B-A3B), Grok 4.1 Fast (Reasoning) here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 8B Reasoning 2512, Phi-4-reasoning 14B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) lab	57.3% ✎	▂▁▁▃▆▃▄▆█▃▄▂ beats 49/71	very hard · avg 9.0%	▇▇ 5.4%	sharp · at its level
LiveCodeBench v6 (Feb–May 2025) lab +2	51.8% ✎	▁▁▃▁█▅▄▇▅▅▃█ beats 5/38	hard · avg 28%	▇ 3.0%	at its level · several runs
SWE-bench Verified	42.0% ✎	▁▁▁▁▁▁▂▄█▆▁▁ beats 30/131	hard · avg 18%	▇ 2.3%	sharp · at its level
LiveCodeBench (version unspecified)	46.5% ✎	█·······▄▄▄▄ beats 3/5	hard · avg 21%	▇ 2.2%	at its level
MultiPL-E lab +1	87.9% ✎	▃··▄·▅▅▇▄█▃▄ beats 31/31	easy · avg 68%	▇ 1.7%	several runs
MBPP+	79.9% ✎	▃·····▃█▃··▃ beats 3/6	medium · avg 61%	▇ 1.5%	typical
HumanEval+	94.5% ✎	▄▄▄·····▄▄·█ beats 6/6	easy · avg 67%	▇ 1.4%	sharp

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #88 of 393 · 8.0% of its score · 3 tests

Beats LongCat 2.0 (1.6T-A48B), Inkling Small (266B-A12B), Qwen3.5 122B-A10B here, though they rank higher overall. Trails Gemma 4 12B, Gemma 4 26B-A4B, Mistral Medium 3.5 (128B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard v2 aggregate lab +1	79.2% ✎	▁▁··▄▃▁▃█▁▁▇ beats 18/22	medium · avg 36%	▇ 2.9%	sharp · small run · several runs
LMArena (text)	1422.4 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 272/379	avg 1268	▇ 2.7%	blunt
IFEval (aggregation unspecified) +2	88.5% ✎	█··█·█··████ beats 4/6	easy · avg 75%	▇ 2.4%	several runs

math▁▂▃▅▆▅▄█▄▁▁▁ #91 of 398 · 11% of its score · 5 tests

Beats Claude Sonnet 4.6, Claude Opus 4.5, MiniMax-M3 here, though they rank higher overall. Trails Gemini 2.5 Flash, Granite 4.2 30B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AIME 2025 +3	68.3% ✎	▄▄···▄▄··▄·█ beats 6/6	hard · avg 34%	▇▇ 5.2%	sharp · small run · several runs
HMMT 2025 (edition unspecified) lab +1	55.4% ✎	▃█▇▅▇▅▁·▁··▁ beats 12/32	hard · avg 19%	▇ 3.7%	sharp · at its level · small run · several runs
PolyMATH lab	50.2% ✎	▁···▃▁·▇█▄▄▄ beats 12/23	hard · avg 25%	▇ 0.9%	at its level
AIME 2024	81.7% ✎	▁▁▁▂▁▃▁▅███▃ beats 57/85	hard · avg 32%	▇ 0.7%	sharp · at its level · noisy
MATH-500	98.8% ✎	▄·▄··█··▄·▄▄ beats 6/6	very easy · avg 86%	▇ 0.6%	maxed out · noisy

other tests23 not in a task · 41% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Pro +3	84.8% ✎	█▄·▄···▄··▄▄ beats 6/6	medium · avg 59%	▇▇ 6.0%	several runs
MMLU	90.2% ✎	▄·····█▄▄▄·▄ beats 4/6	medium · avg 71%	▇ 3.0%	typical
GPQA Diamond	77.4% ✎	█·█·█··██··█ beats 4/5	hard · avg 42%	▇ 2.8%	sharp · at its level
INCLUDE lab +1	79.5% ✎	▂····▂▂▄▄▄█▄ beats 26/36	medium · avg 58%	▇ 2.5%	at its level · several runs
GPQA (subset unspecified) lab	77.5% ✎	▂▁▂▂▄▃▆▃▅█▄▃ beats 91/104	hard · avg 45%	▇ 2.5%	sharp · at its level
MMLU-Redux lab +1	93.1% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 52/63	easy · avg 79%	▇ 2.2%	several runs
ARC-AGI-1 public evaluation set (ARC Prize model_baseline, pass@2)	17.0% ✎	▁·▇▃▁·█▇▁·▇▃ beats 7/27	very hard · avg 4.1%	▇ 2.2%	sharp · at its level · noisy
Creative Writing v3 lab +1	87.5% ✎	▂····▂▆··▆█▆ beats 13/14	easy · avg 69%	▇ 1.8%	several runs

15 more tests

test	result	vs the others in its run	difficulty	counts	why
SuperGPQA lab +1	62.6% ✎	▂▁▂▂▅▆▃▇▇▇█▃ beats 37/46	medium · avg 36%	▇ 1.6%	at its level · several runs
MMLU-ProX lab +1	79.4% ✎	▁·▁▁▁▂▁▃▂█▅▄ beats 22/35	medium · avg 58%	▇ 1.6%	several runs
SimpleQA lab +1	54.3% ✎	▄▄▄▆█▆███·▂█ beats 31/32	hard · avg 17%	▇ 1.5%	at its level · small run · several runs
CMMLU	88.1% ✎	█····▄··▄▄▄▄ beats 5/6	medium · avg 67%	▇ 1.5%	typical
ChineseSimpleQA lab +1	84.3% ✎	▄··▄▄▄··▄··█ beats 6/6	medium · avg 58%	▇ 1.4%	small run · several runs
C-Eval	92.7% ✎	█····▄·▄·▄▄▄ beats 6/6	medium · avg 68%	▇ 1.3%	typical
WritingBench lab +1	85.2% ✎	▁·▁▂▁··▅█▂▂▁ beats 12/20	easy · avg 70%	▇ 1.3%	several runs
LiveBench 2024-11-25 lab +1	75.4% ✎	▄··▂·▂·▂█▆▂▂ beats 7/13	medium · avg 55%	▇ 1.3%	several runs
Multi-IF (aggregation unspecified) lab +1	77.5% ✎	▁▁▁▁▁▁▃▄█▄▁▂ beats 26/41	medium · avg 60%	▇ 1.2%	blunt · several runs
VitaBench	8.5% ✎	█·······▄█·█ beats 1/6	very hard · avg 6.4%	▇ 1.2%	typical
COLLIE (constrained text generation)	49.7% ✎	▄···▄█▄····█ beats 2/6	hard · avg 30%	▇ 1.1%	at its level
DROP	78.6% ✎	▄···▄·▄█·▄·▄ beats 3/6	medium · avg 41%	▇ 1.0%	noisy
ARC-AGI-2 public evaluation set (ARC Prize model_baseline, pass@2)	0.8% ✎	█··▂▅▁▃▁▃·▁▃ beats 5/24	very hard · avg 0.3%	▇ 0.9%	sharp · too hard for it · noisy
ZebraLogic +1	94.2% ✎	█···█··███·█ beats 5/5	medium · avg 53%	▇ 0.7%	noisy · several runs
BeyondAIME	57.6% ✎	█····█·▄▄··▄ beats 6/6	hard · avg 25%	▇ 0.4%	at its level

$ where to run it

Through an API

api · Deep Infra · $0.21/M · $0.09 in / $0.55 out per 1M tokens · 262k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3 235B-A22B Instruct 2507 · needs ~71 GB · from huggingface.co/unsloth
chat in the terminal
llama cli -hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:UD-Q2_K_XL
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:UD-Q2_K_XL

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3 235B-A22B Instruct 2507 · needs ~132.2 GB, Mac only · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Qwen3-235B-A22B-Instruct-2507-MLX-4bit -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Qwen3 235B-A22B Instruct 2507 · needs ~141.5 GB · from ollama.com
chat in the terminal
ollama run qwen3:235b-a22b-instruct-2507-q4_K_M

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3 235B-A22B Instruct 2507 · needs ~132.2 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3-235B-A22B-Instruct-2507-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3-235B-A22B-Instruct-2507-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3-Next-80B-A3B-Instr… 32%, Meituan LongCat-Flash-Chat mo… 31%, Qwen3-VL-235B-A22B-Instruct m… 17% and 6 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Qwen/Qwen3-Next-80B-A3B-Instruct model card · lab-reported	24	▇▇▇▇▇▇▇▇▇▇ 32%	apache-2.0
Meituan LongCat-Flash-Chat model card (evaluation table)	22	▇▇▇▇▇▇▇▇▇ 31%	MIT
Qwen3-VL-235B-A22B-Instruct model card · lab-reported	18	▇▇▇▇▇ 17%	Apache-2.0
Qwen/Qwen3-235B-A22B-Instruct-2507 model card · lab-reported	11	▇▇▇ 11%	apache-2.0
ARC Prize Foundation model_baseline outputs on ARC-AGI-1/2 public evaluation sets	2	▇ 3%	MIT
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Berkeley Function Calling Leaderboard result archive	1	▇ 1%	Apache-2.0
SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages (arXiv:2606.28715v2)	1	▇ 1%	CC-BY-4.0
Galileo Agent Leaderboard (v1 and v2 results)	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 64.5

$ published speed measurements · 1

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S; pre-production Framework Desktop 128GB LPDDR5x-8000

UD-Q3_K_XL

15.2 tok/s

llama.cpp bbd0f9177 (commit from system_info.json), backend vulkan · llama-bench tg128 (128 generated tokens, empty context); backend variant 'llama.cpp-vulkan'; flags: -b 256; mmap disabled per the guide; value = bold best tg128 in this model's Benchmark Results table

Author's own sweep; README value 15.202265 rounded to 2 decimals. Kernel 6.16.0-1-mainline; system_info timestamp 2025-08-02T15:32:15.305374Z; CPU string 'AMD RYZEN AI MAX+ 395 w/ Radeon 8060S'. Pre-production hardware and BIOS per the README.

source ↗

$ quantization evidence · 14 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ per card text; config.json at pinned revision confirms 8-bit int weights/activations; llm-compressor, compressed-tensors); MoE router gates unquantized	MATH-500 0-shot (lighteval math_500 via litellm endpoint, vLLM)	0.89 → 0.897	100.8%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ per card text; config.json at pinned revision confirms 8-bit int weights/activations; llm-compressor, compressed-tensors); MoE router gates unquantized	MMLU-Pro 5-shot (lm-evaluation-harness mmlu_pro_chat, chat completions via vLLM)	0.831 → 0.834	100.4%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ per card text; config.json at pinned revision confirms 8-bit int weights/activations; llm-compressor, compressed-tensors); MoE router gates unquantized	IFEval (lm-evaluation-harness ifeval, chat completions via vLLM; reported metric variant not stated)	0.917 → 0.917	100.0%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ per card text; config.json at pinned revision confirms 8-bit int weights/activations; llm-compressor, compressed-tensors); MoE router gates unquantized	MMLU 5-shot CoT (lm-evaluation-harness mmlu_cot_llama, chat completions via vLLM)	0.893 → 0.893	99.9%	source ↗
INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ per card text; config.json at pinned revision confirms 8-bit int weights/activations; llm-compressor, compressed-tensors); MoE router gates unquantized	GSM8K-Platinum 5-shot CoT (lm-evaluation-harness gsm8k_platinum_cot_llama, chat completions via vLLM)	0.975 → 0.97	99.5%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	OpenLLM v2 average as published (5 tasks listed: BBH, MMLU-Pro, MuSR, IFEval, GPQA; Math-Hard not included; lm-evaluation-harness, vLLM)	0.603 → 0.609	101.0%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	IFEval (OpenLLM v2 leaderboard configuration; lm-evaluation-harness, vLLM)	0.902 → 0.91	100.9%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	GSM8K (OpenLLM v1 configuration, 5-shot; lm-evaluation-harness, vLLM)	0.899 → 0.904	100.5%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	MMLU-Pro (OpenLLM v2 leaderboard configuration, 5-shot; lm-evaluation-harness, vLLM)	0.635 → 0.633	99.7%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	MMLU (OpenLLM v1 configuration, 5-shot; lm-evaluation-harness, vLLM)	0.875 → 0.871	99.5%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	OpenLLM v1 average (6 tasks: arc_challenge, gsm8k, hellaswag, mmlu, truthfulqa_mc2, winogrande; lm-evaluation-harness (neuralmagic fork), vLLM)	0.805 → 0.8	99.3%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	MATH-500 0-shot (lighteval, vLLM)	0.942 → 0.892	94.7%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	Reasoning average (2 tasks: GPQA Diamond 0-shot, MATH-500 0-shot; lighteval, vLLM)	0.817 → 0.769	94.2%	source ↗
NVFP4 W4A4 (FP4 E2M1 weights and activations, group 16; llm-compressor QuantizationModifier NVFP4, compressed-tensors); lm_head, self-attention projections and MoE router gates left unquantized	GPQA Diamond 0-shot (lighteval, vLLM)	0.692 → 0.646	93.4%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json