whatmodel.sh / Qwen3.5 397B-A17B

Qwen3.5 397B-A17B

+ contribute

Qwen3.5 397B-A17B is an open-weights language model from Alibaba, released February 2026. Its published size is 397B parameters (17B active per token) and a 262k-token context window. Licence: apache-2.0.

70.6±0.4#77 of 719 LLMsfrom 128 benchmark results across 21 sources

$ how it compares

The LLMs ranked either side of Qwen3.5 397B-A17B, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
80	GPT-5.1 OpenAI	71±1	api $3.44/M		2025-11
81	GLM-5.1 Z.ai	71±1	api $2.15/M +local	~119GB	2026-04
82	Qwen3.5 397B-A17B Alibaba	71±0	api $1.09/M +local	~65GB	2026-02
83	Ornith 1.5 35B-A3B DeepReinforce	71±2	local	~22GB	2026-08
84	MiniMax-M3 MiniMax	70±1	api $0.48/M +local	~137GB	2026-06
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #66 of 300 · 22% of its score · 21 tests

Beats Gemini 3.6 Flash, GPT-5 Pro, LongCat 2.0 (1.6T-A48B) here, though they rank higher overall. Trails MiMo-V2.5 (310B-A15B), MiniMax-M3 here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
SWE-bench Verified +2	73.6% ✎	█····██·████ beats 2/6	hard · avg 22%	▇▇ 4.4%	sharp · at its level · several runs
SWE-bench Multilingual +1	70.9% ✎	▄··▄▄···▄▄·█ beats 1/6	hard · avg 30%	▇ 2.1%	at its level · several runs
Terminal-Bench 2.1	49.9% ✎	█··▄█·▄····▄ beats 1/6	very hard · avg 12%	▇ 1.5%	at its level
ContractBench success rate (33 observation-contract tool tasks)	70.7% ✎	▆··▆▄▂▄▂▆▆█▄ beats 20/23	very hard · avg 14%	▇ 1.4%	sharp · at its level
BrowseComp +1	40.5% ✎	▄·▄▄···▄··█▄ beats 0/6	hard · avg 16%	▇ 1.3%	at its level · several runs
Humanity's Last Exam with tools +1	48.3% ✎	▄····▄·█·▄·▄ beats 3/5	hard · avg 16%	▇ 1.2%	at its level · several runs
τ²-bench aggregate lab	86.7% ✎	▄····█▄▄···▄ beats 3/5	medium · avg 40%	▇ 1.2%	typical
τ³-bench telecom (tau2-bench ≥ v1.0.1) +1	97.8% ✎	█·▄█▄······▄ beats 6/6	medium · avg 62%	▇ 1.0%	several runs

13 more tests

test	result	vs the others in its run	difficulty	counts	why
τ³-bench banking_knowledge (v1.0.1 grading)	9.8% ✎	▃▃▂▃·▆▃▂█▅▅▂ beats 0/25	very hard · avg 4.1%	▇ 1.0%	typical
SWE-bench Pro public set, Qwen-refined lab	50.9% ✎	▄······▄█·▄▄ beats 2/5	hard · avg 18%	▇ 0.9%	at its level
τ³-bench airline (tau2-bench ≥ v1.0.1)	81.5% ✎	▃······▃·▃█▃ beats 2/6	medium · avg 49%	▇ 0.8%	typical
BrowseComp-ZH lab	70.3% ✎	▁▁▁▁▁▃·▁▄▅▄█ beats 22/23	hard · avg 19%	▇ 0.7%	at its level · small run
τ³-bench	71.0% ✎	▄····▄·█·▄·█ beats 3/6	medium · avg 40%	▇ 0.7%	blunt · at its level
τ³-bench retail (tau2-bench ≥ v1.0.1)	84.4% ✎	▄▄·█··▄·▄··▄ beats 6/6	medium · avg 46%	▇ 0.7%	typical
OSWorld-Verified lab	62.2% ✎	█▄·▂▆▃▄▂▆▄▆█ beats 10/25	hard · avg 20%	▇ 0.6%	at its level · small run
Terminal-Bench 2.0 lab	52.5% ✎	▂▂▅▅████▆▂▃▃ beats 22/36	very hard · avg 8.5%	▇ 0.5%	at its level · small run
EEGAgentBench overall (tool-based EEG analysis agents, 6 tasks)	50.1% ✎	▂·▁▁··▇▇█▂·▁ beats 20/25	hard · avg 29%	▇ 0.5%	blunt · at its level
MCPMark lab	46.1% ✎	▃·····█··▃█▃ beats 8/8	very hard · avg 9.9%	▇ 0.4%	at its level · small run
BFCL V4 (December 2025 archive) lab	72.9% ✎	▁▁▁▂▇▆▆▆█▂▄▃ beats 108/111	medium · avg 39%	▇ 0.3%	at its level · small run
Toolathlon (Tool Decathlon) lab	38.3% ✎	▂▂▄▂▄▄▆▂█▆██ beats 9/27	very hard · avg 10%	▇ 0.3%	at its level · small run
WideSearch lab	74.0% ✎	█·▂▆··▃···▂▂ beats 8/10	hard · avg 35%	▇ 0.3%	at its level · small run

coding▁▁▁▃▆██▇▆▄▃▁ #82 of 458 · 15% of its score · 8 tests

Beats Grok 4.20 (0309 reasoning), GPT-5.4 mini, GPT-5.1 here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, GLM-4.7, Gemini 3.5 Flash-Lite here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
SWE-bench Verified +2	73.6% ✎	█····██·████ beats 2/6	hard · avg 22%	▇▇ 4.4%	sharp · at its level · several runs
Arena WebDev (Code Arena)	1399.4 ✎	▁▂▄▃▅█▄▇▆▄▁▁ beats 45/119	avg 981	▇ 2.8%	sharp
LiveCodeBench v6 (Feb–May 2025) +2	82.6% ✎	▄··█····▄▄·█ beats 3/6	medium · avg 47%	▇ 2.1%	several runs
SWE-bench Multilingual +1	70.9% ✎	▄··▄▄···▄▄·█ beats 1/6	hard · avg 30%	▇ 2.1%	at its level · several runs
SciCode subproblems	48.0% ✎	▃····▃··█·▃▃ beats 3/6	hard · avg 17%	▇ 0.9%	at its level
LiveCodeBench v6 (window unspecified)	79.3% ✎	██···█··████ beats 1/6	medium · avg 45%	▇ 0.9%	typical
SWE-bench Pro public set, Qwen-refined lab	50.9% ✎	▄······▄█·▄▄ beats 2/5	hard · avg 18%	▇ 0.9%	at its level
SciCode (level unspecified)	42.0% ✎	▄·····▄█▄▄·▄ beats 3/6	hard · avg 16%	▇ 0.8%	at its level

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #65 of 393 * · 1.9% of its score · 2 tests

Beats GPT-5.2, Qwen3.8 27B, LongCat 2.0 (1.6T-A48B) here, though they rank higher overall. Trails Nemotron Cascade 2 30B-A3B, Gemma 4 31B, Gemini 2.5 Pro here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1441.8 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 302/379	avg 1268	▇ 1.6%	blunt
IFEval (aggregation unspecified) lab	92.6% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 128/141	easy · avg 79%	▇ 0.3%	maxed out · small run

math▁▂▃▅▆▅▄█▄▁▁▁ #83 of 398 · 20% of its score · 14 tests

Beats Claude Sonnet 4.6, Claude Opus 4.5 here, though they rank higher overall. Trails Granite 4.2 30B, INTELLECT-3, EXAONE 4.5 33B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
FrontierMath tiers 1–3 v2	29.5% ✎	▂··▁▁▃▄▄█▅▃▃ beats 20/82	very hard · avg 1.2%	▇▇▇ 7.5%	sharp · at its level
HMMT February 2026 lab +1	87.9% ✎	▃▃▂▃·██▆█▅▃▅ beats 28/33	medium · avg 36%	▇ 2.8%	sharp · several runs
AIME 2026 +1	92.5% ✎	▄·▄·▄▄··█··▄ beats 3/6	medium · avg 52%	▇ 1.6%	sharp · maxed out · small run · several runs
IMO-AnswerBench +1	83.1% ✎	▄····▄·▄▄·█▄ beats 1/6	medium · avg 36%	▇ 1.6%	small run · several runs
MathArena ArXivMath January 2026 (23 problems)	54.9% ✎	▂··▄▂▄█▆▆▄▄█ beats 10/23	hard · avg 22%	▇ 1.4%	at its level
MathArena ArXivMath December 2025 (17 problems)	38.2% ✎	▃·▃▃█▁·▄·▁▁▁ beats 4/18	hard · avg 19%	▇ 1.0%	at its level
OTIS mock AIME 2024–2025	88.9% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 141/193	very hard · avg 8.5%	▇ 0.7%	sharp · noisy
MATH-Vision lab	88.6% ✎	█·····▄▄·▄·▄ beats 5/5	medium · avg 46%	▇ 0.6%	typical

6 more tests

test	result	vs the others in its run	difficulty	counts	why
HMMT February 2025 lab	94.8% ✎	▁▁▁▃▁▅▅▃██▁▄ beats 26/31	easy · avg 70%	▇ 0.6%	maxed out · small run
HMMT November 2025 lab	92.7% ✎	▁▁····▁▅▃█▇▁ beats 17/23	easy · avg 71%	▇ 0.5%	maxed out · small run
MathVista (testmini) lab	90.3% ✎	▁··▁▄▄▇█▅▆█▄ beats 47/47	medium · avg 65%	▇ 0.4%	small run
PolyMATH lab	73.3% ✎	▁···▃▁·▇█▄▄▄ beats 23/23	hard · avg 32%	▇ 0.4%	at its level · small run
DynaMath lab	86.3% ✎	▂·····▂··█▄▆ beats 9/10	medium · avg 64%	▇ 0.4%	small run
We-Math lab	87.9% ✎	█··········▃ beats 3/3	medium · avg 56%	▇ 0.3%	small run

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #36 of 147 · 7.3% of its score · 21 tests

Beats MiMo-V2.6-Pro, MiMo-V2.6-Flash (309B-A15B), GPT-5.6 Luna here, though they rank higher overall. Trails Qwen3.6 Flash, Qwen3.6 Plus here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CharXiv (reasoning questions) lab +1	80.8% ✎	▄▂▄▆▄▄▆··▄█▂ beats 15/21	medium · avg 39%	▇ 2.0%	typical
Arena vision (style controlled)	1246.9 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 99/145	avg 1068	▇ 1.1%	blunt
MATH-Vision lab	88.6% ✎	█·····▄▄·▄·▄ beats 5/5	medium · avg 46%	▇ 0.6%	typical
BEAR-Bench (bilingual RU/EN enterprise and academic document images, overall accuracy)	75.4% ✎	▃▃·▅·▃·█·█▅▅ beats 14/14	hard · avg 24%	▇ 0.6%	at its level
MathVista (testmini) lab	90.3% ✎	▁··▁▄▄▇█▅▆█▄ beats 47/47	medium · avg 65%	▇ 0.4%	small run
MMMU (val) lab	85.0% ✎	▁····▁▁▄█▇▆▆ beats 52/52	medium · avg 59%	▇ 0.4%	small run
MMMU Pro lab	79.0% ✎	▂▃▁▃▆▇▃▇▇██▅ beats 48/52	medium · avg 47%	▇ 0.4%	at its level · small run
MMStar lab	83.8% ✎	▃▅▂▃▂▂▆█▆▆█▅ beats 34/34	medium · avg 63%	▇ 0.3%	small run

13 more tests

test	result	vs the others in its run	difficulty	counts	why
SimpleVQA lab	67.1% ✎	▃·▃·▃·▃·█▅▃▃ beats 10/10	medium · avg 37%	▇ 0.3%	at its level · small run
CountBench lab	97.2% ✎	▂▂▃▂·▂▂▃▃·▅█ beats 15/18	very easy · avg 89%	▇ 0.3%	maxed out · small run
Diagram-MMU diagram question answering (DQA accuracy, scientific TikZ diagrams)	83.4% ✎	▄·····▄▄·█·█ beats 3/6	medium · avg 55%	▇ 0.3%	blunt
HallusionBench lab	71.4% ✎	▃▁·▃▃▃▄▅▄█▅▅ beats 32/32	medium · avg 56%	▇ 0.2%	blunt · at its level · small run
RealWorldQA lab	83.9% ✎	▂▃▂█▃██▆█▂▆▆ beats 32/38	easy · avg 72%	▇ 0.1%	blunt · small run
CC-OCR lab	82.0% ✎	▁·▂··▂▂▃▃▂▆█ beats 25/26	easy · avg 72%	▇ 0.1%	blunt · small run
OCRBench lab	93.1% ✎	▁·▃▁▆▅▂█▅█▁▃ beats 37/37	easy · avg 83%	▇ 0.1%	blunt · small run
AI2D test lab	93.9% ✎	▁▁▅▇▃▇·▄·█·▃ beats 26/28	easy · avg 85%	▇ 0.1%	blunt · small run
PureDocBench document parsing (Avg3 over Clean/Digital/Real)	66.7% ✎	▂·▁·▂▂▅█·▁▂▂ beats 7/22	medium · avg 62%	▇ 0.0%	blunt · at its level
ScreenSpot-Pro lab	65.6% ✎	▂▃▅▆██▅▂···▂ beats 16/24	medium · avg 51%	—	not counted yet: no other model in the same run to compare it with
MMBench-EN dev v1.1 lab	93.7% ✎	█▄···▂··▄▂▄▆ beats 14/14	easy · avg 79%	—	not counted yet: no other model in the same run to compare it with
OmniDocBench 1.5 overall (higher is better) lab	90.8% ✎	▂··▂▂▂··▂▂█▅ beats 11/13	medium · avg 54%	—	not counted yet: no other model in the same run to compare it with
PMC-VQA lab	64.2% ✎	▃···▃··▃▃·▃█ beats 7/7		—	not counted yet: no other model in the same run to compare it with

web search▂▁▂▆▄█▃▃▁▁·▁ #60 of 80 · 3.9% of its score · 5 tests

Beats DeepSeek V4 Flash (Preview, Apr 2026), GPT-5.4 mini here, though they rank higher overall. Trails GPT-4o mini (2024-07-18), Mistral Small 3.1 24B (2503), Llama 4 Scout (109B-A17B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BrowseComp +1	40.5% ✎	▄·▄▄···▄··█▄ beats 0/6	hard · avg 16%	▇ 1.3%	at its level · several runs
Humanity's Last Exam with tools +1	48.3% ✎	▄····▄·█·▄·▄ beats 3/5	hard · avg 16%	▇ 1.2%	at its level · several runs
BrowseComp-ZH lab	70.3% ✎	▁▁▁▁▁▃·▁▄▅▄█ beats 22/23	hard · avg 19%	▇ 0.7%	at its level · small run
SealQA Seal-0 lab	46.9% ✎	▃····▃·····█ beats 3/4	hard · avg 19%	▇ 0.3%	at its level · small run
WideSearch lab	74.0% ✎	█·▂▆··▃···▂▂ beats 8/10	hard · avg 35%	▇ 0.3%	at its level · small run

other tests53 not in a task · 43% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +2	85.9% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 149/208	hard · avg 45%	▇▇▇ 6.5%	sharp · several runs
Humanity's Last Exam +3	28.5% ✎	▄··█▄··▄·▄·▄ beats 3/6	very hard · avg 5.6%	▇▇ 4.9%	at its level · several runs
GPQA (subset unspecified) +1	87.1% ✎	▄█▄▄·▄·····▄ beats 3/6	hard · avg 47%	▇ 3.5%	sharp · several runs
MMLU-Pro +2	88.3% ✎	▄·····▄▄▄·▄█ beats 6/6	medium · avg 53%	▇ 3.0%	several runs
LongBench v2 +1	68.9% ✎	▂·▁·▂▂▁▃▄▁█▄ beats 28/28	very hard · avg 34%	▇ 1.6%	at its level · several runs
Global-MMLU-Lite	91.9% ✎	▃······▅··█▃ beats 6/6	medium · avg 60%	▇ 1.3%	typical
IFBench (aggregation unspecified) +1	78.8% ✎	▃······█▃▃·▃ beats 5/6	medium · avg 44%	▇ 1.2%	several runs
MMLU-ProX +1	86.4% ✎	▄·····▄·▄▄█▄ beats 6/6	medium · avg 57%	▇ 1.2%	several runs

45 more tests

test	result	vs the others in its run	difficulty	counts	why
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items	43.0% ✎	▁▁▂▁▃▄█▆▄▅▃▃ beats 35/39	very hard · avg 22%	▇ 1.1%	blunt · at its level
DSEffi-Bench Pass (1,000 data-science tasks, greedy)	57.6% ✎	▅··▃▅·█·▃██▃ beats 10/15	hard · avg 24%	▇ 1.1%	at its level
Mystery Game Puzzles (Epoch)	18.0% ✎	█··▂▃▇▄█▂▃▁▁ beats 29/74	very hard · avg 9.4%	▇ 1.1%	sharp · noisy
MultiChallenge +1	63.9% ✎	▁··········█ beats 5/6	hard · avg 27%	▇ 1.1%	at its level · several runs
EQ-Bench 3 (emotional intelligence, judge Elo)	1262.9 ✎	▁·▂▁▄▅▅▃█▄▁▁ beats 38/62	avg 1044	▇ 1.0%	typical
Chess Puzzles (Epoch)	12.0% ✎	█▁▁▁▁▃▁▄▂▂▁▁ beats 61/135	very hard · avg 5.9%	▇ 0.9%	sharp · noisy
AgentWorldBench (open-ended, overall) lab	54.7% ✎	▄▄··▄▄·▄█▄██ beats 6/11	hard · avg 25%	▇ 0.9%	blunt · at its level
WildClawBench	34.5% ✎	▃·▅▅█·▅█▅▃▃▃ beats 7/17	very hard · avg 11%	▇ 0.9%	at its level
NL2Repo lab	32.2% ✎	▄·····█·▄▄·▄ beats 3/5	very hard · avg 8.2%	▇ 0.8%	at its level
IFBench prompt-level (loose)	78.2% ✎	▄▄·▄··▄▄···█ beats 3/6	medium · avg 45%	▇ 0.7%	typical
C-Eval lab	93.0% ✎	▁···▁▁▂▁▂▂▃█ beats 32/33	medium · avg 72%	▇ 0.6%	small run
INCLUDE lab	85.6% ✎	▂····▂▂▄▄▄█▄ beats 35/36	medium · avg 55%	▇ 0.6%	small run
QwenClawBench (Qwen internal) lab	51.8% ✎	█·█··█·····█ beats 0/3	hard · avg 23%	▇ 0.6%	at its level
MMLU-Redux lab	94.9% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 62/63	easy · avg 77%	▇ 0.5%	maxed out · small run
VitaBench lab	49.7% ✎	▄··▂▄▆▆█▄▄▄▃ beats 21/21	very hard · avg 12%	▇ 0.5%	at its level · small run
MMMLU lab	88.5% ✎	▂▁▁▂▃▂▄▄▅██▃ beats 42/46	medium · avg 62%	▇ 0.5%	small run
Claw-Eval Pass^3 lab	48.1% ✎	█▄▄·▄·▄▄·▄·▄ beats 1/8	hard · avg 24%	▇ 0.5%	at its level
SkillsBench 78-task subset (OpenCode, avg of 5) lab	30.0% ✎	█▄·········▄ beats 2/3	very hard · avg 9.9%	▇ 0.4%	at its level
Global PIQA lab	89.8% ✎	▃▃··▃··▃▃▅█▅ beats 11/11	medium · avg 71%	▇ 0.4%	small run
SuperGPQA lab	70.4% ✎	▂▁▂▂▅▆▃▇▇▇█▃ beats 46/46	medium · avg 37%	▇ 0.4%	at its level · small run
MAXIFE lab	88.2% ✎	▂·▂·▂···▆·▄█ beats 11/11	medium · avg 61%	▇ 0.4%	small run
CritPt	2.4% ✎	▄····▄█···█▄ beats 1/6	very hard · avg 1.0%	▇ 0.4%	too hard for it · noisy
VideoMMMU lab	84.7% ✎	▂···▂···▂▂█▆ beats 10/10	medium · avg 61%	▇ 0.4%	small run
DeepPlanning lab	34.3% ✎	▂·····▂··▄█▄ beats 9/9	very hard · avg 9.7%	▇ 0.3%	at its level · small run
LVBench lab	75.5% ✎	▇·█▅▅▄▄··▃▁▁ beats 24/28	hard · avg 51%	▇ 0.3%	at its level · small run
ERQA lab	67.5% ✎	▃··▃··█·▃█▅▅ beats 11/12	medium · avg 36%	▇ 0.3%	at its level · small run
RefSpatial-Bench lab	73.6% ✎	▃·▃····▃▃▅▅█ beats 10/10	medium · avg 43%	▇ 0.3%	at its level · small run
MMVU lab	75.4% ✎	▃···▃···▃▃▅█ beats 8/8	medium · avg 48%	▇ 0.3%	small run
Claw-Eval average score lab	70.7% ✎	▄·▄··█▄▄▄▄▄▄ beats 8/9	medium · avg 47%	▇ 0.3%	blunt · at its level
MMLongBench-Doc lab	61.5% ✎	▃····▃··▃▃▅█ beats 8/8	medium · avg 36%	▇ 0.3%	at its level · small run
MLVU lab	86.7% ✎	▂···▂···▂▄▄█ beats 9/10	easy · avg 69%	▇ 0.3%	small run
NOVA-63 lab	59.1% ✎	▃▃·▃···▃█▃▃█ beats 11/11	hard · avg 35%	▇ 0.3%	blunt · at its level · small run
EmbSpatial-Bench lab	84.5% ✎	▁·····▁·▁·▃█ beats 8/10	easy · avg 68%	▇ 0.3%	small run
Video-MME (without subtitles) lab	83.7% ✎	▁▃▁▃▁·█▄▄▅·▇ beats 26/27	medium · avg 67%	▇ 0.2%	blunt · small run
WMT25 Terminology Translation (term success rate)	87.0% ✎	▂·▂··▂·▄▃·█▆ beats 5/12	medium · avg 64%	▇ 0.2%	noisy
WMT24++ +1	78.5% ✎	█·▃·▅▅▃···▃▃ beats 5/10	easy · avg 71%	▇ 0.2%	blunt · several runs
MVBench lab	77.6% ✎	▁··▃▄▁▃▄▄▄█▄ beats 26/26	medium · avg 66%	▇ 0.1%	blunt · at its level · small run
WMT26 General MT (GEMBA-ESA, LLM judge)	81.6% ✎	▄·▄▄▄·▄██▄█▄ beats 10/12	easy · avg 65%	▇ 0.1%	blunt
AndroidWorld lab	66.8% ✎	▁·▁▁▂▃█▅▃▂·▂ beats 20/24		—	not counted yet: no other model in the same run to compare it with
Video-MME (with subtitles) lab	87.5% ✎	▁▁▁▁▁·▁▃▁▁▁█ beats 15/16		—	not counted yet: no other model in the same run to compare it with
MedXPertQA MM lab	70.0% ✎	▂···▂·▃··▆▂█ beats 13/13		—	not counted yet: no other model in the same run to compare it with
ZEROBench subquestions lab	41.0% ✎	▄·▄···▄·▄██▄ beats 8/8		—	not counted yet: no other model in the same run to compare it with
SLAKE lab	79.9% ✎	▄·····▄▄·██▄ beats 5/7		—	not counted yet: no other model in the same run to compare it with
LingoQA lab	81.6% ✎	▄······▄·█·█ beats 4/5		—	not counted yet: no other model in the same run to compare it with
SecCodeBench lab	68.3% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with

$ where to run it

Through an API

api · Deep Infra · $1.09/M · $0.45 in / $3 out per 1M tokens · 262k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3.5 397B-A17B · needs ~238.7 GB · from huggingface.co/unsloth
chat in the terminal
llama cli -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3.5 397B-A17B · needs ~238.7 GB · from huggingface.co/unsloth
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF@UD-Q4_K_XL -y
Then run lms ls to see its name, and lms chat followed by that name

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3.5 397B-A17B · needs ~223.9 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Qwen/Qwen3.5-397B-A17B model… 27%, NVIDIA — Nemotron 3 Ultra tec… 19%, Epoch AI — internal evaluatio… 12% and 18 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Qwen/Qwen3.5-397B-A17B model card · lab-reported	62	▇▇▇▇▇▇▇▇ 27%	apache-2.0
NVIDIA — Nemotron 3 Ultra technical report	18	▇▇▇▇▇▇ 19%	CC BY 4.0
Epoch AI — internal evaluations	5	▇▇▇▇ 12%	CC BY 4.0
Qwen/Qwen3.6-27B model card · lab-reported	12	▇▇▇ 12%	apache-2.0
A.X K2 Technical Report (SK Telecom), arXiv 2608.30181v1	7	▇▇ 8%	CC BY 4.0
LMArena leaderboard dataset	3	▇▇ 5%	CC BY 4.0
τ²/τ³-bench leaderboard: Sierra's own evaluation runs	4	▇ 3%	MIT
MathArena ArXivMath / BrokenArXiv / ArXivLean model-output datasets	2	▇ 2%	CC BY-SA 4.0
Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · lab-reported	1	▇ 2%	Factual vendor-reported score (not licensed content)
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)	1	▇ 1%	CC BY 4.0
LGAI-EXAONE/K-EXAONE-2.0-750B-A37B model card	1	▇ 1%	Apache-2.0
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)	1	▇ 1%	CC BY 4.0
DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation — authors' own model runs	1	▇ 1%	CC BY 4.0
EQ-Bench 3 canonical leaderboard results	1	▇ 1%	MIT
Qwen/Qwen-AgentWorld-35B-A3B model card · lab-reported	1	▇ 1%	Apache-2.0
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation (arXiv:2605.10912v1)	1	▇ 1%	CC BY 4.0
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models (arXiv:2608.17895v1)	1	▇ 1%	CC BY 4.0
EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis (arXiv:2609.31632v1)	1	▇ 0%	CC BY 4.0
Cohere — North Small Translate (arXiv:2609.13916v1)	3	▇ 0%	CC BY 4.0
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams (arXiv:2608.12262v1)	1	▇ 0%	CC BY 4.0
How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings (arXiv:2605.07492v2)	1	▇ 0%	CC BY 4.0
prior · pull toward the middle		▇ 0%
weighted average		= 70.6

not counted (12)

Qwen/Qwen3.5-397B-A17B model card	AndroidWorld [correct]	66.8%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	CharXiv (reasoning questions) [correct]	80.8%	the only model in this source's run of the test, so there is nothing to compare it with there; another run of the test counts
Qwen/Qwen3.5-397B-A17B model card	LingoQA [correct]	81.6%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	MedXPertQA MM [correct]	70.0%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	MMBench-EN dev v1.1 [correct]	93.7%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	OmniDocBench 1.5 overall (higher is better) [correct]	90.8%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	PMC-VQA [correct]	64.2%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	ScreenSpot-Pro [correct]	65.6%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	SecCodeBench [correct]	68.3%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	SLAKE [correct]	79.9%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	Video-MME (with subtitles) [correct]	87.5%	fewer than three models have this test yet, or this source ran it for this model alone
Qwen/Qwen3.5-397B-A17B model card	ZEROBench subquestions [correct]	41.0%	fewer than three models have this test yet, or this source ran it for this model alone

$ quantization evidence · 7 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	AIME 2025 0-shot (lighteval via litellm; mean of 8 seeds)	0.929 → 0.946	101.8%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	IFEval prompt-level strict 0-shot (lm-evaluation-harness ifeval, chat completions; mean of 3 seeds)	0.906 → 0.907	100.1%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	MMLU-Pro 0-shot (lm-evaluation-harness mmlu_pro_chat, chat completions; mean of 3 seeds)	0.883 → 0.883	100.1%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	GSM8K-Platinum 0-shot (lm-evaluation-harness gsm8k_platinum_cot_llama, chat completions; mean of 3 seeds)	0.955 → 0.954	99.8%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	LiveCodeBench v6 code generation 0-shot (lighteval via litellm; mean of 3 seeds)	0.838 → 0.836	99.8%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	MATH-500 0-shot (lighteval via litellm; mean of 3 seeds)	0.849 → 0.845	99.5%	source ↗
FP8 W8A8 dynamic (E4M3; weights static per-channel, activations dynamic per-token; llm-compressor FP8_DYNAMIC, compressed-tensors); lm_head, MoE router/shared-expert gates, linear-attention (Gated DeltaNet), vision-encoder and MTP layers unquantized	GPQA Diamond 0-shot (lighteval via litellm; mean of 3 seeds)	0.897 → 0.887	98.9%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json