whatmodel.sh / Qwen3 1.7B

Qwen3 1.7B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Qwen3 1.7B is an open-weights language model from Alibaba, released April 2025. Its published size is 2.03B parameters and a 33k-token context window. Licence: apache-2.0.

43.7±0.6#469 of 719 LLMsfrom 72 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of Qwen3 1.7B, out of 828 with a score, then more from Alibaba.

#modelscorewhereRAMreleased
556Falcon3 10B Instruct TII44±3local~7.6GB2024-12
557Code Tulu 2 34B Allen Institute for AI44±2local~72GB2023-11
558Qwen3 1.7B Alibaba44±1local~2.6GB2025-04
559Phi-4-mini 3.8B Microsoft44±1api $0.13/M +local~3.8GB2025-02
560Ministral 8B (2410) Mistral44±1local~6.2GB2024-10
more from Alibaba
18Qwen3.8 Max (2.4T-A95B) Alibaba78±1api $3.00/M +local~380GB2026-08
34Qwen3.8 Max (0902) Alibaba76±1api $3.00/M2026-09
37Qwen3.8-Flash-Next Alibaba75±1local~59GB2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #234 of 300 · 4.5% of its score · 4 tests

Beats o3-mini, Ministral 3 14B Reasoning 2512, DeepSeek R1 0528 Qwen3 8B here, though they rank higher overall. Trails Granite 4.0 1B, SmolLM3 3B, Phi-3.5-mini-instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BFCL v3 +254.7% ✎▁▁▁▁▁▃▅█▅▄▁▁ beats 56/114medium · avg 59%▇ 2.2%blunt · at its level · several runs
BFCL V4 (December 2025 archive)28.4% ✎▁▁▁▂██▅▅▆▂▂▂ beats 27/77hard · avg 32%▇ 1.1%typical
EEGAgentBench overall (tool-based EEG analysis agents, 6 tasks)23.3% ✎▃·▁▁··▇▇█▂·▁ beats 1/25hard · avg 29%▇ 0.6%blunt
BFCL (version not stated)88.8% ✎▄▆▄▂·█▆·▄··▂ beats 15/17very easy · avg 86%▇ 0.6%blunt · small run
coding▁▁▁▃▆██▇▆▄▃▁ #359 of 458 * · 2.7% of its score · 2 tests

Beats LFM2.5 8B-A1B, Qwen3-VL 2B Thinking, DeepSeek R1 Distill Qwen 7B here, though they rank higher overall. Trails Qwen2.5 1.5B Instruct, Phi-3-mini-128k-instruct (Apr 2024), Code Tulu 2 13B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench v6 (window unspecified) lab31.3% ✎▃▅█▅▅▃▅···▃█ beats 2/16medium · avg 35%▇ 1.8%typical
LiveCodeBench v434.4% ✎█·█········█ beats 1/2medium · avg 47%▇ 0.9%at its level · small run
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #272 of 393 · 3.1% of its score · 2 tests

Beats Ministral 3 14B Reasoning 2512, Ministral 3 14B, Gemma 4 E4B here, though they rank higher overall. Trails Flan-T5 XXL, EXAONE 3.5 2.4B Instruct, LFM2 1.2B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
IFEval (aggregation unspecified) +274.0% ✎▄····█▄···▄█ beats 5/6easy · avg 82%▇ 2.1%several runs
IFEval mean of four measures71.7% ✎█·█··█·██·██ beats 2/6very easy · avg 87%▇ 1.1%typical
math▁▂▃▅▆▅▄█▄▁▁▁ #264 of 398 · 27% of its score · 11 tests

Beats Molmo 2 8B, Llama 3.3 70B Instruct, DeepSeek-V2.5 here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), InternVL2-4B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
GSM8K +277.9% ✎█····█··▄·▄█ beats 0/7very easy · avg 86%▇▇▇ 8.5%sharp · several runs
AIME 2025 lab +236.8% ✎▃·▃▃█··▃▅▃·█ beats 1/12medium · avg 53%▇▇ 6.1%sharp · at its level · small run · several runs
MGSM66.6% ✎█···█▄··▄··▄ beats 6/6easy · avg 77%▇ 2.9%at its level
MATH (subset unspecified)70.6% ✎█▄··▄█▄····▄ beats 1/7easy · avg 81%▇ 2.5%at its level
MGSM (Marco evaluation)63.6% ✎▄·▄·▄▄▄·▄··█ beats 2/7easy · avg 70%▇ 1.4%at its level
HMMT February 2025 lab10.2% ✎▃▁▁▃▁▅▅▃██▁▄ beats 0/31medium · avg 36%▇ 1.3%small run
PolyMath (Marco evaluation)23.4% ✎▄··▄██··▄··▄ beats 1/7hard · avg 30%▇ 1.1%typical
MATH-50081.9% ✎▄··█▄·▄···▄▄ beats 5/6easy · avg 80%▇ 0.9%at its level
3 more tests
testresultvs the others in its rundifficultycountswhy
HMMT November 2025 lab8.9% ✎▃▁····▁▅▃█▇▁ beats 0/23medium · avg 41%▇ 0.9%at its level · small run
PolyMATH lab25.2% ✎▁···▃▁·▇█▄▄▄ beats 1/23hard · avg 29%▇ 0.8%small run
OTIS mock AIME 2024–20258.1% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 47/193very hard · avg 8.5%▇ 0.3%sharp · too hard for it · noisy
other tests32 not in a task · 63% of its score
testresultvs the others in its rundifficultycountswhy
MMLU-Pro lab +458.1% ✎▄·▄·▄▄▄▄█▄██ beats 1/12medium · avg 63%▇▇▇▇▇ 11%at its level · several runs
GPQA Diamond +338.0% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 34/208medium · avg 53%▇▇▇ 8.5%sharp · small run · several runs
MMLU +162.4% ✎▄···▄▄·▄█··█ beats 0/7easy · avg 75%▇▇▇ 7.1%at its level · several runs
GPQA (subset unspecified) lab +227.9% ✎▃····▂·▂▂█▆█ beats 0/14hard · avg 49%▇▇▇ 6.5%sharp · several runs
IFBench (aggregation unspecified) +225.9% ✎▄·█·█······█ beats 4/6medium · avg 40%▇ 3.6%at its level · several runs
MMMLU +146.5% ✎▄···▄·▄▄··▄█ beats 5/6medium · avg 59%▇ 3.0%at its level · several runs
MMLU-ProX-Lite (Marco)28.6% ✎█··▄··█··█·▄ beats 1/7medium · avg 43%▇ 1.9%at its level
C-Eval lab68.1% ✎▁···▁▁▂▁▃▂▃█ beats 10/33easy · avg 75%▇ 1.5%at its level · small run
24 more tests
testresultvs the others in its rundifficultycountswhy
INCLUDE lab51.8% ✎▂····▂▂▄▄▄█▄ beats 7/36medium · avg 59%▇ 1.5%at its level · small run
GlobalMMLU (Marco evaluation)46.3% ✎▃··▃▃··▅···█ beats 1/7medium · avg 57%▇ 1.3%at its level
BRIDGE clinical text, zero-shot average (87 tasks)26.3% ✎▁▁▁▁▁▂▃▃▄▆█▃ beats 22/97hard · avg 32%▇ 1.3%blunt · at its level
MMLU-Redux lab73.9% ✎▁▁▃▃▂▃▄▅▇▅█▄ beats 6/63easy · avg 80%▇ 1.3%at its level · small run
Open Persian LLM Leaderboard v2 average (21 tasks)41.3% ✎▂▄▃▃█▄▃▆▃▅▂▂ beats 15/81medium · avg 58%▇ 1.3%blunt · at its level
MMMLU (Marco evaluation)49.0% ✎▄···▄·█·▄·▄█ beats 1/7medium · avg 60%▇ 1.2%at its level
Global PIQA lab63.1% ✎▃▃··▃··▃▃▅█▅ beats 1/11medium · avg 70%▇ 1.0%at its level · small run
MultiChallenge lab27.2% ✎▁·▃·▁▃▆▅▃▁█▅ beats 1/29hard · avg 34%▇ 1.0%small run
AGIEval (Marco suite unspecified)39.6% ✎▄··▄█▄··▄··█ beats 0/7medium · avg 47%▇ 1.0%blunt · at its level
MAXIFE lab50.7% ✎▂·▃·▂···▆·▄█ beats 1/11medium · avg 63%▇ 1.0%at its level · small run
SuperGPQA lab31.2% ✎▂▁▃▂▅▆▃▇▇▇█▃ beats 4/46medium · avg 39%▇ 0.9%at its level · small run
MMLU-ProX lab49.4% ✎▁·▃▁▁▂▁▃▂█▅▄ beats 1/35medium · avg 57%▇ 0.9%at its level · small run
Global-MMLU62.3% ✎▃▅▃·▃···▅·▅█ beats 2/11medium · avg 70%▇ 0.9%at its level · small run
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items18.3% ✎▁▁▃▁▃▄█▆▄▅▂▃ beats 3/39very hard · avg 22%▇ 0.9%blunt
Open PT-LLM Leaderboard average (9 Portuguese tasks)68.1% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 148/276medium · avg 73%▇ 0.7%at its level · noisy
MixEval-Hard33.9% ✎█····█·····█ beats 1/2medium · avg 40%▇ 0.6%at its level · small run
NOVA-63 lab40.3% ✎▃▃·▃···▃█▃▃█ beats 0/11medium · avg 46%▇ 0.6%blunt · at its level · small run
LongBench v2 lab26.5% ✎▃·▁·▂▂▁▃▄▁█▄ beats 1/28very hard · avg 35%▇ 0.6%too hard for it · noisy · small run
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy46.3% ✎▃·▁▁▂▁▃▅█▇▄▁ beats 24/100medium · avg 68%▇ 0.5%at its level · noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy68.4% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 36/146easy · avg 83%▇ 0.4%at its level · noisy
GSM-Plus79.4% ✎▁··▁▁▁·▃▁▄█▁ beats 4/16easy · avg 83%▇ 0.4%blunt · small run
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)95.6% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 78/140very easy · avg 94%▇ 0.2%blunt · noisy
WMT24++ lab39.3% ✎▁·▃▁·▁·▁▂▆█▃ beats 1/27easy · avg 66%▇ 0.1%blunt · at its level · small run
Chess Puzzles (Epoch)0.0% ✎█▁▁▁▁▂▁▄▂▂▁▁ beats 33/135very hard · avg 5.9%▇ 0.0%sharp · too hard for it · noisy

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Qwen3 1.7B · needs ~1.3 GB · from huggingface.co/ggml-org
    chat in the terminal
    llama cli -hf ggml-org/Qwen3-1.7B-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf ggml-org/Qwen3-1.7B-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Qwen3 1.7B · needs ~1.3 GB · from huggingface.co/ggml-org

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/ggml-org/Qwen3-1.7B-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Qwen3 1.7B · needs ~1.72 GB · from ollama.com
    chat in the terminal
    ollama run qwen3:1.7b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Qwen3 1.7B · needs ~1 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/Qwen3-1.7B-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/Qwen3-1.7B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Marco-MoE: Open Multilingual… 24%, Qwen/Qwen3.5-2B model card 19%, Qwen (Alibaba) vendor-reporte… 13% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling11▇▇▇▇▇▇▇ 24%CC BY 4.0
Qwen/Qwen3.5-2B model card · lab-reported19▇▇▇▇▇▇ 19%apache-2.0
Qwen (Alibaba) vendor-reported benchmark scores for its own models (qwen.ai launch posts, Qwen2.5-VL and Qwen3-VL technical reports, Qwen3.8-Flash-Next card) · lab-reported5▇▇▇▇ 13%Factual vendor-reported score (not licensed content)
LiquidAI/LFM2-1.2B model card7▇▇▇▇ 13%LFM Open License v1.0
HuggingFaceTB/SmolLM3-3B model card (Evaluation section)8▇▇▇ 10%Apache-2.0
LiquidAI/LFM2.5-1.2B-Thinking model card8▇▇▇ 9%LFM Open License v1.0
Epoch AI — internal evaluations3▇ 2%CC BY 4.0
Qwen/Qwen3.5-0.8B model card · lab-reported1▇ 2%apache-2.0
Berkeley Function Calling Leaderboard result archive2▇ 2%Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 1%MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 1%Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 1%MIT
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)1▇ 1%CC BY 4.0
Open Portuguese LLM Leaderboard (archived)1▇ 1%Apache-2.0
EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis (arXiv:2609.31632v1)1▇ 1%CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 43.7
$ quantization evidence · 33 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.40155.0%source ↗
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.30254.4%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.71898.5%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.53195.8%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.71598.1%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.52394.4%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.72198.9%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.52995.5%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.384 → 0.389101.3%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)AIME 2025 — published row 'AIME 2025' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.323 → 0.323100.0%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.498 → 0.49198.6%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.568 → 0.5698.6%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)OpenLLM v2 average of 6 tasks (MMLU-Pro (5-shot); IFEval (0-shot); BBH (3-shot); Math-lvl-5 (4-shot); GPQA (0-shot); MuSR (0-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.243 → 0.23395.8%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)GSM8K — published row 'GSM-8K (5-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.437 → 0.41595.0%source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)AIME 2024 — published row 'AIME 2024' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.44 → 0.40191.2%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.6386.5%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.44981.0%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.729 → 0.72599.5%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.554 → 0.54798.8%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)MMLU-Redux, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.739 → 0.749101.4%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)LiveBench 2024-11-25, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.356 → 0.35599.7%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)GPQA (subset not stated in card), non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.286 → 0.28399.0%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)MMLU-Redux, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.644 → 0.6397.8%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)GPQA (subset not stated in card), thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.401 → 0.39197.5%source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-1.7B-GPTQ-Int8)LiveBench 2024-11-25, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)0.511 → 0.49897.5%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.568 → 0.55197.0%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.498 → 0.46894.0%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)OpenLLM v2 average of 6 tasks (MMLU-Pro (5-shot); IFEval (0-shot); BBH (3-shot); Math-lvl-5 (4-shot); GPQA (0-shot); MuSR (0-shot)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 16384, chat template, few-shot as multiturn); per-task scale (raw vs Open LLM Leaderboard-normalized) not stated in card0.243 → 0.22492.1%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.384 → 0.34890.8%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)AIME 2025 — published row 'AIME 2025' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.323 → 0.28287.4%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)LiveCodeBench — published row 'LiveCodeBench' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.334 → 0.26478.9%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)AIME 2024 — published row 'AIME 2024' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 327680.44 → 0.32173.0%source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)GSM8K — published row 'GSM-8K (5-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)0.437 → 0.30670.1%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json