whatmodel.sh / Qwen2.5 14B Instruct

Qwen2.5 14B Instruct

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Qwen2.5 14B Instruct is an open-weights language model from Alibaba, released September 2024. Its published size is 14.8B parameters and a 131k-token context window. Licence: apache-2.0.

51.5±0.8#319 of 719 LLMsfrom 36 benchmark results across 13 sources

$ how it compares

The LLMs ranked either side of Qwen2.5 14B Instruct, out of 828 with a score, then more from Alibaba.

#modelscorewhereRAMreleased
385Gemma 3 12B Google52±1api $0.075/M +local~9.1GB2025-03
386Qwen2.5 Coder 7B Instruct Alibaba52±1local~6GB2024-09
387Qwen2.5 14B Instruct Alibaba52±1local~10GB2024-09
388Cogito v1 Preview Llama 8B Deep Cogito51±5local~6.2GB2025-03
389Hermes 3 Llama 3.1 405B Nous Research51±1local~114GB2024-08
more from Alibaba
18Qwen3.8 Max (2.4T-A95B) Alibaba78±1api $3.00/M +local~380GB2026-08
34Qwen3.8 Max (0902) Alibaba76±1api $3.00/M2026-09
37Qwen3.8-Flash-Next Alibaba75±1local~59GB2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #215 of 300 * · 6.5% of its score · 2 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 (Jan '25) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Granite 4.1 3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
τ²-bench Telecom17.5% ✎▁·▅█▆▅▂▃▃▂▃▁ beats 9/89very hard · avg 13%▇▇ 5.2%sharp · small run
BFCL v358.7% ✎▁▁▁▁▁▃▅█▅▄▁▁ beats 78/114medium · avg 52%▇ 1.3%blunt · at its level
coding▁▁▁▃▆██▇▆▄▃▁ #238 of 458 · 17% of its score · 7 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails QwQ-32B-Preview, InternLM2.5 20B Chat, Gemma 2 27B IT here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HumanEval+79.1% ✎▁·▁▁▂▃▄▄█▅▄▃ beats 106/145easy · avg 74%▇▇▇ 6.5%sharp
HumanEval72.1% ✎▁▁▁▁▁▃▅▅█▃▂▁ beats 131/190easy · avg 74%▇▇ 3.9%sharp · small run
CodeReviewQA: change localisation (hard), invariant accuracy57.1% ✎█▂▁▁▂▂▂▃▅▅▅▇ beats 39/47medium · avg 36%▇ 2.0%sharp · at its level · noisy
HumanEval (simple-evals)72.1% ✎▃▅▆█▅▃··▂··▂ beats 0/19easy · avg 78%▇ 1.9%typical
CodeReviewQA: solution identification (hard), invariant accuracy81.9% ✎▃··▁▂▂▃▂█▃▄▂ beats 44/47medium · avg 54%▇ 1.6%sharp · at its level · noisy
CodeReviewQA: automated code refinement, exact match rate36.8% ✎▁▁▁▁▁▁▂▂▃█▅▄ beats 36/47hard · avg 33%▇ 1.3%at its level
CodeReviewQA: change type recognition, invariant accuracy73.3% ✎▁▁▁······▁▃█ beats 26/47easy · avg 67%▇ 0.0%blunt · at its level · noisy
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #248 of 393 · 9.2% of its score · 4 tests

Beats Mistral Small 4 (119B-A6.5B), Ministral 3 14B Reasoning 2512, Qwen2.5 Coder 32B Instruct here, though they rank higher overall. Trails Flan-T5 XXL, Granite 4.0 H 1B, Granite 4.0 1B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AlpacaEval 2 length-controlled34.6% ✎▃▃·▁▁█▇▅·▁·▁ beats 11/22medium · avg 42%▇▇ 4.2%at its level
Arena-Hard (version/judge unspecified)70.2% ✎▃▃▂▅▃▆▅█▅▅▂▃ beats 20/30easy · avg 66%▇ 3.1%at its level
IFEval prompt-level loose82.4% ✎▁▃▁▃▁▄▁▁█▃▃▁ beats 13/22easy · avg 80%▇ 1.1%typical
IFEval (aggregation unspecified)78.7% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 48/141easy · avg 77%▇ 0.8%typical
math▁▂▃▅▆▅▄█▄▁▁▁ #236 of 398 · 12% of its score · 4 tests

Beats Command A (03-2025), GPT-4o (2024-11-20), GPT-4o (2024-05-13) here, though they rank higher overall. Trails Tulu 2 DPO 70B, OpenThinker 7B, Olmo 3 7B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MATH (subset unspecified) +170.6% ✎▃▃█▂▂▆·····▁ beats 17/22easy · avg 73%▇▇ 4.9%small run · several runs
MGSM79.6% ✎▃·▁·▁··▂▄█▄▄ beats 24/26easy · avg 72%▇ 2.9%at its level · small run
GSM8K83.9% ✎▂▃··▃▂▅█▆▃▂▃ beats 12/22very easy · avg 88%▇ 2.6%sharp
MGSM (simple-evals)79.6% ✎▁·▃▁·▁▇▃█▁▃▁ beats 1/20easy · avg 82%▇ 1.6%blunt
other tests17 not in a task · 55% of its score
testresultvs the others in its rundifficultycountswhy
MMLU +181.1% ✎▁▂·▁·▁▁█·▃▃▅ beats 15/22easy · avg 77%▇▇▇▇ 10%at its level · several runs
BIG-Bench Hard78.4% ✎▁▄·▁▁·▅█·▄▄▁ beats 16/22easy · avg 75%▇▇▇ 7.4%sharp · at its level
GPQA Diamond (simple-evals)42.9% ✎▁·····▄█▂▂▃▃ beats 4/22hard · avg 41%▇▇ 5.2%sharp
MMLU-Pro63.2% ✎▁▁·▁▁▁▃▂▄▅█▁ beats 67/188medium · avg 59%▇▇ 4.2%at its level
CPTUB average (SpeakLeash, 0-5 judge scale)70.9% ✎▁▂▂▂▃█▃▆▅▄▂▃ beats 56/95easy · avg 66%▇▇ 3.9%at its level
EvalPerf correctness (118 tasks, 20240328)82.5% ✎▂·▂▂▂▆▅▆▆▆▃█ beats 20/29easy · avg 76%▇ 3.5%typical
GPQA (subset unspecified)42.9% ✎▂▁▂▂▄▃▆▃▅█▄▃ beats 45/104hard · avg 45%▇ 3.2%sharp · at its level · small run
Polish EQ-Bench v2 (SpeakLeash)69.6% ✎▁··▁▁▂▂▆▅▆█▆ beats 52/70medium · avg 64%▇ 3.1%at its level
9 more tests
testresultvs the others in its rundifficultycountswhy
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)36.9% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 35/66hard · avg 31%▇ 2.9%at its level
Open Persian LLM Leaderboard v2 average (21 tasks)55.6% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 45/81medium · avg 58%▇ 2.5%blunt · at its level
MMLU (simple-evals, zero-shot CoT)79.9% ✎▁···▃▁▅█▂▂▃▁ beats 1/22easy · avg 80%▇ 2.1%typical
SimpleQA5.4% ✎▄▄▄▆█▆███·▂█ beats 5/32very hard · avg 5.3%▇ 2.0%too hard for it · small run
PopQA21.1% ✎▁▁▁▃▃▃▁█▁▃·▄ beats 7/21hard · avg 32%▇ 1.7%blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)79.0% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 249/276medium · avg 73%▇ 1.2%at its level · noisy
TruthfulQA MC270.8% ✎▃▃▂·▃▃·▅█▂▅▂ beats 20/21medium · avg 63%▇ 1.2%blunt · at its level
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)95.8% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 81/140very easy · avg 94%▇ 0.3%blunt · maxed out · noisy
DROP F1, 3-shot (simple-evals)85.5% ✎▅▂▆█▆·▃·▂▅·▂ beats 17/23easy · avg 80%▇ 0.3%blunt

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Qwen2.5 14B Instruct · needs ~9 GB · from huggingface.co/Qwen (official)
    chat in the terminal
    llama cli -hf Qwen/Qwen2.5-14B-Instruct-GGUF:q4_k_m
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf Qwen/Qwen2.5-14B-Instruct-GGUF:q4_k_m

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Qwen2.5 14B Instruct · needs ~9 GB · from huggingface.co/Qwen (official)

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-GGUF@q4_k_m -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Qwen2.5 14B Instruct · needs ~9.38 GB · from ollama.com
    chat in the terminal
    ollama run qwen2.5:14b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Qwen2.5 14B Instruct · needs ~8.3 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/Qwen2.5-14B-Instruct-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/Qwen2.5-14B-Instruct-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · 2 OLMo 2 Furious (Team OLMo,… 27%, Microsoft Research — Phi-4 Te… 26%, microsoft/phi-4 model card 18% and 10 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models8▇▇▇▇▇▇▇▇ 27%CC BY 4.0
Microsoft Research — Phi-4 Technical Report9▇▇▇▇▇▇▇▇ 26%CC BY 4.0
microsoft/phi-4 model card6▇▇▇▇▇ 18%MIT
Reinforcing Real-world Service Agents: InteractCS-RL (arXiv:2602.22697v1)1▇▇ 5%CC-BY-4.0
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)4▇▇ 5%CC-BY-4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 4%MIT
EvalPlus / EvalPerf published evaluation runs1▇ 3%Apache-2.0
SpeakLeash Polish EQ-Bench leaderboard1▇ 3%Apache-2.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)1▇ 3%Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 3%Apache-2.0
Berkeley Function Calling Leaderboard result archive1▇ 1%Apache-2.0
Open Portuguese LLM Leaderboard (archived)1▇ 1%Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 51.5
$ quantization evidence · 3 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; algorithm not stated, llm-compressor)GSM8K — published row 'GSM-8K (5-shot, strict-match)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 4096, add_bos_token)0.836 → 0.855102.4%source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; algorithm not stated, llm-compressor)OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 4096, add_bos_token)0.778 → 0.781100.4%source ↗
INT8 W8A8 (weights INT8 per-channel, activations INT8 per-token dynamic; algorithm not stated, llm-compressor)MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 4096, add_bos_token)0.799 → 0.79799.8%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json