whatmodel.sh / Phi-4 14B
Phi-4 14B
↗ share
+ contribute
Phi-4 14B is an open-weights language model from Microsoft, released December 2024. Its published size is 14.7B parameters and a 16k-token context window. Licence: mit.
$ how it compares
The LLMs ranked either side of Phi-4 14B, out of 828 with a score, then more from Microsoft.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 363 |  Gemini 1.5 Flash (002) | 53±1 | comparison only | ||
| 364 |  Gemma 3 27B | 53±1 | api $0.10/M +local | ||
| 365 | Phi-4 14B | 53±1 | api $0.087/M +local | ||
| 366 | Gemma 4 E4B | 53±1 | local | ||
| 367 |  Ministral 3 8B | 53±1 | api $0.15/M +local | ||
| more from Microsoft | |||||
| 252 | Phi-4-reasoning-plus 14B | 59±1 | local | ||
| 261 | Phi-4-reasoning 14B | 58±1 | api $0.22/M +local | ||
| 422 | WizardLM-2 8x22B | 50±1 | local | ||
$ what it's good at
agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #264 of 300 * · 2.6% of its score · 2 tests
Beats Magistral Small 1.0 (2506), Llama 4 Scout (109B-A17B) here, though they rank higher overall. Trails Qwen2.5 1.5B Instruct, Granite 4.0 H 1B, Granite 4.0 1B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BFCL V4 (December 2025 archive) | 28.8% ✎ | ▁▁▁▂██▅▅▆▂▂▂ beats 28/77 | 1.8% |
| BFCL v3 | 47.0% ✎ | ▁▁▁▁▁▃▅█▅▄▁▁ beats 28/114 | 0.9% |
coding▁▁▁▃▆██▇▆▄▃▁ #225 of 458 · 19% of its score · 7 tests
Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B) here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, GPT-4 (0613) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| HumanEval+ lab +2 | 82.8% ✎ | ▁·▁▁▂▃▄▄█▅▄▃ beats 116/145 | 13% |
| HumanEval lab | 82.6% ✎ | ▁▁▁▁▁▃▅▅█▃▂▁ beats 168/190 | 2.4% |
| CodeReviewQA: change localisation (hard), invariant accuracy | 44.8% ✎ | █▂▁▁▂▂▂▃▅▅▅▇ beats 34/47 | 1.4% |
| HumanEval (simple-evals) lab | 82.6% ✎ | ▂▅▆█▅▃··▂··▂ beats 3/19 | 1.2% |
| CodeReviewQA: solution identification (hard), invariant accuracy | 77.5% ✎ | ▃··▁▂▂▃▂█▃▄▂ beats 42/47 | 1.0% |
| CodeReviewQA: automated code refinement, exact match rate | 37.1% ✎ | ▁▁▁▁▁▁▂▂▃█▅▄ beats 37/47 | 0.9% |
| CodeReviewQA: change type recognition, invariant accuracy | 76.6% ✎ | ▁▁▁······▁▃█ beats 35/47 | 0.0% |
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #268 of 393 · 12% of its score · 4 tests
Beats Ministral 3 14B Reasoning 2512, Ministral 3 14B here, though they rank higher overall. Trails Flan-T5 XXL, LFM2 1.2B, Granite 4.0 H 1B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena-Hard (version/judge unspecified) lab +2 | 75.4% ✎ | ▃▃▂▅▃▆▅█▅▅▂▃ beats 23/30 | 6.3% |
| LMArena (text) | 1256.3 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 94/379 | 3.8% |
| IFEval (aggregation unspecified) lab +1 | 63.0% ✎ | ▁▁▃▃▂▄▄▆█▃▂▁ beats 24/141 | 1.3% |
| IFEval strict (prompt/instruction level unspecified) lab | 62.3% ✎ | █·········██ beats 0/2 | 1.0% |
math▁▂▃▅▆▅▄█▄▁▁▁ #219 of 398 · 16% of its score · 5 tests
Beats GPT-4o (2024-08-06), Command A (03-2025), GPT-4o (2024-11-20) here, though they rank higher overall. Trails Olmo 3 7B Instruct, DeepSeek R1 Distill Llama 8B, Phi-4-mini-reasoning 3.8B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH level 5 | 64.9% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 56/92 | 10% |
| OTIS mock AIME 2024–2025 | 13.8% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 52/193 | 1.9% |
| MGSM lab | 80.6% ✎ | ▃·▁·▁··▂▄█▄▄ beats 25/26 | 1.8% |
| MATH (subset unspecified) lab | 80.4% ✎ | ▂▄▄▃█▇▅▆▄▃▄▁ beats 54/80 | 1.7% |
| MGSM (simple-evals) lab | 80.6% ✎ | ▁·▁▃·▁▇▃█▁▃▁ beats 2/20 | 1.0% |
other tests24 not in a task · 49% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MMLU-Pro lab +2 | 70.4% ✎ | ▁▁·▁▁▁▃▂▄▅█▁ beats 92/188 | 7.4% |
| GPQA Diamond | 56.1% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 81/208 | 5.5% |
| CLIcK Korean (Daekeun Kim) | 59.2% ✎ | ▆▄▄▄█▄▂·▄█·▄ beats 5/23 | 5.4% |
| HAE-RAE 1.0 (Daekeun Kim) | 59.2% ✎ | ▄▄▄▂▂█▂▄▂█▂▄ beats 4/22 | 4.6% |
| GPQA Diamond (simple-evals) lab | 56.1% ✎ | ▁·····▄█▃▂▃▃ beats 13/22 | 3.5% |
| FlenQA (3K-token subset) lab +1 | 82.0% ✎ | ▄···▄····▄·█ beats 0/4 | 3.3% |
| CPTUB average (SpeakLeash, 0-5 judge scale) | 66.1% ✎ | ▁▂▂▂▃█▃▆▅▄▂▃ beats 46/95 | 2.4% |
| GPQA (subset unspecified) lab | 56.1% ✎ | ▂▁▂▂▄▃▆▃▅█▄▃ beats 59/104 | 2.1% |
16 more tests
$ where to run it
Through an API
api · DeepInfra · $0.088/M · $0.07 in / $0.14 out per 1M tokens · 16k context · deepinfra.com ↗ · price sourceUSD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: microsoft/phi-4. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD.
llama.cpp · recommended
version
- Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.macOS / Linux
curl -LsSf https://llama.app/install.sh | shWindowsirm https://llama.app/install.ps1 | iex - Run Phi-4 14B · needs ~9.1 GB · from huggingface.co/lmstudio-communitychat in the terminal
llama cli -hf lmstudio-community/phi-4-GGUF:Q4_K_M - Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/phi-4-GGUF:Q4_K_M
LM Studio
version
- Install LM Studio · Open the app once before using the lms command.
- Run Phi-4 14B · needs ~9.1 GB · from huggingface.co/lmstudio-community
- Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/phi-4-GGUF@Q4_K_M -y - Then run lms ls to see its name, and lms chat followed by that name
Ollama
version
- Install Ollama
download it (macOS / Windows) ↗
Linuxcurl -fsSL https://ollama.com/install.sh | sh - Run Phi-4 14B · needs ~9.32 GB · from ollama.comchat in the terminal
ollama run phi4:14b
MLX (Mac)
version
- Install MLX (Mac) · Apple silicon Macs only, text only.with uv
uv tool install mlx-lmorpip install mlx-lm - Run Phi-4 14B · needs ~8.2 GB, Mac only · from huggingface.co/mlx-communitychat in the terminal
mlx_lm.chat --model mlx-community/phi-4-4bit - Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/phi-4-4bit
Official weights and documentation: huggingface.co/microsoft ↗
$ how the score was worked out · Epoch AI — internal evaluatio… 17%, Microsoft Research — Phi-4 Te… 16%, Microsoft — Phi-4-reasoning T… 13% and 14 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
not counted (1)
| Microsoft Research — Phi-4 Technical Report | LiveBench (release not stated) [correct] | 47.6% | fewer than three models have this test yet, or this source ran it for this model alone |
$ quantization evidence · 9 measurements
How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.
| precision | test | full → quantized | kept | |
|---|---|---|---|---|
| FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token) | GSM8K 5-shot strict-match (lm-evaluation-harness `openllm`, vLLM) | 0.901 → 0.907 | 100.7% | source ↗ |
| FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token) | MMLU 5-shot (lm-evaluation-harness `openllm`, vLLM) | 0.803 → 0.803 | 100.0% | source ↗ |
| FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token) | OpenLLM v1 average of 6 tasks (ARC-Challenge 25-shot, GSM8K 5-shot, HellaSwag 10-shot, MMLU 5-shot, TruthfulQA MC2 0-shot, Winogrande 5-shot; lm-evaluation-harness `openllm`, vLLM) | 0.765 → 0.765 | 99.9% | source ↗ |
| INT4 W4A16 (weights only, symmetric per-group, group size 128; GPTQ) | GSM8K 5-shot strict-match (lm-evaluation-harness `openllm`, vLLM) | 0.901 → 0.897 | 99.6% | source ↗ |
| INT4 W4A16 (weights only, symmetric per-group, group size 128; GPTQ) | MMLU 5-shot (lm-evaluation-harness `openllm`, vLLM) | 0.803 → 0.799 | 99.5% | source ↗ |
| INT4 W4A16 (weights only, symmetric per-group, group size 128; GPTQ) | OpenLLM v1 average of 6 tasks (ARC-Challenge 25-shot, GSM8K 5-shot, HellaSwag 10-shot, MMLU 5-shot, TruthfulQA MC2 0-shot, Winogrande 5-shot; lm-evaluation-harness `openllm`, vLLM) | 0.765 → 0.76 | 99.3% | source ↗ |
| INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ) | GSM8K 5-shot strict-match (lm-evaluation-harness `openllm`, vLLM) | 0.901 → 0.903 | 100.3% | source ↗ |
| INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ) | MMLU 5-shot (lm-evaluation-harness `openllm`, vLLM) | 0.803 → 0.804 | 100.1% | source ↗ |
| INT8 W8A8 (weights static per-channel, activations dynamic per-token; SmoothQuant + GPTQ) | OpenLLM v1 average of 6 tasks (ARC-Challenge 25-shot, GSM8K 5-shot, HellaSwag 10-shot, MMLU 5-shot, TruthfulQA MC2 0-shot, Winogrande 5-shot; lm-evaluation-harness `openllm`, vLLM) | 0.765 → 0.763 | 99.8% | source ↗ |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json