whatmodel.sh / DeepSeek R1 (Jan '25)

DeepSeek R1 (Jan '25)

+ contribute

DeepSeek R1 (Jan '25) is an open-weights language model from DeepSeek, released January 2025. Its published size is 685B parameters (37B active per token) and a 128k-token context window. Licence: mit.

63.2±0.5#164 of 719 LLMsfrom 64 benchmark results across 21 sources

$ how it compares

The LLMs ranked either side of DeepSeek R1 (Jan '25), out of 828 with a score, then more from DeepSeek.

#	model	score	where	RAM	released
187	Grok 3 Beta SpaceXAI (xAI)	63±1	comparison only		2025-04
188	Qwen3-Next 80B-A3B Thinking Alibaba	63±1	api $0.41/M +local	~27GB	2025-09
189	DeepSeek R1 (Jan '25) DeepSeek	63±1	api $4.00/M +local	~110GB	2025-01
190	Gemini 2.5 Flash Preview (09-2025) Google	63±1	comparison only		2025-09
191	Qwen3-Next 80B-A3B Instruct Alibaba	63±1	api $0.34/M +local	~27GB	2025-09
	more from DeepSeek
21	DeepSeek V4.1 Flash DeepSeek	78±1	api $0.26/M +local	~537GB	2026-09
31	DeepSeek V4 Pro (0813) DeepSeek	76±1	api $1.63/M +local	~1010GB	2026-08
35	DeepSeek V3.2 Speciale DeepSeek	76±3	comparison only

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #218 of 300 · 4.1% of its score · 3 tests

Beats o1 (2024-12-17), o3-mini here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
SWE-bench Verified lab +1	49.2% ✎	▁▁▁▁▁▁▂▄█▆▁▁ beats 39/131	hard · avg 19%	▇ 3.0%	sharp · at its level · small run · several runs
Multi-SWE-bench, 7 languages (MagentLess scaffold)	5.9% ✎	▄···▄▄·▄▄·▄█ beats 5/7	very hard · avg 2.6%	▇ 0.6%	blunt · too hard for it
BFCL v3	56.9% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 67/114	medium · avg 52%	▇ 0.5%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #137 of 458 · 26% of its score · 10 tests

Beats Qwen3.5 Plus, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall. Trails Devstral Small 2 24B, Llama 3.1 Nemotron Ultra 253B v1, Nemotron 3.5 Lightning 30B-A3B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) +2	56.9% ✎	▂·▂▃▅▂▃█▆▄▂▂ beats 32/49	very hard · avg 8.5%	▇▇▇▇▇ 13%	sharp · at its level · several runs
SWE-bench Verified lab +1	49.2% ✎	▁▁▁▁▁▁▂▄█▆▁▁ beats 39/131	hard · avg 19%	▇ 3.0%	sharp · at its level · small run · several runs
SciCode main problems (official leaderboard)	4.6% ✎	▃···█··▂·▃·▂ beats 15/17	very hard · avg 1.1%	▇ 2.8%	sharp · too hard for it
LiveCodeBench (version unspecified) lab	65.9% ✎	▄····▄▄··▄█▄ beats 6/6	medium · avg 41%	▇ 2.0%	at its level
WebApp1K-Duo-React pass@1 (onekq-ai)	68.7% ✎	▁··▁·▁▁▁▆█▇█ beats 28/45	hard · avg 32%	▇ 1.2%	at its level
HumanEval (Vector Institute, Inspect Evals)	95.7% ✎	▄·▄·█·█·█·▄▄ beats 7/9	easy · avg 79%	▇ 1.1%	maxed out
LiveCodeBench 2024-08–2025-05 (version unstated) lab	63.5% ✎	▃···█··▅··▃▃ beats 2/7	medium · avg 42%	▇ 0.7%	at its level · small run
LiveCodeBench v6 (Aug 2024–May 2025) lab	63.5% ✎	▂▅▅▃█▃▂····▂ beats 4/17	medium · avg 48%	▇ 0.7%	at its level · small run

2 more tests

test	result	vs the others in its run	difficulty	counts	why
Multi-SWE-bench, 7 languages (MagentLess scaffold)	5.9% ✎	▄···▄▄·▄▄·▄█ beats 5/7	very hard · avg 2.6%	▇ 0.6%	blunt · too hard for it
WebApp1K-React pass@1 (onekq-ai)	92.7% ✎	▃▁▁█▁▄▅▃▄▇▃▄ beats 28/31	medium · avg 44%	▇ 0.5%	sharp · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #112 of 393 · 12% of its score · 6 tests

Beats LongCat 2.0 (1.6T-A48B), Step 3.5 Flash (196B-A11B), GPT-5 mini here, though they rank higher overall. Trails Nemotron-Cascade-8B, Gemini 2.5 Flash, Mistral Large 3 (675B-A41B) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge)	58.0% ✎	▂·▃▅▅·▃█▆▂▂▂ beats 16/22	very hard · avg 12%	▇▇ 3.8%	sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judge) lab	92.3% ✎	▂▁▁▅█▅▃▃▃▅▃▃ beats 62/62	medium · avg 54%	▇ 2.7%	sharp · maxed out · small run
LMArena (text)	1398.1 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 242/379	avg 1268	▇ 2.4%	blunt
IFEval prompt-level strict +1	83.4% ✎	▄▄··▄·▄··█·▄ beats 3/6	easy · avg 74%	▇ 1.3%	several runs
AlpacaEval 2 length-controlled lab	87.6% ✎	▁▁▃▄▆██▆▃▃▁▃ beats 155/155	medium · avg 50%	▇ 1.1%	small run
Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges)	77.0% ✎	▆▂▂▆▂▂▃█▂▃·▂ beats 21/22	hard · avg 19%	▇ 0.7%	sharp · at its level · noisy

math▁▂▃▅▆▅▄█▄▁▁▁ #150 of 398 · 16% of its score · 7 tests

Beats MiMo-V2.5 (310B-A15B), MiniMax-M2.7, MiniMax-M2.1 here, though they rank higher overall. Trails Nemotron 3 Nano 30B-A3B, Qwen3-VL 8B Thinking, Olmo 3 32B Think here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AIME 2024 lab +3	79.8% ✎	▃·▃·▃▃···█·▃ beats 7/7	medium · avg 51%	▇ 3.7%	sharp · small run · several runs
AIME 2025 +2	70.4% ✎	█··▄··█··▄·▄ beats 3/6	medium · avg 41%	▇ 3.6%	sharp · small run · several runs
MATH level 5	93.1% ✎	▂▂▅▆▄█▆▃▅▃▃▄ beats 80/92	hard · avg 32%	▇ 2.8%	sharp
OTIS mock AIME 2024–2025	53.3% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 80/193	very hard · avg 8.5%	▇ 2.4%	sharp · at its level · noisy
MATH (Vector Institute, Inspect Evals)	92.7% ✎	▄··██▄··▄·▄█ beats 6/9	medium · avg 53%	▇ 2.4%	sharp
HMMT 2025 (edition unspecified) lab	41.7% ✎	▃█▇▅▇▅▁·▁··▁ beats 4/32	hard · avg 22%	▇ 1.0%	sharp · at its level · noisy · small run
MATH-500 lab	97.3% ✎	▄·▄▄·▄▄█···▄ beats 7/7	very easy · avg 91%	▇ 0.4%	maxed out

other tests23 not in a task · 45% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +4	71.7% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 109/208	hard · avg 40%	▇▇▇▇▇▇ 15%	sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzles)	78.7% ✎	▁▁▃▆██▁▁·▃▁▁ beats 55/57	hard · avg 18%	▇▇ 6.1%	sharp · at its level
GPQA Diamond (Vector Institute, Inspect Evals)	70.5% ✎	▃··▃▃▃▃·▃·▃█ beats 6/9	hard · avg 41%	▇▇ 3.9%	sharp · at its level
MMLU-Pro lab +1	84.0% ✎	▁▁·▁▁▁▃▂▄▅█▁ beats 162/188	medium · avg 60%	▇ 1.9%	small run · several runs
SimpleQA lab +1	30.1% ✎	▄▄▄▆█▆███·▂█ beats 27/32	very hard · avg 11%	▇ 1.9%	at its level · small run · several runs
MMLU-Redux lab +1	92.9% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 51/63	easy · avg 81%	▇ 1.7%	small run · several runs
MMLU-Pro (Vector Institute, Inspect Evals)	83.8% ✎	▃····▃▅·▃▅·█ beats 8/9	medium · avg 60%	▇ 1.7%	typical
EQ-Bench 3 (emotional intelligence, judge Elo)	1224.7 ✎	▁·▂▁▄▅▅▃█▄▁▁ beats 35/62	avg 1044	▇ 1.5%	typical

15 more tests

test	result	vs the others in its run	difficulty	counts	why
BRIDGE clinical text, zero-shot average (87 tasks)	44.3% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 92/97	hard · avg 32%	▇ 1.4%	blunt · at its level
DROP lab	92.2% ✎	▅··▅·█▅█▅▃▃▅ beats 17/17	easy · avg 73%	▇ 1.2%	small run
CPTUB average (SpeakLeash, 0-5 judge scale)	82.8% ✎	▁▂▂▂▃█▃▆▅▄▃▃ beats 86/95	easy · avg 66%	▇ 1.1%	typical
MMLU lab	90.8% ✎	▁▁▁▁▁▂▃▅█▃▆▃ beats 181/183	easy · avg 76%	▇ 1.1%	small run
FRAMES lab +1	82.5% ✎	█··········█ beats 0/1	medium · avg 64%	▇ 1.0%	small run · several runs
C-Eval lab	91.8% ✎	▁···▁▁▂▁▂▂▃█ beats 29/33	easy · avg 75%	▇ 0.9%	small run
CNMO 2024 lab +1	78.8% ✎	█··········█ beats 0/1	medium · avg 58%	▇ 0.9%	small run · several runs
Humanity's Last Exam lab	8.5% ✎	▁▁▁▃▆▆▇██▆▁▃ beats 12/104	very hard · avg 4.2%	▇ 0.9%	small run
LongSocialBench (full-context, 4-option MCQ over discussion threads)	47.3% ✎	▆▂·▄█·▂▃·▄▄▂ beats 11/16	very hard · avg 36%	▇ 0.8%	blunt · at its level
Judgemark v2 (LLM-as-judge for creative writing)	77.0% ✎	▃▃··▃▅▅···█▃ beats 7/10	medium · avg 55%	▇ 0.7%	typical
ChineseSimpleQA lab	63.7% ✎	▄··▄▄▄··▄··█ beats 2/6	medium · avg 48%	▇ 0.6%	at its level · small run
Omni-MATH	85.0% ✎	█·█·█·██··██ beats 6/6	medium · avg 64%	▇ 0.1%	blunt · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	85.7% ✎	▁▁▃▃▄▄█▆▃▂▂▁ beats 15/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)	88.7% ✎	▁▂▁▂▅█▂▃▄▃▁▁ beats 30/98	very easy · avg 90%	▇ 0.0%	blunt
CLUEWSC lab	92.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with

$ where to run it

Through an API

api · Together AI · $4.00/M · $3 in / $7 out per 1M tokens · 128k context · docs.together.ai ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run DeepSeek R1 (Jan '25) · needs ~411.5 GB · from huggingface.co/unsloth
chat in the terminal
llama cli -hf unsloth/DeepSeek-R1-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf unsloth/DeepSeek-R1-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run DeepSeek R1 (Jan '25) · needs ~411.5 GB · from huggingface.co/unsloth
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/unsloth/DeepSeek-R1-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

MLX (Mac)

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run DeepSeek R1 (Jan '25) · needs ~419.5 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/DeepSeek-R1-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/DeepSeek-R1-4bit

Weights and documentation: huggingface.co/deepseek-ai ↗

$ how the score was worked out · deepseek-ai/DeepSeek-R1 model… 21%, deepseek-ai/DeepSeek-R1-0528… 16%, Epoch AI — internal evaluatio… 10% and 18 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
deepseek-ai/DeepSeek-R1 model card · lab-reported	18	▇▇▇▇▇▇ 21%	mit
deepseek-ai/DeepSeek-R1-0528 model card · lab-reported	13	▇▇▇▇▇ 16%	mit
Epoch AI — internal evaluations	3	▇▇▇ 10%	CC BY 4.0
Vector Institute Evaluation Leaderboard (Inspect Evals runs)	4	▇▇▇ 9%	Apache-2.0
Aider polyglot benchmark runs	1	▇▇ 7%	Apache-2.0
WildEval ZeroEval published results	1	▇▇ 6%	Apache-2.0
Microsoft — Phi-4-reasoning Technical Report	4	▇▇ 6%	CC BY 4.0
deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card · lab-reported	4	▇▇ 6%	MIT
Arena-Hard-Auto v2.0-Preview leaderboard (repository README)	2	▇ 4%	Apache-2.0
SciCode leaderboard (repository README)	1	▇ 3%	Apache-2.0
LMArena leaderboard dataset	1	▇ 2%	CC BY 4.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 2%	MIT
EQ-Bench 3 canonical leaderboard results	1	▇ 1%	MIT
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 1%	MIT
OpenCompass CompassAcademic leaderboard (full version) Space: committed predictions file	1	▇ 1%	Apache-2.0
LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? (arXiv:2610.04118v1)	1	▇ 1%	CC BY 4.0
Judgemark v2 results (EQ-Bench)	1	▇ 1%	MIT
Multi-SWE-bench experiments (ByteDance Seed baseline runs)	1	▇ 1%	Apache-2.0
Berkeley Function Calling Leaderboard result archive	1	▇ 0%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	2	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 63.2

not counted (1)

deepseek-ai/DeepSeek-R1 model card

CLUEWSC [correct]

92.8%

fewer than three models have this test yet, or this source ran it for this model alone

$ quantization evidence · 7 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	OpenLLM v1 average of 6 tasks (ARC-Challenge 25-shot, GSM8K 5-shot, HellaSwag 10-shot, MMLU 5-shot, TruthfulQA MC2 0-shot, Winogrande 5-shot; lm-evaluation-harness `openllm`, vLLM)	0.81 → 0.813	100.3%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	MATH-500 pass@1 (LightEval/open-r1, mean of 10 seeds)	0.972 → 0.971	99.8%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	GSM8K 5-shot (lm-evaluation-harness, vLLM)	0.959 → 0.957	99.8%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	MMLU 5-shot (lm-evaluation-harness, vLLM)	0.872 → 0.87	99.7%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	Reasoning average of 3 tasks (AIME 2024 pass@1, MATH-500 pass@1, GPQA Diamond pass@1)	0.83 → 0.82	98.8%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	AIME 2024 pass@1 (LightEval/open-r1, mean of 10 seeds)	0.783 → 0.77	98.3%	source ↗
INT4 W4A16 (weights only; GPTQ via MoE-Quant, Red Hat/ISTA)	GPQA Diamond pass@1 (LightEval/open-r1, mean of 10 seeds)	0.734 → 0.719	98.0%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json