whatmodel.sh / DeepSeek V3 (Dec '24)

DeepSeek V3 (Dec '24)

+ contribute

DeepSeek V3 (Dec '24) is an open-weights language model from DeepSeek, released December 2024. Its published size is 671B parameters (37B active per token) and a 131k-token context window. Licence: deepseek.

59.3±0.6#204 of 719 LLMsfrom 46 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of DeepSeek V3 (Dec '24), out of 828 with a score, then more from DeepSeek.

#	model	score	where	RAM	released
244	QwQ-32B Alibaba	59±1	local	~21GB	2025-03
245	OpenReasoning-Nemotron-32B NVIDIA	59±2	local	~21GB	2025-07
246	DeepSeek V3 (Dec '24) DeepSeek	59±1	api $0.46/M +local	~337GB	2024-12
247	Gemma 2 9B IT SimPO Princeton NLP	59±1	local	~7.6GB	2024-07
248	Gemma 2 9B IT DPO (Princeton) Princeton NLP	59±1	local	~7.6GB	2024-07
	more from DeepSeek
21	DeepSeek V4.1 Flash DeepSeek	78±1	api $0.26/M +local	~537GB	2026-09
31	DeepSeek V4 Pro (0813) DeepSeek	76±1	api $1.63/M +local	~1010GB	2026-08
35	DeepSeek V3.2 Speciale DeepSeek	76±3	comparison only

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #217 of 300 · 8.1% of its score · 4 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 (Jan '25) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
SWE-bench Verified +1	45.8% ✎	▃···▃·▃·▃█·▃ beats 7/7	hard · avg 19%	▇▇ 5.8%	sharp · at its level · several runs
τ-bench Airline	25.5% ✎	▁▃▁·▅▄▁█▇▁▁▇ beats 1/28	hard · avg 23%	▇ 1.0%	at its level
τ-bench Retail	54.8% ✎	▁····▃·▂▇█▂▆ beats 7/28	hard · avg 29%	▇ 0.7%	at its level · noisy
BFCL v3	57.6% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 70/114	medium · avg 52%	▇ 0.6%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #150 of 458 · 42% of its score · 10 tests

Beats Gemini 2.5 Pro, Gemini 3.1 Flash-Lite, Qwen3-Coder-Next (80B-A3B) here, though they rank higher overall. Trails Ministral 3 8B Reasoning 2512, Phi-4-reasoning 14B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts) +2	48.4% ✎	▂·▂▃▅▂▃█▆▄▂▂ beats 25/49	very hard · avg 8.5%	▇▇▇▇▇▇▇ 17%	sharp · at its level · several runs
LiveCodeBench official (Jan 1–Nov 1, 2024)	48.9% ✎	▁▁·▁▂▄▆█▅▃·▂ beats 28/31	hard · avg 26%	▇▇▇ 8.5%	sharp · at its level
SWE-bench Verified +1	45.8% ✎	▃···▃·▃·▃█·▃ beats 7/7	hard · avg 19%	▇▇ 5.8%	sharp · at its level · several runs
LiveCodeBench v5 (Oct 2024–Feb 2025)	33.5% ✎	▄▄·▄·····█▄█ beats 7/7	hard · avg 19%	▇ 2.5%	at its level
SciCode main problems (official leaderboard)	3.1% ✎	▃···█··▃·▃·▂ beats 12/17	very hard · avg 1.1%	▇ 2.4%	sharp · too hard for it
WebApp1K-Duo-React pass@1 (onekq-ai)	58.5% ✎	▁··▁·▁▁▁▆█▇█ beats 12/45	hard · avg 32%	▇ 1.6%	at its level
MBPP+	90.0% ✎	██·██···████ beats 7/7	easy · avg 79%	▇ 1.2%	typical
BIRD-SQL (dev)	53.1% ✎	▄··▄·▄·█··▄█ beats 4/7	medium · avg 49%	▇ 1.1%	at its level

2 more tests

test	result	vs the others in its run	difficulty	counts	why
LiveCodeBench (version unspecified) lab	39.2% ✎	▁···▁▁▃▅▅█▄▅ beats 8/37	hard · avg 30%	▇ 1.1%	at its level · small run
WebApp1K-React pass@1 (onekq-ai)	87.2% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 25/31	medium · avg 44%	▇ 1.0%	sharp · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #148 of 393 · 10% of its score · 4 tests

Beats INTELLECT-3, MiniMax-M2, MiniCPM5 2B here, though they rank higher overall. Trails LFM2.5 1.2B Thinking, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge) lab	85.5% ✎	▂▁▁▅█▅▃▃▃▅▃▂ beats 59/62	medium · avg 54%	▇ 3.6%	sharp · small run
LMArena (text)	1358.4 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 202/379	avg 1268	▇ 3.0%	blunt
IFEval prompt-level strict +1	85.8% ✎	▄▄··▄·▄··█·▄ beats 5/6	easy · avg 74%	▇ 1.9%	several runs
AlpacaEval 2 length-controlled lab	70.0% ✎	▁▁▃▄▆██▆▃▃▁▁ beats 149/155	medium · avg 50%	▇ 1.5%	at its level · small run

math▁▂▃▅▆▅▄█▄▁▁▁ #218 of 398 · 10% of its score · 4 tests

Trails Olmo 3 7B Instruct, DeepSeek R1 Distill Llama 8B, Phi-4-mini-reasoning 3.8B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	64.9% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 55/92	hard · avg 32%	▇▇ 5.7%	sharp
OTIS mock AIME 2024–2025	15.8% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 53/193	very hard · avg 8.5%	▇ 3.0%	sharp · at its level · noisy
AIME 2024 lab	39.2% ✎	▃·▃·▃▃···█·▃ beats 1/7	medium · avg 50%	▇ 0.9%	sharp · noisy
MATH-500 lab	90.2% ✎	▄·▄▄·▄▄█···▄ beats 2/7	very easy · avg 91%	▇ 0.6%	maxed out

other tests21 not in a task · 36% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +1	56.5% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 83/208	hard · avg 40%	▇▇▇▇ 9.0%	sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzles)	42.1% ✎	▁▁▃▆██▃▁·▁▁▁ beats 52/57	hard · avg 18%	▇▇▇ 7.7%	sharp · at its level
KCL-Essay vanilla	34.3% ✎	▁···▁▃▂█▂▃▃▂ beats 13/23	hard · avg 18%	▇ 2.0%	at its level
DROP lab	91.6% ✎	▅··▅·█▅█▅▃▃▅ beats 16/17	easy · avg 73%	▇ 1.6%	small run
CPTUB average (SpeakLeash, 0-5 judge scale)	80.5% ✎	▁▂▂▂▃█▃▆▅▄▂▃ beats 82/95	easy · avg 66%	▇ 1.6%	typical
MMLU lab	88.5% ✎	▁▁▁▁▁▂▃▅█▃▆▃ beats 173/183	easy · avg 76%	▇ 1.5%	small run
BigCodeBench (Instruct)	48.6% ✎	▄··▄▄···▄▄█▄ beats 7/7	medium · avg 36%	▇ 1.5%	at its level
RepoQA (32K, threshold 0.8)	92.2% ✎	▄▄▄··▄▄··▄·█ beats 6/7	easy · avg 81%	▇ 1.5%	typical

13 more tests

test	result	vs the others in its run	difficulty	counts	why
C-Eval lab	86.5% ✎	▁···▁▁▂▁▂▂▃█ beats 16/33	easy · avg 75%	▇ 1.3%	small run
MMLU-Pro lab	75.9% ✎	▁▁·▁▁▁▃▂▄▅█▁ beats 105/188	medium · avg 60%	▇ 1.3%	at its level · small run
MMLU-Redux lab	89.1% ✎	▁▁▃▁▂▃▄▅▇▅█▄ beats 31/63	easy · avg 80%	▇ 1.2%	small run
SimpleQA lab	24.9% ✎	▄▄▄▆█▆███·▂█ beats 24/32	very hard · avg 11%	▇ 1.1%	small run
EvalPerf correctness (118 tasks, 20240328)	88.6% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 28/29	easy · avg 76%	▇ 1.1%	typical
KCL-MCQA vanilla	37.3% ✎	▂···█▂▆▆██▆▂ beats 9/23	very hard · avg 30%	▇ 1.0%	typical
ChineseSimpleQA lab	68.0% ✎	▄··▄▄▄··▄··█ beats 3/6	medium · avg 48%	▇ 0.8%	at its level · small run
FRAMES lab	73.3% ✎	█··········█ beats 0/1	medium · avg 63%	▇ 0.7%	small run
CNMO 2024 lab	43.2% ✎	█··········█ beats 0/1	medium · avg 48%	▇ 0.4%	at its level · noisy · small run
DROP F1, 3-shot (simple-evals)	92.1% ✎	▅▂▆█▆·▃·▂▅·▃ beats 23/23	easy · avg 81%	▇ 0.1%	blunt · small run
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	96.1% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 88/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)	93.9% ✎	▁▂▁▂▅█▂▃▄▃▁▁ beats 76/98	very easy · avg 90%	▇ 0.0%	blunt
CLUEWSC lab	90.9% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with

$ where to run it

Through an API

api · Deep Infra · $0.46/M · $0.32 in / $0.89 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run DeepSeek V3 (Dec '24) · needs ~404.4 GB · from huggingface.co/unsloth
chat in the terminal
llama cli -hf unsloth/DeepSeek-V3-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf unsloth/DeepSeek-V3-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run DeepSeek V3 (Dec '24) · needs ~404.4 GB · from huggingface.co/unsloth
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/unsloth/DeepSeek-V3-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run DeepSeek V3 (Dec '24) · needs ~403.1 GB · from ollama.com
chat in the terminal
ollama run deepseek-v3:671b

MLX (Mac)

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run DeepSeek V3 (Dec '24) · needs ~377.6 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/DeepSeek-V3-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/DeepSeek-V3-4bit

Weights and documentation: huggingface.co/deepseek-ai ↗

$ how the score was worked out · deepseek-ai/DeepSeek-R1 model… 26%, Cohere — Command A technical… 18%, Epoch AI — internal evaluatio… 14% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
deepseek-ai/DeepSeek-R1 model card · lab-reported	17	▇▇▇▇▇▇▇▇ 26%	mit
Cohere — Command A technical report	9	▇▇▇▇▇ 18%	CC BY 4.0
Epoch AI — internal evaluations	3	▇▇▇▇ 14%	CC BY 4.0
LiveCodeBench official website results (2024-12-26)	1	▇▇▇ 8%	CC-BY-SA-4.0
Aider polyglot benchmark runs	1	▇▇ 8%	Apache-2.0
WildEval ZeroEval published results	1	▇▇ 8%	Apache-2.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities	2	▇ 3%	CC BY 4.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 3%	MIT
SciCode leaderboard (repository README)	1	▇ 2%	Apache-2.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 2%	MIT
OpenCompass CompassAcademic leaderboard (full version) Space: committed predictions file	2	▇ 1%	Apache-2.0
EvalPlus / EvalPerf published evaluation runs	1	▇ 1%	Apache-2.0
deepseek-ai/DeepSeek-V3-0324 model card · lab-reported	1	▇ 1%	MIT
Berkeley Function Calling Leaderboard result archive	1	▇ 1%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	2	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 59.3