whatmodel.sh / Mistral Small 3.1 24B (2503)

Mistral Small 3.1 24B (2503)

+ contribute

Mistral Small 3.1 24B (2503) is an open-weights language model from Mistral, released March 2025. Its published size is 24.01B parameters and a 128k-token context window. Licence: apache-2.0.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2025-03-17). Mistral Small 3.1 (version 25.03). Epoch labels 'mistral-small-2503' (Mistral API id; retired 11/30/2025) and 'mistral-small-3.1-2503' are both model_metadata group 'Mistral Small 3.1'; the selection rule is applied across both. 24,011,361,280 parameters (includes the vision encoder).

54.0±0.9#288 of 719 LLMsfrom 28 benchmark results across 13 sources

$ how it compares

The LLMs ranked either side of Mistral Small 3.1 24B (2503), out of 828 with a score, then more from Mistral.

#	model	score	where	RAM	released
340	Mellum2 12B-A2.5B Thinking JetBrains	54±1	local	~8.6GB	2026-05
341	Mistral Large 2 (2407) Mistral	54±1	local	~45GB	2024-07
342	Mistral Small 3.1 24B (2503) Mistral	54±1	local	~17GB	2025-03
343	GPT-4 Turbo Preview (1106) OpenAI	54±1	comparison only		2023-11
344	GPT-4 Turbo Preview (0125) OpenAI	54±1	comparison only		2024-01
	more from Mistral
79	Mistral Large 4 (public preview) Mistral	71±2	api $2.06/M		2026-10
174	Mistral Medium 3.5 (128B) Mistral	65±1	api $3.00/M +local	~42GB	2026-04
180	Mistral Medium 3.1 (2508) Mistral	64±2	comparison only		2025-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #219 of 300 · 5.4% of its score · 3 tests

Beats o1 (2024-12-17), o3-mini, Llama 3.1 Nemotron Ultra 253B v1 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	10.8% ✎	█·▃▃·█▅▅▅▅▃▅ beats 5/18	very hard · avg 5.8%	▇ 2.6%	sharp
BFCL v3	57.7% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 72/114	medium · avg 52%	▇ 1.6%	blunt · at its level
Galileo Agent Leaderboard v1 (tool selection quality)	91.2% ✎	▃▃▃▃▆▅▆▅█▂▅▂ beats 26/31	easy · avg 79%	▇ 1.2%	blunt

codingnot ranked on this task yet · 2 tests

test	result	vs the others in its run	difficulty	counts	why
MBPP+ pass@5 lab	74.6% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
HumanEval+ pass@5 lab	89.0% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #221 of 393 * · 11% of its score · 2 tests

Beats Granite 4.2 8B, Mistral Small 4 (119B-A6.5B), Olmo 3.1 32B Think here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, Gemma 3 4B, Olmo 3 7B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1303.1 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 129/379	avg 1268	▇▇▇ 7.3%	blunt
Arena-Hard v2 aggregate lab	19.6% ✎	▁▃··▄▃▁▃█▁▁▇ beats 1/22	hard · avg 22%	▇ 3.5%	sharp · at its level · small run

math▁▂▃▅▆▅▄█▄▁▁▁ #255 of 398 · 28% of its score · 4 tests

Beats GPT-4 Turbo (2024-04-09), Molmo 2 8B, Llama 3.3 70B Instruct here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), Tulu 2 DPO 70B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	46.8% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 40/92	hard · avg 32%	▇▇▇▇▇▇▇ 18%	sharp · at its level
OTIS mock AIME 2024–2025	3.9% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 25/193	very hard · avg 8.5%	▇▇ 4.4%	sharp · noisy
MATH (subset unspecified) lab	69.3% ✎	▂▄▄▃█▇▅▆▄▃▄▁ beats 41/80	medium · avg 63%	▇ 3.4%	at its level · small run
MathVista (testmini) lab	68.9% ✎	▁··▁▄▄▇█▅▆█▄ beats 24/47	medium · avg 63%	▇ 2.4%	at its level · small run

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #105 of 147 · 14% of its score · 5 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4o (2024-08-06), GPT-4 Turbo (2024-04-09) here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Arena vision (style controlled)	1127.1 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 39/145	avg 1068	▇▇ 5.0%	blunt
MMMU (val) lab +1	64.0% ✎	▁····▁▁▄█▇▆▆ beats 26/52	medium · avg 57%	▇▇ 4.6%	at its level · small run · several runs
MathVista (testmini) lab	68.9% ✎	▁··▁▄▄▇█▅▆█▄ beats 24/47	medium · avg 63%	▇ 2.4%	at its level · small run
ChartQA lab +1	86.2% ✎	▁··▁▁▁▁▁▂▅▅█ beats 27/44	easy · avg 82%	▇ 1.5%	blunt · small run · several runs
DocVQA lab	94.1% ✎	▁··▃▁▁▁▇█▇▄▅ beats 12/28	very easy · avg 93%	▇ 0.8%	blunt · maxed out · small run

web search▂▁▂▆▄█▃▃▁▁·▁ #11 of 80 * · 2.6% of its score · 1 test

Beats Claude Opus 5.5, Claude Fable 5.1, Claude Sonnet 5.5 here, though they rank higher overall. Trails GPT-4o mini (2024-07-18) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)	10.8% ✎	█·▃▃·█▅▅▅▅▃▅ beats 5/18	very hard · avg 5.8%	▇ 2.6%	sharp

other tests14 not in a task · 43% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +1	41.9% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 44/208	hard · avg 40%	▇▇▇▇▇▇▇ 17%	sharp · at its level · several runs
CPTUB average (SpeakLeash, 0-5 judge scale)	78.0% ✎	▁▂▂▂▃█▃▆▅▄▂▃ beats 75/95	easy · avg 66%	▇▇ 4.5%	at its level
EQ-Bench 3 (emotional intelligence, judge Elo)	1055.9 ✎	▁·▂▃▄▅▅▃█▄▁▁ beats 5/62	avg 1044	▇▇ 4.5%	typical
BRIDGE clinical text, zero-shot average (87 tasks)	39.7% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 68/97	hard · avg 32%	▇▇ 3.8%	blunt · at its level
MMLU lab	80.6% ✎	▁▁▁▁▁▂▃▅█▃▆▃ beats 139/183	easy · avg 74%	▇ 3.5%	at its level · small run
MMLU-Pro lab	66.8% ✎	▁▁·▁▁▁▃▂▄▅█▁ beats 78/188	medium · avg 58%	▇ 3.0%	at its level · small run
SimpleQA lab	10.4% ✎	▄▄▄▆█▆███·▂█ beats 12/32	very hard · avg 7.9%	▇ 2.8%	small run
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)	61.2% ✎	▁··▃▁▁▁▅█▅▁▅ beats 10/24	medium · avg 52%	▇ 1.4%	blunt · at its level

6 more tests

test	result	vs the others in its run	difficulty	counts	why
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	93.1% ✎	▃·▁▁▂▁▂▅█▇▄▃ beats 99/100	medium · avg 68%	▇ 1.2%	at its level · noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	94.4% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 139/146	easy · avg 83%	▇ 0.7%	noisy
Chess Puzzles (Epoch)	1.0% ✎	█▁▁▁▁▂▁▄▂▂▁▁ beats 4/135	very hard · avg 5.9%	▇ 0.5%	sharp · too hard for it · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	94.4% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 52/140	very easy · avg 94%	▇ 0.4%	blunt · maxed out · noisy
WildBench v2 lab	55.6% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
GPQA Main lab	44.4% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Mistral Small 3.1 24B (2503) · needs ~14.3 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Mistral Small 3.1 24B (2503) · needs ~14.3 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Mistral-Small-3.1-24B-Instruct-2503-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Mistral Small 3.1 24B (2503) · needs ~14.91 GB · from ollama.com
chat in the terminal
ollama run mistral-small3.1:24b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Mistral Small 3.1 24B (2503) · needs ~15.1 GB, Mac only · from huggingface.co/luiscalisto
chat in the terminal
mlx_lm.chat --model luiscalisto/Mistral-Small-3.1-24B-Instruct-2503-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model luiscalisto/Mistral-Small-3.1-24B-Instruct-2503-MLX-4bit

Weights and documentation: huggingface.co/mistralai ↗

$ how the score was worked out · Epoch AI — internal evaluatio… 34%, mistralai/Mistral-Small-3.2-2… 26%, LMArena leaderboard dataset 12% and 10 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Epoch AI — internal evaluations	4	▇▇▇▇▇▇▇▇▇▇ 34%	CC BY 4.0
mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card · lab-reported	9	▇▇▇▇▇▇▇▇ 26%	Apache-2.0
LMArena leaderboard dataset	2	▇▇▇▇ 12%	CC BY 4.0
Mistral AI vendor-reported benchmark scores for its own models (Hugging Face model cards) · lab-reported	3	▇▇ 5%	Factual vendor-reported score (not licensed content)
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 4%	MIT
EQ-Bench 3 canonical leaderboard results	1	▇ 4%	MIT
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 4%	MIT
Finance Agent Benchmark (Vals AI, arXiv:2508.00828v1)	1	▇ 3%	CC-BY-4.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 2%	MIT
Berkeley Function Calling Leaderboard result archive	1	▇ 2%	Apache-2.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)	1	▇ 1%	CC BY 4.0
Galileo Agent Leaderboard (v1 and v2 results)	1	▇ 1%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 54.0

not counted (4)

mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card	WildBench v2 [correct]	55.6%	fewer than three models have this test yet, or this source ran it for this model alone
mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card	GPQA Main [correct]	44.4%	fewer than three models have this test yet, or this source ran it for this model alone
mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card	MBPP+ pass@5 [correct]	74.6%	fewer than three models have this test yet, or this source ran it for this model alone
mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card	HumanEval+ pass@5 [correct]	89.0%	fewer than three models have this test yet, or this source ran it for this model alone

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json