whatmodel.sh / Mistral Large 2 (2407)

Mistral Large 2 (2407)

+ contribute

Mistral Large 2 (2407) is an open-weights language model from Mistral, released July 2024. Its published size is 122.61B parameters and a 128k-token context window. Licence: mistral-research.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-07-24). API id mistral-large-2407 ('Mistral Large 2.0', version 24.07; retired 3/30/2025); weights Mistral-Large-Instruct-2407 under the Mistral Research License. Distinct from mistral-large-2 (2411). 122,610,069,504 parameters. Note: ollama mistral-large:123b now points to the 2411 build, so the 2407 tags are pinned explicitly.

54.1±0.5#287 of 719 LLMsfrom 34 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of Mistral Large 2 (2407), out of 828 with a score, then more from Mistral.

#	model	score	where	RAM	released
339	Nova Premier Amazon	54±2	api $5.00/M		2025-04
340	Mellum2 12B-A2.5B Thinking JetBrains	54±1	local	~8.6GB	2026-05
341	Mistral Large 2 (2407) Mistral	54±1	local	~45GB	2024-07
342	Mistral Small 3.1 24B (2503) Mistral	54±1	local	~17GB	2025-03
343	GPT-4 Turbo Preview (1106) OpenAI	54±1	comparison only		2023-11
	more from Mistral
79	Mistral Large 4 (public preview) Mistral	71±2	api $2.06/M		2026-10
174	Mistral Medium 3.5 (128B) Mistral	65±1	api $3.00/M +local	~42GB	2026-04
180	Mistral Medium 3.1 (2508) Mistral	64±2	comparison only		2025-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #221 of 300 * · 1.0% of its score · 2 tests

Beats o1 (2024-12-17), o3-mini, Llama 3.1 Nemotron Ultra 253B v1 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BFCL v3	57.7% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 71/114	medium · avg 52%	▇ 0.6%	blunt · at its level
BFCL (version not stated)	85.1% ✎	▄▆▄▂·█▆·▄··▂ beats 13/17	easy · avg 81%	▇ 0.5%	blunt · small run

coding▁▁▁▃▆██▇▆▄▃▁ #219 of 458 · 16% of its score · 5 tests

Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B) here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, DeepSeek-Coder-V2 Instruct (0724) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	75.1% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 37/47	medium · avg 61%	▇▇ 6.1%	sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)	60.2% ✎	▁·▁▂▄▁▆▄█▆▂▂ beats 30/62	medium · avg 54%	▇▇ 4.6%	at its level
HumanEval (Vector Institute, Inspect Evals)	86.6% ✎	▄·▄·█·█·█·▄▄ beats 3/9	easy · avg 79%	▇ 2.4%	typical
WebApp1K-Duo-React pass@1 (onekq-ai)	44.9% ✎	▁··▁·▁▃▁▆█▇█ beats 5/45	hard · avg 32%	▇ 1.3%	at its level
WebApp1K-React pass@1 (onekq-ai)	78.0% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 20/31	medium · avg 44%	▇ 1.2%	sharp · at its level · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #205 of 393 · 9.3% of its score · 3 tests

Beats Granite 4.2 8B, Qwen3-VL 30B-A3B Thinking, Mistral Small 4 (119B-A6.5B) here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, Gemma 3n E4B, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	70.4% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 46/60	medium · avg 52%	▇▇ 6.2%	sharp · at its level
LMArena (text)	1314.4 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 145/379	avg 1268	▇ 2.6%	blunt
IFEval (aggregation unspecified)	87.8% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 91/141	easy · avg 80%	▇ 0.6%	small run

math▁▂▃▅▆▅▄█▄▁▁▁ #258 of 398 · 21% of its score · 6 tests

Beats Molmo 2 8B, Llama 3.3 70B Instruct here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), Tulu 2 DPO 70B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	44.8% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 37/92	hard · avg 32%	▇▇▇ 6.5%	sharp · at its level
MATH (Vector Institute, Inspect Evals)	65.7% ✎	▄··██▄··▄·▄█ beats 3/9	medium · avg 53%	▇▇ 4.9%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	48.5% ✎	▁▁▁▆█▂▇▇▄··▂ beats 25/38	hard · avg 34%	▇▇ 4.0%	sharp · at its level
GSM8K (ZeroEval, zero-shot)	95.5% ✎	▁▄·▁▄█▅▅▅███ beats 37/43	very easy · avg 89%	▇ 2.0%	sharp · maxed out
GSM8K	91.0% ✎	▂▂▂▅▅▆▅▇█▇▄▃ beats 127/146	easy · avg 85%	▇ 1.6%	sharp · small run
OTIS mock AIME 2024–2025	8.5% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 48/193	very hard · avg 8.5%	▇ 1.6%	sharp · noisy

other tests18 not in a task · 53% of its score

test	result	vs the others in its run	difficulty	counts	why
LiveBench 2024-07-26	48.0% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 50/61	medium · avg 41%	▇▇▇▇▇ 11%	at its level
MMLU-Redux (ZeroEval, 2,778 questions)	83.0% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 39/53	easy · avg 77%	▇▇▇▇ 9.5%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	29.0% ✎	▁▁▃▆██▁▁·▁▁▁ beats 47/57	hard · avg 18%	▇▇ 5.4%	sharp · at its level
GPQA Diamond	49.0% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 66/208	hard · avg 40%	▇▇ 4.0%	sharp · at its level
GPQA Diamond (Vector Institute, Inspect Evals)	47.3% ✎	▃··▃▃▃▃·▃·▃█ beats 3/9	hard · avg 41%	▇ 3.6%	sharp · at its level
WildBench v2 WB-Score (task macro)	55.5683 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 49/52	avg 45.199 WB-Score	▇ 2.9%	sharp
GPQA (subset unspecified)	40.7% ✎	▂▁▂▂▄▃▆▃▅█▄▃ beats 41/104	hard · avg 37%	▇ 2.7%	sharp · small run
MMLU-Pro (Vector Institute, Inspect Evals)	69.4% ✎	▃····▃▅·▃▅·█ beats 3/9	medium · avg 60%	▇ 2.4%	at its level

10 more tests

test	result	vs the others in its run	difficulty	counts	why
BIG-Bench Hard	70.8% ✎	▁▁▁▁▂▄▅▅█▃▄▁ beats 92/129	medium · avg 65%	▇ 2.3%	sharp · at its level · small run
CPTUB average (SpeakLeash, 0-5 judge scale)	78.7% ✎	▁▂▂▂▃█▃▆▅▄▂▃ beats 78/95	easy · avg 66%	▇ 1.6%	at its level
MMLU	82.5% ✎	▁▁▁▁▁▂▃▅█▃▆▃ beats 146/183	easy · avg 77%	▇ 1.4%	small run
MMLU-Pro	54.2% ✎	▁▁·▁▁▁▃▃▄▅█▁ beats 54/188	medium · avg 50%	▇ 1.4%	at its level · small run
EvalPerf correctness (118 tasks, 20240328)	82.4% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 19/29	easy · avg 76%	▇ 1.3%	typical
Polish EQ-Bench v2 (SpeakLeash)	78.1% ✎	▁··▁▁▂▂▆▅▆█▆ beats 70/70	medium · avg 64%	▇ 1.3%	at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	64.8% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 63/81	medium · avg 58%	▇ 1.1%	blunt · at its level
RULER 64k	78.8% ✎	▁▁·▁·▂▃▄█▂·▁ beats 6/22	easy · avg 83%	▇ 0.4%	typical
ARC Challenge	65.0% ✎	▆▁▁▂▂▅▇█▄▃▃▂ beats 76/97	easy · avg 81%	▇ 0.3%	sharp · noisy · small run
RULER 128k	23.7% ✎	▄··▂▄▂▆██▄▄▂ beats 1/21	easy · avg 68%	▇ 0.2%	at its level · noisy

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Mistral Large 2 (2407) · needs ~73.2 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Mistral Large 2 (2407) · needs ~73.2 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Mistral-Large-Instruct-2407-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Mistral Large 2 (2407) · needs ~74.07 GB · from ollama.com
chat in the terminal
ollama run mistral-large:123b-instruct-2407-q4_K_M

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Mistral Large 2 (2407) · needs ~69 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/Mistral-Large-Instruct-2407-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/Mistral-Large-Instruct-2407-4bit

Weights and documentation: huggingface.co/mistralai ↗

$ how the score was worked out · WildEval ZeroEval published r… 27%, Vector Institute Evaluation L… 13%, Epoch AI — internal evaluatio… 12% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇▇▇ 27%	Apache-2.0
Vector Institute Evaluation Leaderboard (Inspect Evals runs)	4	▇▇▇▇ 13%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇▇ 12%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇ 11%	Apache-2.0 (stated in the LiveBench datasheet)
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale (AI21, arXiv 2408.12570v1)	8	▇▇▇ 11%	CC BY-SA 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇▇ 6%	Apache-2.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 5%	Apache-2.0
WildBench v2 leaderboard data (Ai2)	1	▇ 3%	Apache-2.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 3%	MIT
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 2%	MIT
EvalPlus / EvalPerf published evaluation runs	1	▇ 1%	Apache-2.0
SpeakLeash Polish EQ-Bench leaderboard	1	▇ 1%	Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
NVIDIA RULER repository README results table	2	▇ 1%	Apache-2.0
Berkeley Function Calling Leaderboard result archive	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 54.1