whatmodel.sh / Mistral Medium 3 (2505)
Mistral Medium 3 (2505)
↗ share
+ contribute
Mistral Medium 3 (2505) is a closed-weights language model from Mistral, released May 2025. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2025-05-07). API model mistral-medium-2505 ('Mistral Medium 3', version 25.05; deprecated 5/22/2026, retired 8/31/2026 per Mistral's legacy-models table). Distinct from mistral-medium-3-5.
$ how it compares
The LLMs ranked either side of Mistral Medium 3 (2505), out of 828 with a score, then more from Mistral.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 234 |  Gemini 2.0 Flash (001) | 60±1 | comparison only | ||
| 235 |  ERNIE 4.5 300B-A47B | 60±1 | local | ||
| 236 | Mistral Medium 3 (2505) | 60±1 | comparison only | ||
| 237 |  Qwen3 235B-A22B (Apr 2025, hybrid) | 60±1 | api $1.23/M +local | ||
| 238 |  Claude 3.5 Sonnet (2024-10-22) | 60±1 | comparison only | ||
| more from Mistral | |||||
| 79 | Mistral Large 4 (public preview) | 71±2 | api $2.06/M | ||
| 174 | Mistral Medium 3.5 (128B) | 65±1 | api $3.00/M +local | ||
| 180 | Mistral Medium 3.1 (2508) | 64±2 | comparison only | ||
$ what it's good at
agent tasks▁▁▂▆█▆▅▄▄▃▂▁ #225 of 340 * · 6.6% of its score · 1 test
Beats o1 (2024-12-17), o3-mini, DeepSeek R1 (Jan '25) here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Mellum2 12B-A2.5B Instruct here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BFCL V4 (December 2025 archive) | 37.6% ✎ | ▁▁▁▂██▅▅▆▂▂▂ beats 43/77 | 6.6% |
coding▁▁▁▃▅▇█▇▅▄▃▁ #200 of 524 * · 6.9% of its score · 1 test
Beats Qwen3.5 9B, Devstral 2 (2512), Qwen3-VL 32B Thinking here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 8B Reasoning 2512, Phi-4-reasoning 14B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| WebApp1K-Duo-React pass@1 (onekq-ai) | 53.7% ✎ | ▁··▁·▁▁▁▆█▇█ beats 7/45 | 6.9% |
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #142 of 463 * · 13% of its score · 1 test
Beats LongCat 2.0 (1.6T-A48B), MiMo-V2-Flash (309B-A15B), MiniMax-M2.1 here, though they rank higher overall. Trails LFM2.5 2.6B, Nemotron-Cascade-8B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| LMArena (text) | 1387.5 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 229/379 | 13% |
math▁▂▃▅▇▅▅█▄▁▁▁ #205 of 462 * · 37% of its score · 2 tests
Beats MiniMax-M2.1, DeepSeek V3 0324 here, though they rank higher overall. Trails Phi-4-mini-reasoning 3.8B, OpenThinker3 7B, Olmo 3 7B Think here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH level 5 | 81.6% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 66/92 | 23% |
| OTIS mock AIME 2024–2025 | 32.2% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 68/193 | 13% |
vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #113 of 182 * · 9.1% of its score · 1 test
Beats DeepSeek V4 Flash Vision (Exp), GPT-5 nano here, though they rank higher overall. Trails GPT-4o (2024-11-20), Mistral Small 4 (119B-A6.5B), Qwen3-VL 30B-A3B Thinking here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena vision (style controlled) | 1155.2 ✎ | ▁▁▂▂▂▂▄▅▅▆██ beats 50/145 | 9.1% |
other tests2 not in a task · 27% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| GPQA Diamond | 59.5% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 87/208 | 24% |
| DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options) | 60.3% ✎ | ▁··▃▁▁▁▅█▅▁▅ beats 9/24 | 2.7% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Epoch AI — internal evaluatio… 61%, LMArena leaderboard dataset 22%, WebApp1K models leaderboard (… 7% and 2 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Epoch AI — internal evaluations | 3 | 61% |
 LMArena leaderboard dataset | 2 | 22% |
 WebApp1K models leaderboard (onekq-ai) | 1 | 7% |
| Berkeley Function Calling Leaderboard result archive | 1 | 7% |
 DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1) | 1 | 3% |
| prior · pull toward the middle | 1% | |
| weighted average | = 59.9 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json