whatmodel.sh / Gemini 1.5 Pro (002)
Gemini 1.5 Pro (002)
↗ share
+ contribute
Gemini 1.5 Pro (002) is a closed-weights language model from Google, released September 2024. Licence: proprietary.
Catalogue identity notes
Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-09-24). Stable API version gemini-1.5-pro-002 (September 2024); distinct from -001. Gemini API changelog, September 29, 2025: Gemini 1.5 models shut down.
$ how it compares
The LLMs ranked either side of Gemini 1.5 Pro (002), out of 828 with a score, then more from Google.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 287 |  Qwen3 4B Thinking 2507 | 57±1 | local | ||
| 288 | Gemini 1.5 Pro Experimental (0827) | 56±1 | comparison only | ||
| 289 | Gemini 1.5 Pro (002) | 56±1 | comparison only | ||
| 290 |  Athene-V2-Chat 72B | 56±2 | local | ||
| 291 | Gemini 2.0 Flash Experimental | 56±1 | comparison only | ||
| more from Google | |||||
| 9 | Gemini 4 Argon (limited preview) | 82±1 | limited preview | ||
| 23 | Gemini 3.8 Flash | 77±1 | api $1.50/M | ||
| 32 | Gemini 3.7 Flash | 76±1 | api $1.50/M | ||
$ what it's good at
coding▁▁▁▃▅▇█▇▅▄▃▁ #246 of 524 · 51% of its score · 5 tests
Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B) here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, DeepSeek-Coder-V2 Instruct (0724) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| HumanEval | 89.0% ✎ | ▁·▁▁▃▄▆▅█▃▁▁ beats 95/99 | 15% |
| HumanEval+ | 79.3% ✎ | ▁·▁▁▁▃▄▄▄█▃▁ beats 91/99 | 14% |
| LiveCodeBench official (Jan 1–Nov 1, 2024) | 36.4% ✎ | ▁▁·▁▂▄▆█▅▂·▂ beats 21/31 | 11% |
| Aider code editing (133 Exercism Python exercises, two attempts) | 65.4% ✎ | ▁·▁▂▄▁▆▄█▆▂▂ beats 36/62 | 6.6% |
| MBPP+ | 74.6% ✎ | ▁···▁▁▂▅▇█▆▃ beats 71/75 | 3.7% |
chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #184 of 463 * · 4.0% of its score · 1 test
Beats MiniMax-M2, MiniCPM5 2B, Granite 4.2 30B here, though they rank higher overall. Trails LFM2.5 1.2B Thinking, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| LMArena (text) | 1351.3 ✎ | ▁▁▁▂▃▃▃▆▆▇█▃ beats 195/379 | 4.0% |
math▁▂▃▅▇▅▅█▄▁▁▁ #234 of 462 * · 13% of its score · 2 tests
Beats DeepSeek V3 (Dec '24), GPT-4o (2024-08-06), Command A (03-2025) here, though they rank higher overall. Trails Olmo 3 7B Instruct, DeepSeek R1 Distill Llama 8B, Phi-4-mini-reasoning 3.8B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH level 5 | 70.4% ✎ | ▂▂▅▆▄█▆▃▅▂▃▄ beats 61/92 | 10% |
| OTIS mock AIME 2024–2025 | 23.1% ✎ | ▃▃▄▃▃▃▇▇█▃▂▅ beats 61/193 | 3.2% |
vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #102 of 182 * · 4.7% of its score · 2 tests
Beats DeepSeek V4 Flash Vision (Exp), GLM-4.6V, GPT-5 nano here, though they rank higher overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| Arena vision (style controlled) | 1179.4 ✎ | ▁▁▂▂▂▂▄▅▅▆██ beats 60/145 | 2.8% |
| CapArena-Auto detailed image captioning (Score_Avg, GPT-4o judge) | 56.1700 ✎ | ▃▃▃·▃█·▅▃▃▃▃ beats 12/12 | 1.9% |
other tests16 not in a task · 28% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| GPQA Diamond | 57.2% ✎ | ▁▁▁▁▂▄▆▄▅▇█▅ beats 86/208 | 6.9% |
| MMLU-Pro | 70.2% ✎ | ▃▂▅█▃·█▃▅··▂ beats 18/23 | 6.4% |
| AHELM Speech Robust Bench (LibriSpeech-clean with noise) transcription WER | 20.7% ✎ | ▂·█··▆▄·▂▂·▄ beats 7/13 | 3.8% |
| LiveBench 2024-08-31 | 54.9% ✎ | ▄█·▄·▄▄·▄··▄ beats 5/7 | 2.5% |
| BRIDGE clinical text, zero-shot average (87 tasks) | 43.9% ✎ | ▁▁▁▁▁▂▂▃▄▆█▃ beats 89/97 | 2.2% |
| EvalPerf correctness (118 tasks, 20240328) | 83.7% ✎ | ▂·▂▂▂▆▅▆▆▆▃█ beats 21/29 | 1.8% |
| AHELM CoRe-Bench conversational audio reasoning (pseudo-exact match) | 79.9% ✎ | ▂·▄··▂▂·▄▂█▄ beats 12/13 | 1.2% |
| AHELM VoxCeleb2 same-speaker detection (exact match) | 58.5% ✎ | ▂·▂·▂█▂▄▄··▄ beats 11/13 | 0.8% |
8 more tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| AHELM LibriSpeech transcription by audio LLMs (WER) | 5.2% ✎ | ▃···▃·█▃▅▃▅█ beats 9/13 | 0.8% |
| AHELM MUStARD audio sarcasm detection (exact match) | 57.7% ✎ | ▂·······▆█▆▆ beats 5/13 | 0.6% |
| AHELM AIR-Bench Foundation music subsets (exact match) | 55.4% ✎ | ▃█▃▅▃·▃·▃▅▃▃ beats 5/13 | 0.3% |
| AHELM MELD audio emotion classification (pseudo-exact match) | 51.6% ✎ | ▃··▃▃▃·▃▅▅█▅ beats 13/13 | 0.2% |
| Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate) | 93.7% ✎ | ▁▁▂▃▄▄█▆▃▂▂▁ beats 49/140 | 0.2% |
| AHELM FLEURS 7-language transcription by audio LLMs (WER) | 34.2% ✎ | ▄▂···▂▆▂█▃·▂ beats 12/13 | 0.1% |
| AHELM Multilingual LibriSpeech 7-language transcription by audio LLMs (WER) | 5.4% ✎ | ▂▂▂··▂▂·▂▂█▆ beats 13/13 | 0.0% |
| AHELM VocalSound human-sound recognition (prefix exact match) | 52.8% ✎ | ██▄▄·█·█·█▄▄ beats 4/13 | 0.0% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · EvalPlus / EvalPerf published… 35%, Epoch AI — internal evaluatio… 20%, LiveCodeBench official websit… 11% and 8 more
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json