whatmodel.sh / Gemini 1.5 Pro (001)

Gemini 1.5 Pro (001)

+ contribute

Gemini 1.5 Pro (001) is a closed-weights language model from Google, released May 2024. Licence: proprietary.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-05-14). Stable API version gemini-1.5-pro-001 (May 2024); distinct from -002. Gemini API changelog, September 29, 2025: Gemini 1.5 models shut down.

53.1±0.7comparison only: no current run option verifiedfrom 27 benchmark results across 9 sources

$ how it compares

The LLMs ranked either side of Gemini 1.5 Pro (001), out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
352	Gemini 1.5 Flash Experimental (0827) Google	53±1	comparison only		2024-08
353	GPT-4o mini (2024-07-18) OpenAI	53±0	api $0.26/M		2024-07
354	Gemini 1.5 Pro (001) Google	53±1	comparison only		2024-05
355	Llama 3.1 Tulu 3 70B DPO Ai2	53±1	local	~22GB	2024-11
356	Qwen Turbo (2024-11-01) Alibaba	53±2	comparison only		2024-11
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▅▇█▇▅▄▃▁ #266 of 524 · 15% of its score · 3 tests

Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B) here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, GPT-4 (0613) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	68.0% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 25/47	medium · avg 61%	▇▇▇▇ 11%	sharp · at its level
WebApp1K-React pass@1 (onekq-ai)	68.1% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 17/31	medium · avg 44%	▇ 2.1%	sharp · at its level · noisy
SciCode main problems (official leaderboard)	1.5% ✎	▃···█··▂·▃·▂ beats 3/17	very hard · avg 1.1%	▇ 1.7%	sharp · too hard for it

chat & writing▁▁▁▃▅▄▆█▇▇▇▃ #221 of 463 * · 4.4% of its score · 1 test

Beats Seed-OSS 36B Instruct, Granite 4.2 8B, Qwen3-VL 30B-A3B Thinking here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking, Gemma 3 12B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1324.0 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 160/379	avg 1268	▇▇ 4.4%	blunt

math▁▂▃▅▇▅▅█▄▁▁▁ #314 of 462 · 24% of its score · 4 tests

Beats Molmo 2 8B here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), InternVL2-4B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	40.7% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 33/92	hard · avg 32%	▇▇▇▇ 11%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	39.8% ✎	▁▁▁▆█▂▇▇▄··▂ beats 20/38	hard · avg 34%	▇▇▇ 6.7%	sharp · at its level
GSM8K (ZeroEval, zero-shot)	93.4% ✎	▁▄·▁▄█▅▅▅███ beats 29/43	very easy · avg 89%	▇ 3.7%	sharp · maxed out
OTIS mock AIME 2024–2025	6.8% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 40/193	very hard · avg 8.5%	▇ 2.3%	sharp · noisy

vision & documents▁▁▃▃▂▃▅▄▇▇█▃ #137 of 182 · 16% of its score · 5 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4 Turbo (2024-04-09), Pixtral Large (2411) here, though they rank higher overall. Trails Gemma 3 27B, Qwen2.5-VL 32B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CharXiv descriptive questions	72.0% ✎	▄█▄▄▄·██·▄·█ beats 7/12	easy · avg 71%	▇▇ 4.5%	typical
Arena vision (style controlled)	1119.7 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 35/145	avg 1068	▇ 3.0%	blunt
CharXiv (reasoning questions)	43.3% ✎	▂▂▂█▃▂·▃▂··▂ beats 11/13	medium · avg 36%	▇ 3.0%	at its level
MMMU val (VHELM, zero-shot exact match)	61.9% ✎	▄▄··▄▄█··▄▄█ beats 6/9	medium · avg 51%	▇ 2.8%	at its level
MathVista testmini (VHELM, zero-shot exact match)	57.6% ✎	▃·▅···▃▃▃·▃█ beats 9/9	medium · avg 43%	▇ 2.5%	at its level

other tests14 not in a task · 41% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Redux (ZeroEval, 2,778 questions)	82.8% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 38/53	easy · avg 77%	▇▇▇▇▇▇▇ 17%	typical
ZebraLogic (ZeroEval, 1,000 puzzles)	19.4% ✎	▁▁▃▆██▁▁·▁▁▁ beats 28/57	hard · avg 18%	▇▇▇ 8.6%	sharp · at its level
GPQA Diamond	45.9% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 51/208	hard · avg 40%	▇▇▇ 6.5%	sharp · at its level
AHELM Speech Robust Bench (LibriSpeech-clean with noise) transcription WER	21.3% ✎	▂·█··▆▄·▂▂·▄ beats 6/13	avg 33%	▇▇ 4.1%	typical
AHELM CoRe-Bench conversational audio reasoning (pseudo-exact match)	65.9% ✎	▂·▄··▂▂·▄▂█▄ beats 6/13	medium · avg 53%	▇ 1.4%	at its level
AHELM LibriSpeech transcription by audio LLMs (WER)	7.1% ✎	▃···▃·█▃▅▃▅█ beats 7/13	avg 11%	▇ 0.9%	blunt
AHELM VoxCeleb2 same-speaker detection (exact match)	52.4% ✎	▂·▂·▂█▂▄▄··▄ beats 3/13	very hard · avg 54%	▇ 0.7%	sharp · noisy
AHELM MUStARD audio sarcasm detection (exact match)	56.4% ✎	▂·······▆█▆▆ beats 3/13	very hard · avg 57%	▇ 0.6%	blunt

6 more tests

test	result	vs the others in its run	difficulty	counts	why
AHELM AIR-Bench Foundation music subsets (exact match)	51.1% ✎	▃█▃▅▃·▃·▃▅▃▃ beats 2/13	medium · avg 57%	▇ 0.4%	blunt · at its level
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	90.9% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 27/140	very easy · avg 94%	▇ 0.2%	blunt · maxed out · noisy
AHELM MELD audio emotion classification (pseudo-exact match)	46.9% ✎	▃··▃▃▃·▃▅▅█▅ beats 8/13	medium · avg 40%	▇ 0.2%	blunt · at its level
AHELM FLEURS 7-language transcription by audio LLMs (WER)	46.3% ✎	▄▂···▂▆▂█▂·▂ beats 8/13	avg 80%	▇ 0.1%	blunt · noisy
AHELM Multilingual LibriSpeech 7-language transcription by audio LLMs (WER)	7.3% ✎	▂▂▂··▂▂·▂▂█▆ beats 7/13	avg 13%	▇ 0.0%	blunt · noisy
AHELM VocalSound human-sound recognition (prefix exact match)	49.2% ✎	██▄▄·█·█·█▄▄ beats 2/13	easy · avg 67%	▇ 0.0%	blunt · at its level · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · WildEval ZeroEval published r… 47%, Epoch AI — internal evaluatio… 20%, AHELM: A Holistic Evaluation… 8% and 6 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47%	Apache-2.0
Epoch AI — internal evaluations	3	▇▇▇▇▇▇ 20%	CC BY 4.0
AHELM: A Holistic Evaluation of Audio-Language Models (Stanford CRFM, arXiv:2508.21376v2)	10	▇▇▇ 8%	CC BY 4.0
CharXiv leaderboard (princeton-nlp/CharXiv README)	2	▇▇ 8%	Apache-2.0
LMArena leaderboard dataset	2	▇▇ 7%	CC BY 4.0
VHELM: A Holistic Evaluation of Vision Language Models (Stanford CRFM), arXiv 2410.07112v2	2	▇▇ 5%	CC BY 4.0
WebApp1K models leaderboard (onekq-ai)	1	▇ 2%	MIT
SciCode leaderboard (repository README)	1	▇ 2%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 53.1