whatmodel.sh / Gemini 1.5 Pro Experimental (0827)
Gemini 1.5 Pro Experimental (0827)
↗ share
+ contribute
Gemini 1.5 Pro Experimental (0827) is a closed-weights language model from Google, released August 2024. Licence: proprietary.
Catalogue identity notes
Comparison-only: retired or no longer listed API model, kept so its independent runs can be scored. Gemini API changelog, August 27, 2024: 'Released the following experimental models: gemini-1.5-pro-exp-0827, gemini-1.5-flash-exp-0827, gemini-1.5-flash-8b-exp-0827'; experimental, shut down. Distinct from the stable gemini-1.5-pro-002 (September 2024).
$ how it compares
The LLMs ranked either side of Gemini 1.5 Pro Experimental (0827), out of 828 with a score, then more from Google.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 286 |  Qwen2.5 Coder 32B Instruct | 57±1 | api $0.74/M +local | ||
| 287 | Qwen3 4B Thinking 2507 | 57±1 | local | ||
| 288 | Gemini 1.5 Pro Experimental (0827) | 56±1 | comparison only | ||
| 289 | Gemini 1.5 Pro (002) | 56±1 | comparison only | ||
| 290 |  Athene-V2-Chat 72B | 56±2 | local | ||
| more from Google | |||||
| 9 | Gemini 4 Argon (limited preview) | 82±1 | limited preview | ||
| 23 | Gemini 3.8 Flash | 77±1 | api $1.50/M | ||
| 32 | Gemini 3.7 Flash | 76±1 | api $1.50/M | ||
$ what it's good at
coding▁▁▁▃▅▇█▇▅▄▃▁ #231 of 524 * · 24% of its score · 2 tests
Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Falcon-H1R 7B here, though they rank higher overall. Trails QwQ-32B-Preview, Ministral 3 3B Reasoning 2512, DeepSeek-Coder-V2 Instruct (0724) here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| CRUXEval (ZeroEval, 800 problems) | 79.6% ✎ | ▁▂▆▃▆█▆▃▂·▁▁ beats 41/47 | 13% |
| Aider code editing (133 Exercism Python exercises, two attempts) | 66.9% ✎ | ▁·▁▂▄▁▆▄█▆▂▂ beats 42/62 | 11% |
math▁▂▃▅▇▅▅█▄▁▁▁ #227 of 462 * · 10% of its score · 1 test
Beats MiniMax-M2.1, Qwen3 235B-A22B (Apr 2025, hybrid), DeepSeek V3 (Dec '24) here, though they rank higher overall. Trails Phi-4-mini-reasoning 3.8B, DeepSeek R1 Distill Qwen 7B, OpenThinker2 7B here, though they rank lower overall.
| test | result | vs the others in its run | counts |
|---|---|---|---|
| MATH Level 5 (ZeroEval, 721 problems) | 68.1% ✎ | ▁▁▁▆█▂▇▇▄··▂ beats 36/38 | 10% |
other tests3 not in a task · 66% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| LiveBench 2024-07-26 | 53.8% ✎ | ▁▁▁▂▂▃▂█▆▃▅▃ beats 57/61 | 29% |
| MMLU-Redux (ZeroEval, 2,778 questions) | 86.1% ✎ | ▁·▁▁▄▄▄▅█▄▂▁ beats 47/53 | 21% |
| ZebraLogic (ZeroEval, 1,000 puzzles) | 30.5% ✎ | ▁▁▃▆██▁▁·▁▁▁ beats 48/57 | 15% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · WildEval ZeroEval published r… 60%, LiveBench model judgments (HF… 29%, Aider code editing benchmark… 11%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 WildEval ZeroEval published results | 4 | 60% |
| LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets | 1 | 29% |
 Aider code editing benchmark runs (old edit leaderboard) | 1 | 11% |
| prior · pull toward the middle | 0% | |
| weighted average | = 56.4 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json