Gemma 2 9B IT

+ contribute

Gemma 2 9B IT is an open-weights language model from Google, released June 2024. Its published size is 9.24B parameters and a 8k-token context window. Licence: gemma.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-06-24). Instruction-tuned checkpoint google/gemma-2-9b-it (not the PT base; distinct from Gemma 3). 9,241,705,984 parameters (safetensors metadata). ollama gemma2:9b = 9b-instruct-q4_0 (same digest).

47.2±0.5#393 of 719 LLMsfrom 56 benchmark results across 20 sources

$ how it compares

The LLMs ranked either side of Gemma 2 9B IT, out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
468	Bielik 4.5B v3.0 Instruct SpeakLeash	48±2	local	~4.3GB	2025-04
469	Qwen3.5 2B Alibaba	47±1	local	~2.9GB	2026-03
470	Gemma 2 9B IT Google	47±1	local	~7GB	2024-06
471	Command R Plus (08-2024) Cohere	47±1	api $4.38/M +local	~63GB	2024-08
472	GLM-4 9B Chat Zhipu AI (Z.ai)	47±2	local	~7.9GB	2024-06
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▆██▇▆▄▃▁ #262 of 458 · 7.8% of its score · 5 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, QwQ-32B-Preview, Phi-3 Medium 128K Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	46.0% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 11/47	medium · avg 61%	▇▇ 6.2%	sharp · at its level
CodeReviewQA: change localisation (hard), invariant accuracy	52.0% ✎	█▂▁▁▂▂▂▃▅▅▅▇ beats 36/47	medium · avg 36%	▇ 0.6%	sharp · at its level · noisy
CodeReviewQA: solution identification (hard), invariant accuracy	49.6% ✎	▃··▁▂▂▃▂█▃▄▂ beats 25/47	medium · avg 54%	▇ 0.6%	sharp · at its level · noisy
CodeReviewQA: automated code refinement, exact match rate	39.0% ✎	▁▁▁▁▁▁▂▂▃█▅▄ beats 38/47	hard · avg 33%	▇ 0.4%	at its level
CodeReviewQA: change type recognition, invariant accuracy	74.1% ✎	▁▁▁······▁▃█ beats 29/47	easy · avg 67%	▇ 0.0%	blunt · at its level · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #257 of 393 · 8.5% of its score · 6 tests

Beats Ministral 3 14B Reasoning 2512, Ministral 3 14B, OpenThinker2 32B here, though they rank higher overall. Trails Flan-T5 XXL, Granite 4.0 H 1B, Granite 4.0 1B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)	1204.0 ✎	▂▁▂▁▃▇█▆▇▄▂▂ beats 82/95	avg 1201	▇ 2.1%	blunt
LMArena (text)	1267.0 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 101/379	avg 1268	▇ 2.1%	blunt
Arabic IFEval prompt-level (Inception)	51.5% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 28/38	medium · avg 49%	▇ 1.8%	sharp · at its level
AlpacaEval 2 length-controlled	43.7% ✎	▃▃·▁▁█▇▅·▁·▁ beats 17/22	medium · avg 42%	▇ 1.4%	at its level
Arena-Hard (version/judge unspecified)	43.2% ✎	▃▃▂▅▃▆▅█▅▅▂▃ beats 5/30	medium · avg 58%	▇ 0.8%	at its level · small run
IFEval prompt-level loose	69.9% ✎	▁▃▁▃▁▄▁▁█▃▃▁ beats 7/22	easy · avg 80%	▇ 0.4%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #322 of 398 · 21% of its score · 8 tests

Beats Athene-70B here, though they rank higher overall. Trails Gemma 7B (base), DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 13B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
GSM8K +2	79.7% ✎	▂▃··▃▂▅█▆▃▂▃ beats 6/22	very easy · avg 91%	▇▇ 5.6%	sharp · several runs
MATH level 5	21.0% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 17/92	hard · avg 32%	▇ 3.6%	sharp
GSM8K (ZeroEval, zero-shot)	87.4% ✎	▁▄·▁▄█▅▅▅███ beats 20/43	very easy · avg 89%	▇ 3.2%	sharp
MATH Level 5 (ZeroEval, 721 problems)	19.4% ✎	▁▁▁▆█▂▇▇▄··▂ beats 7/38	hard · avg 34%	▇ 2.5%	sharp · at its level
MGSM	75.1% ✎	▅▃▃█·▅···▃·▃ beats 9/10	easy · avg 71%	▇ 2.3%	at its level
MATH (subset unspecified) +1	29.8% ✎	▃▃█▂▂▆·····▁ beats 3/22	medium · avg 60%	▇ 2.0%	at its level · several runs
MATH	51.3% ✎	█▄██···▄▄▄·▄ beats 6/10	medium · avg 64%	▇ 1.5%	at its level
OTIS mock AIME 2024–2025	0.6% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 4/193	very hard · avg 8.5%	▇ 0.4%	sharp · too hard for it · noisy

other tests29 not in a task · 63% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Redux (ZeroEval, 2,778 questions)	72.8% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 18/53	easy · avg 77%	▇▇▇▇ 10%	at its level
LiveBench 2024-07-26	31.3% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 31/61	medium · avg 41%	▇▇▇ 8.5%	at its level
MMLU +2	69.1% ✎	▁▂·▁·▁▁█·▃▃▅ beats 9/22	easy · avg 77%	▇▇▇ 7.2%	at its level · several runs
BIG-Bench Hard +1	64.9% ✎	▁▄·▁▁·▅█·▄▄▁ beats 7/22	easy · avg 75%	▇▇ 5.9%	sharp · at its level · several runs
ARC Challenge	89.8% ✎	▂▂▂█▂···▄··▂ beats 8/10	very easy · avg 91%	▇▇ 3.8%	sharp
ZebraLogic (ZeroEval, 1,000 puzzles)	12.8% ✎	▁▁▃▆██▁▁·▁▁▁ beats 16/57	hard · avg 18%	▇ 2.4%	sharp
WildBench v2 WB-Score (task macro)	42.6962 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 31/52	avg 45.199 WB-Score	▇ 2.4%	sharp
OpenBookQA	90.0% ✎	▂··▄█▂·▂···▄ beats 10/10	easy · avg 87%	▇ 2.4%	typical

21 more tests

test	result	vs the others in its run	difficulty	counts	why
MMLU-Pro	50.1% ✎	█▄▄·█▄▄▄·▄·▄ beats 7/10	medium · avg 57%	▇ 2.0%	at its level
GPQA Diamond	27.5% ✎	▁▁▃▁▂▄▆▄▅▇█▅ beats 11/208	hard · avg 40%	▇ 1.9%	sharp
GPQA (subset unspecified) +1	39.1% ✎	█··▄▂··▂▂··▄ beats 9/10	hard · avg 40%	▇ 1.8%	sharp · several runs
Multilingual MMLU (Microsoft)	63.8% ✎	█▄▄██··█···▄ beats 8/10	easy · avg 65%	▇ 1.7%	at its level
EvalPerf correctness (118 tasks, 20240328)	68.1% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 4/29	easy · avg 76%	▇ 1.5%	at its level
EQ-Bench 3 (emotional intelligence, judge Elo)	975.3 ✎	▁·▃▁▄▅▅▃█▄▁▁ beats 1/62	avg 1044	▇ 1.3%	typical
WinoGrande	74.0% ✎	▃····▃▃█▃·▅▅ beats 9/10	medium · avg 74%	▇ 1.2%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	35.1% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 44/97	hard · avg 32%	▇ 1.0%	blunt · at its level
HellaSwag	87.1% ✎	▃▅█·▃▃▃···▃▃ beats 9/10	easy · avg 84%	▇ 1.0%	at its level
BoolQ	85.7% ✎	▃··▃··██▃·▃▃ beats 9/10	easy · avg 85%	▇ 1.0%	at its level
Social IQA	74.7% ✎	▃·▃▅▅█·▃···▃ beats 5/10	easy · avg 78%	▇ 0.9%	blunt · at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	60.6% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 53/81	medium · avg 58%	▇ 0.9%	blunt · at its level
TruthfulQA MC2 +1	61.4% ✎	▃▃▂·▃▃·▅█▂▅▂ beats 9/21	medium · avg 63%	▇ 0.7%	blunt · at its level · several runs
PopQA	28.3% ✎	▁▁▁▃▃▃▁█▁▃·▄ beats 10/21	hard · avg 32%	▇ 0.5%	blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)	74.3% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 216/276	medium · avg 73%	▇ 0.5%	at its level · noisy
SurgiQ surgical MCQ (zero-shot, log-probability scoring)	43.9% ✎	▂···▂▂·▃▂▃██ beats 3/17	medium · avg 53%	▇ 0.5%	at its level
CMMLU	60.3% ✎	▅·▅····█▃▅█▅ beats 2/14	easy · avg 75%	▇ 0.4%	at its level · noisy
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	78.3% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 67/100	medium · avg 68%	▇ 0.4%	at its level · noisy
PIQA	83.7% ✎	▃·▅·▃█·▃▅··▃ beats 9/10	easy · avg 83%	▇ 0.3%	blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	84.3% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 94/146	easy · avg 83%	▇ 0.3%	at its level · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	89.9% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 24/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Gemma 2 9B IT · needs ~5.8 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/gemma-2-9b-it-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/gemma-2-9b-it-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Gemma 2 9B IT · needs ~5.8 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/gemma-2-9b-it-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Gemma 2 9B IT · needs ~6.04 GB · from ollama.com
chat in the terminal
ollama run gemma2:9b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Gemma 2 9B IT · needs ~5.2 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/gemma-2-9b-it-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/gemma-2-9b-it-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · microsoft/Phi-4-mini-instruct… 28%, WildEval ZeroEval published r… 24%, 2 OLMo 2 Furious (Team OLMo,… 11% and 17 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
microsoft/Phi-4-mini-instruct model card	16	▇▇▇▇▇▇▇▇ 28%	MIT
WildEval ZeroEval published results	5	▇▇▇▇▇▇▇ 24%	Apache-2.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models	8	▇▇▇ 11%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇ 8%	Apache-2.0 (stated in the LiveBench datasheet)
Epoch AI — internal evaluations	3	▇▇ 6%	CC BY 4.0
InternLM GitHub repository model cards (Shanghai AI Laboratory)	5	▇▇ 6%	Apache-2.0
WildBench v2 leaderboard data (Ai2)	1	▇ 2%	Apache-2.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)	1	▇ 2%	CC-BY-4.0
LMArena leaderboard dataset	1	▇ 2%	CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	1	▇ 2%	Apache-2.0
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)	4	▇ 2%	CC-BY-4.0
EvalPlus / EvalPerf published evaluation runs	1	▇ 1%	Apache-2.0
EQ-Bench 3 canonical leaderboard results	1	▇ 1%	MIT
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale (AI21, arXiv 2408.12570v1)	1	▇ 1%	CC BY-SA 4.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 0%	Apache-2.0
SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models (arXiv:2606.08071v1)	1	▇ 0%	CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 47.2