Gemma 2 27B IT

+ contribute

Gemma 2 27B IT is an open-weights language model from Google, released June 2024. Its published size is 27.23B parameters and a 8k-token context window. Licence: gemma.

Catalogue identity notes

Comparison model from the Epoch AI internal-evaluation CSVs (Epoch release date 2024-06-24). Instruction-tuned checkpoint google/gemma-2-27b-it. 27,227,128,320 parameters (safetensors metadata). ollama gemma2:27b = 27b-instruct-q4_0 (same digest).

49.8±0.6#345 of 719 LLMsfrom 34 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of Gemma 2 27B IT, out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
413	DeepSeek-Coder 33B Instruct DeepSeek	50±1	local	~21GB	2023-11
414	Yi 34B 200K (base) 01.AI	50±5	local	~23GB	2023-11
415	Gemma 2 27B IT Google	50±1	local	~18GB	2024-06
416	Bielik 11B v2.3 Instruct SpeakLeash	50±2	local	~8.6GB	2024-08
417	Qwen2.5-VL 7B Instruct Alibaba	50±1	local	~6.4GB	2025-01
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▁▃▆██▇▆▄▃▁ #231 of 458 · 11% of its score · 5 tests

Beats Qwen3.5 4B, Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B) here, though they rank higher overall. Trails QwQ-32B-Preview here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	57.3% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 17/47	medium · avg 61%	▇▇▇ 8.2%	sharp · at its level
CodeReviewQA: change localisation (hard), invariant accuracy	58.7% ✎	█▂▁▁▂▂▂▃▅▅▅▇ beats 41/47	medium · avg 36%	▇ 0.9%	sharp · at its level · noisy
CodeReviewQA: solution identification (hard), invariant accuracy	65.7% ✎	▃··▁▂▂▃▂█▃▄▂ beats 35/47	medium · avg 54%	▇ 0.8%	sharp · at its level · noisy
CodeReviewQA: automated code refinement, exact match rate	46.4% ✎	▁▁▁▁▁▁▂▂▃█▅▄ beats 44/47	hard · avg 33%	▇ 0.6%	at its level
CodeReviewQA: change type recognition, invariant accuracy	74.0% ✎	▁▁▁······▁▃█ beats 28/47	easy · avg 67%	▇ 0.0%	blunt · at its level · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #236 of 393 · 19% of its score · 6 tests

Beats Mistral Small 4 (119B-A6.5B), Ministral 3 14B Reasoning 2512, Qwen2.5 Coder 32B Instruct here, though they rank higher overall. Trails Flan-T5 XXL, LFM2.5 1.2B Instruct, Gemma 3 4B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	57.5% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 40/60	medium · avg 52%	▇▇▇ 8.1%	sharp · at its level
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)	1232.0 ✎	▂▁▂▁▃▇█▆▇▄▃▂ beats 89/95	avg 1201	▇ 2.9%	blunt
LMArena (text)	1289.4 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 122/379	avg 1268	▇ 2.9%	blunt
Arabic IFEval prompt-level (Inception)	59.4% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 33/38	medium · avg 49%	▇ 2.5%	sharp · at its level
AlpacaEval 2 length-controlled	49.0% ✎	▃▃·▁▁█▇▅·▁·▁ beats 19/22	medium · avg 42%	▇ 2.0%	at its level
IFEval prompt-level loose	63.2% ✎	▁▃▁▃▁▄▁▁█▃▃▁ beats 5/22	easy · avg 80%	▇ 0.6%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #300 of 398 · 19% of its score · 7 tests

Beats GPT-4 (0613), Athene-70B here, though they rank higher overall. Trails Gemma 7B (base), DeepSeek R1 Distill Qwen 1.5B, CodeGemma 7B (v1.0) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	27.9% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 26/92	hard · avg 32%	▇▇▇ 6.3%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	26.6% ✎	▁▁▁▆█▂▇▇▄··▂ beats 16/38	hard · avg 34%	▇▇ 4.0%	sharp · at its level
GSM8K (ZeroEval, zero-shot)	90.2% ✎	▁▄·▁▄█▅▅▅███ beats 22/43	very easy · avg 89%	▇ 3.5%	sharp
MATH (subset unspecified) +1	35.1% ✎	▃▃█▂▂▆·····▁ beats 5/22	medium · avg 61%	▇ 1.7%	at its level · several runs
GSM8K	80.7% ✎	▂▃··▃▂▅█▆▃▂▃ beats 8/22	very easy · avg 88%	▇ 1.5%	sharp
OTIS mock AIME 2024–2025	1.4% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 11/193	very hard · avg 8.5%	▇ 0.9%	sharp · noisy
OMEGA (math generalisation)	9.1% ✎	▅···▃▃···▃▃█ beats 0/8	hard · avg 24%	▇ 0.8%	typical

other tests15 not in a task · 51% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU-Redux (ZeroEval, 2,778 questions)	75.7% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 21/53	easy · avg 77%	▇▇▇▇▇ 13%	at its level
LiveBench 2024-07-26	41.3% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 46/61	medium · avg 41%	▇▇▇▇▇ 12%	at its level
ZebraLogic (ZeroEval, 1,000 puzzles)	16.3% ✎	▁▁▃▆██▁▁·▁▁▁ beats 21/57	hard · avg 18%	▇▇ 4.3%	sharp
BIG-Bench Hard	72.7% ✎	▁▄·▁▁·▅█·▄▄▁ beats 15/22	easy · avg 75%	▇▇ 3.9%	sharp · at its level
MMLU	70.7% ✎	▁▂·▁·▁▁█·▃▃▅ beats 11/22	easy · avg 77%	▇ 3.7%	at its level
GPQA Diamond	36.5% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 31/208	hard · avg 40%	▇ 3.4%	sharp
WildBench v2 WB-Score (task macro)	48.5402 WB-Score ✎	▁·▁▁·▃█▄▇▆▅▆ beats 39/52	avg 45.199 WB-Score	▇ 3.3%	sharp
EvalPerf correctness (118 tasks, 20240328)	80.2% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 18/29	easy · avg 76%	▇ 1.8%	typical

7 more tests

test	result	vs the others in its run	difficulty	counts	why
BRIDGE clinical text, zero-shot average (87 tasks)	38.2% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 61/97	hard · avg 32%	▇ 1.4%	blunt · at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	60.8% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 54/81	medium · avg 58%	▇ 1.2%	blunt · at its level
PopQA	33.9% ✎	▁▁▁▃▃▃▁█▃▃·▄ beats 16/21	hard · avg 32%	▇ 0.8%	blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)	76.3% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 232/276	medium · avg 73%	▇ 0.6%	at its level · noisy
TruthfulQA MC2	64.6% ✎	▃▃▂·▃▃·▅█▂▅▂ beats 14/21	medium · avg 63%	▇ 0.6%	blunt · at its level
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	83.0% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 78/100	medium · avg 68%	▇ 0.5%	at its level · noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	88.2% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 111/146	easy · avg 83%	▇ 0.3%	noisy

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Gemma 2 27B IT · needs ~16.6 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/gemma-2-27b-it-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/gemma-2-27b-it-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Gemma 2 27B IT · needs ~16.6 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/gemma-2-27b-it-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Gemma 2 27B IT · needs ~16.84 GB · from ollama.com
chat in the terminal
ollama run gemma2:27b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Gemma 2 27B IT · needs ~15.3 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/gemma-2-27b-it-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/gemma-2-27b-it-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · WildEval ZeroEval published r… 33%, 2 OLMo 2 Furious (Team OLMo,… 14%, LiveBench model judgments (HF… 12% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇▇▇▇▇ 33%	Apache-2.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models	8	▇▇▇▇ 14%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇▇ 12%	Apache-2.0 (stated in the LiveBench datasheet)
Epoch AI — internal evaluations	3	▇▇▇ 11%	CC BY 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇▇ 8%	Apache-2.0
WildBench v2 leaderboard data (Ai2)	1	▇ 3%	Apache-2.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)	1	▇ 3%	CC-BY-4.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	1	▇ 3%	Apache-2.0
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)	4	▇ 2%	CC-BY-4.0
EvalPlus / EvalPerf published evaluation runs	1	▇ 2%	Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
allenai/Olmo-3.1-32B-Think model card	2	▇ 1%	Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 49.8