Gemma 3 27B

+ contribute

Gemma 3 27B is an open-weights language model from Google, released March 2025. Its published size is 27.4B parameters and a 131k-token context window. Licence: gemma.

52.7±0.5#303 of 719 LLMsfrom 85 benchmark results across 29 sources

$ how it compares

The LLMs ranked either side of Gemma 3 27B, out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
362	GPT-4.1 nano OpenAI	53±1	api $0.17/M		2025-04
363	Gemini 1.5 Flash (002) Google	53±1	comparison only		2024-09
364	Gemma 3 27B Google	53±1	api $0.10/M +local	~18GB	2025-03
365	Phi-4 14B Microsoft	53±1	api $0.087/M +local	~10GB	2024-12
366	Gemma 4 E4B Google	53±1	local	~6.9GB	2026-04
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #236 of 300 · 7.0% of its score · 7 tests

Beats o3-mini, Ministral 3 14B Reasoning 2512, DeepSeek R1 0528 Qwen3 8B here, though they rank higher overall. Trails Granite 4.0 1B, SmolLM3 3B, Phi-3.5-mini-instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Tau2 – airline lab	39.0% ✎	▃·▃··▃·····█ beats 1/5	medium · avg 42%	▇ 1.5%	at its level
τ²-bench Telecom lab	3.1% ✎	▄····█··▄·▄▄ beats 0/5	very hard · avg 14%	▇ 1.2%	sharp · noisy
τ²-bench aggregate lab	16.2% ✎	▄·▄··▄····█▄ beats 0/5	hard · avg 32%	▇ 1.2%	at its level
Terminal-Bench Hard lab	4.0% ✎	██··█·█·█··█ beats 1/5	very hard · avg 4.7%	▇ 1.1%	sharp · too hard for it
BFCL V4 (December 2025 archive)	29.5% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 29/77	hard · avg 32%	▇ 1.0%	at its level
Tau2 – retail lab	6.6% ✎	▄····▄··▄·▄█ beats 0/5	medium · avg 40%	▇ 0.5%	sharp · at its level · noisy
BFCL v3	59.1% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 80/114	medium · avg 52%	▇ 0.5%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #288 of 458 · 14% of its score · 11 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, WizardLM 70B V1.0, Yi 1.5 9B Chat here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Aider polyglot (225 exercises, two attempts)	4.9% ✎	▃·▂▃▅▂▃█▆▄▂▂ beats 1/49	very hard · avg 8.5%	▇ 3.1%	sharp
HumanEval lab	87.8% ✎	▁▁▁▁▁▃▅▅█▃▃▁ beats 179/190	easy · avg 85%	▇ 1.6%	sharp
Luau Leetcode fully solved (Torpedo Software)	16.3% ✎	▁▁▃▁·▁▂▃▅▃█▇ beats 3/49	hard · avg 18%	▇ 1.4%	typical
LiveCodeBench (version unspecified) lab	29.7% ✎	▁···▁▁▃▅▅█▄▅ beats 5/37	hard · avg 25%	▇ 1.4%	at its level
VisEditBench strict pass rate (1,395 visualization-code editing tasks)	43.0% ✎	▄··█·▄··▄█·▄ beats 3/7	hard · avg 34%	▇ 1.2%	at its level
WebApp1K-Duo-React pass@1 (onekq-ai)	19.8% ✎	▁··▃·▁▁▁▆█▇█ beats 1/45	hard · avg 32%	▇ 1.1%	at its level
MBPP lab	74.4% ✎	▁▁▁▄▅█▄▅▃▁▂▁ beats 73/81	easy · avg 72%	▇ 1.1%	typical
WebApp1K-React pass@1 (onekq-ai)	65.5% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 15/31	medium · avg 44%	▇ 1.1%	sharp · at its level · noisy

3 more tests

test	result	vs the others in its run	difficulty	counts	why
SciCode lab	21.0% ✎	█▄·······▄▄▄ beats 0/5	hard · avg 21%	▇ 0.9%	typical
LiveCodeBench v6 (Feb–May 2025) lab	29.1% ✎	▄··▄·▄···▄·█ beats 0/5	medium · avg 43%	▇ 0.8%	at its level
BIRD-SQL (dev) lab	54.4% ✎	▂·······▂▂█▆ beats 8/11	medium · avg 44%	▇ 0.5%	at its level

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #145 of 393 · 7.3% of its score · 6 tests

Beats INTELLECT-3, Nemotron 3 Super 120B-A12B, MiniMax-M2 here, though they rank higher overall. Trails LFM2.5 1.2B Thinking, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1365.4 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 210/379	avg 1268	▇ 2.2%	blunt
Arena-Hard v2.0 hard prompts (style control, Gemini-2.5 judge)	15.0% ✎	▂·▃▅▅·▃█▆▂▂▂ beats 5/22	very hard · avg 12%	▇ 2.1%	sharp
AlpacaEval 2 length-controlled	63.4% ✎	▃▃·▁▁█▇▅·▃·▁ beats 21/22	medium · avg 42%	▇ 1.6%	at its level
Arena-Hard v2.0 creative writing (GPT-4.1 + Gemini-2.5 judges)	69.9% ✎	▆▂▂▆▂▂▃█▃▃·▂ beats 19/22	hard · avg 19%	▇ 0.5%	sharp · at its level · noisy
IFEval lab	90.4% ✎	▄··▄··█···▄▄ beats 0/5	very easy · avg 94%	▇ 0.5%	maxed out
IFEval prompt-level loose	83.4% ✎	▁▃▁▃▁▄▁▁█▃▃▁ beats 16/22	easy · avg 80%	▇ 0.4%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #196 of 398 · 15% of its score · 8 tests

Beats MiniMax-M2.1, Qwen3 235B-A22B (Apr 2025, hybrid), DeepSeek V3 (Dec '24) here, though they rank higher overall. Trails Phi-4-mini-reasoning 3.8B, DeepSeek R1 Distill Qwen 7B, OpenThinker2 7B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	74.0% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 63/92	hard · avg 32%	▇▇ 5.6%	sharp · at its level
MATH (subset unspecified) +2	76.2% ✎	▃▃█▂▂▆·····▁ beats 20/22	easy · avg 83%	▇ 2.4%	several runs
GSM8K +1	91.1% ✎	▂▃··▃▂▅█▆▃▂▃ beats 19/22	very easy · avg 93%	▇ 2.1%	sharp · maxed out · several runs
MATH-Vision lab	46.0% ✎	██·█·····███ beats 0/5	medium · avg 53%	▇ 1.3%	at its level
AIME 2026 lab	20.8% ✎	▄·▄▄····▄··█ beats 0/5	medium · avg 41%	▇ 1.3%	sharp · at its level · noisy
OTIS mock AIME 2024–2025	22.5% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 60/193	very hard · avg 8.5%	▇ 1.1%	sharp · noisy
MathVista (testmini) lab	67.6% ✎	▁··▁▄▄▇█▅▆█▄ beats 22/47	medium · avg 61%	▇ 0.9%	at its level · small run
OMEGA (math generalisation)	24.0% ✎	▅···▃▃···▃▃█ beats 3/8	hard · avg 24%	▇ 0.7%	at its level

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #107 of 147 · 11% of its score · 14 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4 Turbo (2024-04-09), Molmo 2 8B here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Arena vision (style controlled)	1159.8 ✎	▁▁▂▂▂▂▄▅▅▆██ beats 54/145	avg 1068	▇ 1.6%	blunt
MMMU Pro lab	49.7% ✎	██·█····█·██ beats 1/5	medium · avg 49%	▇ 1.4%	at its level
InfographicVQA lab	70.6% ✎	▄█·······█·▄ beats 2/5	easy · avg 71%	▇ 1.4%	at its level
MATH-Vision lab	46.0% ✎	██·█·····███ beats 0/5	medium · avg 53%	▇ 1.3%	at its level
VisEditBench strict pass rate (1,395 visualization-code editing tasks)	43.0% ✎	▄··█·▄··▄█·▄ beats 3/7	hard · avg 34%	▇ 1.2%	at its level
MMMU (val) lab	64.9% ✎	▁····▁▁▄█▇▆▆ beats 29/52	medium · avg 58%	▇ 0.9%	at its level · small run
MathVista (testmini) lab	67.6% ✎	▁··▁▄▄▇█▅▆█▄ beats 22/47	medium · avg 61%	▇ 0.9%	at its level · small run
OmniDocBench 1.5 ↓ lab	36.5% ✎	█·█·····████ beats 0/5	avg 31%	▇ 0.5%	blunt

6 more tests

test	result	vs the others in its run	difficulty	counts	why
VMMU (Vietnamese multimodal understanding and reasoning, mean accuracy over 7 tasks)	39.1% ✎	▅······▃·▅·█ beats 4/7	very hard · avg 36%	▇ 0.5%	noisy
VQAv2 (val) lab	71.0% ✎	▃▅▃█·▃▅·▃·▃▅ beats 5/13	easy · avg 73%	▇ 0.3%	blunt · small run
ChartQA lab	78.0% ✎	▁··▁▁▁▁▁▂▅▅█ beats 11/44	easy · avg 79%	▇ 0.3%	blunt · small run
DocVQA lab	86.6% ✎	▁··▃▁▁▁▇█▇▄▅ beats 1/28	very easy · avg 90%	▇ 0.2%	blunt · small run
TextVQA lab	65.1% ✎	▁▁▃▄▅▁▃·▃▃▃█ beats 2/24	easy · avg 75%	▇ 0.1%	blunt · small run
AI2D test lab	84.5% ✎	▁▁▅▇▃▇·▄·█·▃ beats 17/28	easy · avg 84%	▇ 0.1%	blunt · small run

other tests36 not in a task · 49% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +1	47.7% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 61/208	hard · avg 40%	▇▇ 6.1%	sharp · at its level · several runs
BIG-Bench Hard +1	83.7% ✎	▁▄·▁▁·▅█·▄▄▁ beats 20/22	easy · avg 75%	▇▇ 5.1%	sharp · at its level · several runs
MMLU +1	81.8% ✎	▁▂·▁·▁▁█·▃▃▅ beats 16/22	easy · avg 77%	▇▇ 4.3%	at its level · several runs
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)	0.4610 ✎	▁▄▃▃▇▄█▃▅▁▁▁ beats 12/30	avg 0.423	▇ 3.4%	sharp
MMLU-Pro lab	67.6% ✎	▄··█···▄··▄▄ beats 1/5	medium · avg 61%	▇ 1.9%	at its level
MMMLU lab	70.7% ✎	██··█···█·██ beats 1/5	medium · avg 69%	▇ 1.7%	at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)	46.1% ✎	▂▃▅▂▆▃█▅█▅▃▆ beats 11/34	medium · avg 42%	▇ 1.7%	blunt · at its level
SimpleQA lab	10.0% ✎	▄▄▄▆█▆███·▂█ beats 11/32	very hard · avg 7.4%	▇ 1.6%	typical

28 more tests

test	result	vs the others in its run	difficulty	counts	why
Global-MMLU-Lite lab	75.1% ✎	▂···▂·▂▃▂·██ beats 3/12	medium · avg 65%	▇ 1.6%	at its level
Polish medical board exams, bias-mitigated MCQ protocol (7-exam average)	59.2% ✎	▄·▂▂▂··█▃▄·▄ beats 9/13	hard · avg 48%	▇ 1.5%	at its level
CPTUB average (SpeakLeash, 0-5 judge scale)	76.1% ✎	▁▂▂▂▃█▃▆▅▄▂▃ beats 72/95	easy · avg 66%	▇ 1.4%	at its level
RobloxQA v2.0 multiple choice (Torpedo Software)	84.5% ✎	▁▁▃▁█▄▃▃▆▂▂▂ beats 4/49	easy · avg 78%	▇ 1.4%	at its level
EQ-Bench 3 (emotional intelligence, judge Elo)	1159.4 ✎	▁·▂▁▄▅▅▃█▄▁▁ beats 22/62	avg 1044	▇ 1.4%	typical
IFBench lab	32.0% ✎	▄▄·▄······▄█ beats 0/5	medium · avg 40%	▇ 1.2%	at its level
KCL-Essay vanilla	23.6% ✎	▁···▁▃▂█▂▃▃▂ beats 4/23	hard · avg 18%	▇ 1.2%	typical
BRIDGE clinical text, zero-shot average (87 tasks)	39.9% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 74/97	hard · avg 32%	▇ 1.2%	blunt · at its level
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	50.3% ✎	▁▁▁▁·▂·█▆█▇▁ beats 8/31	medium · avg 48%	▇ 1.2%	at its level
MIMIC-CDM full information (4 pathologies)	75.5% ✎	▄·······▄█·▄ beats 3/4	easy · avg 73%	▇ 1.2%	typical
MIMIC-CDM clinical decision making (4 pathologies)	70.5% ✎	█······█··██ beats 3/3	medium · avg 62%	▇ 1.1%	at its level
MRCR v2 8-needle, 128k lab	13.5% ✎	▄▄·▄···█···▄ beats 0/5	hard · avg 22%	▇ 1.1%	at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	64.6% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 62/81	medium · avg 58%	▇ 0.9%	blunt · at its level
Big Bench Extra Hard micro avg lab	19.3% ✎	█··▄···▄·▄·▄ beats 0/5	hard · avg 28%	▇ 0.9%	at its level
RULER 128k lab	66.0% ✎	▃··▅▄▇▅█▇·▃▃ beats 12/32	medium · avg 61%	▇ 0.8%	at its level · small run
FACTS Grounding lab	74.9% ✎	▄········▄·█ beats 2/3	easy · avg 73%	▇ 0.7%	typical
PopQA	30.9% ✎	▁▁▁▃▃▃▁█▁▃·▄ beats 15/21	hard · avg 32%	▇ 0.6%	blunt · at its level
LongSocialBench (full-context, 4-option MCQ over discussion threads)	36.5% ✎	▆▂·▄█·▂▂·▄▄▂ beats 2/16	very hard · avg 36%	▇ 0.5%	blunt
KCL-MCQA vanilla	35.1% ✎	▂···█▂▆▆██▆▂ beats 7/23	very hard · avg 30%	▇ 0.5%	typical
Open PT-LLM Leaderboard average (9 Portuguese tasks)	78.7% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 247/276	medium · avg 73%	▇ 0.5%	at its level · noisy
TruthfulQA MC2	63.9% ✎	▃▃▂·▃▃·▅█▂▅▂ beats 12/21	medium · avg 63%	▇ 0.5%	blunt · at its level
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	82.1% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 77/100	medium · avg 68%	▇ 0.4%	at its level · noisy
WMT24++ (chrF) lab	53.4% ✎	█······█··██ beats 3/3	medium · avg 53%	▇ 0.4%	blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	92.1% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 126/146	easy · avg 83%	▇ 0.2%	noisy
WMT24++ lab	83.1% ✎	█···█·█··███ beats 3/5	easy · avg 72%	▇ 0.1%	blunt · at its level
Chess Puzzles (Epoch)	0.0% ✎	█▁▁▁▁▂▁▄▂▂▁▁ beats 20/135	very hard · avg 5.9%	▇ 0.1%	sharp · too hard for it · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	97.0% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 105/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)	92.6% ✎	▁▂▁▂▅█▂▃▄▃▁▁ beats 67/98	very easy · avg 90%	▇ 0.0%	blunt

$ where to run it

Through an API

api · DeepInfra · $0.10/M · $0.08 in / $0.16 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceUSD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-3-27b-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Gemma 3 27B · needs ~16.5 GB · from huggingface.co/ggml-org
chat in the terminal
llama cli -hf ggml-org/gemma-3-27b-it-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf ggml-org/gemma-3-27b-it-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Gemma 3 27B · needs ~16.5 GB · from huggingface.co/ggml-org
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/ggml-org/gemma-3-27b-it-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Gemma 3 27B · needs ~16.94 GB · from ollama.com
chat in the terminal
ollama run gemma3:27b

MLX (Mac)

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Gemma 3 27B · needs ~15.2 GB, Mac only · from huggingface.co/Outlier-Ai
chat in the terminal
mlx_lm.chat --model Outlier-Ai/gemma-3-27b-it-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model Outlier-Ai/gemma-3-27b-it-MLX-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Gemma Team — Gemma 4 Technica… 22%, Google DeepMind Gemma 3 270M… 16%, Epoch AI — internal evaluatio… 10% and 26 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Gemma Team — Gemma 4 Technical Report · lab-reported	18	▇▇▇▇▇▇▇ 22%	CC BY 4.0
Google DeepMind Gemma 3 270M IT model card · lab-reported	15	▇▇▇▇▇ 16%	CC-BY-4.0
Epoch AI — internal evaluations	4	▇▇▇ 10%	CC BY 4.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models	8	▇▇▇ 10%	CC BY 4.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models	2	▇▇ 5%	CC BY 4.0
LMArena leaderboard dataset	2	▇ 4%	CC BY 4.0
Gemma Team — Gemma 3 Technical Report · lab-reported	5	▇ 3%	CC BY 4.0
Aider polyglot benchmark runs	1	▇ 3%	Apache-2.0
Roblox LLM Leaderboard (Torpedo Software): RobloxQA v2.0 and Luau Leetcode	2	▇ 3%	MIT
Arena-Hard-Auto v2.0-Preview leaderboard (repository README)	2	▇ 3%	Apache-2.0
MIMIC Clinical Decision Making leaderboard	2	▇ 2%	Apache-2.0
WebApp1K models leaderboard (onekq-ai)	2	▇ 2%	MIT
Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities	2	▇ 2%	CC BY 4.0
Berkeley Function Calling Leaderboard result archive	2	▇ 2%	Apache-2.0
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? (arXiv:2606.12250v1)	1	▇ 2%	CC BY-SA 4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 1%	MIT
EQ-Bench 3 canonical leaderboard results	1	▇ 1%	MIT
google/gemma-4-31B-it model card · lab-reported	1	▇ 1%	Apache-2.0
VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? (arXiv:2608.10408v1)	1	▇ 1%	CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	1	▇ 1%	Apache-2.0
allenai/Olmo-3.1-32B-Think model card	2	▇ 1%	Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? (arXiv:2610.04118v1)	1	▇ 1%	CC BY 4.0
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark (arXiv:2508.13680v5)	1	▇ 0%	CC-BY-4.0
Open Portuguese LLM Leaderboard (archived)	1	▇ 0%	Apache-2.0
TranslateGemma Technical Report (Google), arXiv 2601.09012v2 · lab-reported	1	▇ 0%	CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	2	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 52.7