Gemma 3 12B

+ contribute

Gemma 3 12B is an open-weights language model from Google, released March 2025. Its published size is 12.2B parameters and a 131k-token context window. Licence: gemma.

51.6±0.6#317 of 719 LLMsfrom 56 benchmark results across 16 sources

$ how it compares

The LLMs ranked either side of Gemma 3 12B, out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
383	Nemotron-4 340B Instruct NVIDIA	52±2	comparison only		2024-06
384	Aya Expanse 32B Cohere	52±2	api $0.75/M +local	~22GB	2024-10
385	Gemma 3 12B Google	52±1	api $0.075/M +local	~9.1GB	2025-03
386	Qwen2.5 Coder 7B Instruct Alibaba	52±1	local	~6GB	2024-09
387	Qwen2.5 14B Instruct Alibaba	52±1	local	~10GB	2024-09
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #250 of 300 * · 2.3% of its score · 2 tests

Beats DeepSeek R1 0528 Qwen3 8B, Magistral Small 1.0 (2506), Llama 4 Scout (109B-A17B) here, though they rank higher overall. Trails Granite 4.0 H 1B, Granite 4.0 1B, SmolLM3 3B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BFCL V4 (December 2025 archive)	30.4% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 31/77	hard · avg 32%	▇ 1.6%	at its level
BFCL v3	50.6% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 36/114	medium · avg 52%	▇ 0.8%	blunt · at its level

coding▁▁▁▃▆██▇▆▄▃▁ #293 of 458 · 9.0% of its score · 5 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, WizardLM 70B V1.0, Yi 1.5 9B Chat here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval lab	85.4% ✎	▁▁▁▁▁▃▅▅█▃▂▁ beats 177/190	easy · avg 85%	▇ 2.7%	sharp
LiveCodeBench (version unspecified) lab	24.6% ✎	▁···▁▁▃▅▅█▄▅ beats 3/37	hard · avg 25%	▇ 2.0%	at its level
VisEditBench strict pass rate (1,395 visualization-code editing tasks)	35.0% ✎	▄··█·▄··▄█·▄ beats 1/7	hard · avg 34%	▇ 1.8%	at its level
MBPP lab	73.0% ✎	▁▁▁▄▅█▄▅▃▁▂▁ beats 69/81	easy · avg 72%	▇ 1.8%	at its level
BIRD-SQL (dev) lab	47.9% ✎	▂·······▂▂█▆ beats 4/11	medium · avg 44%	▇ 0.8%	at its level

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #168 of 393 * · 4.0% of its score · 2 tests

Beats MiniCPM5 2B, Granite 4.2 30B, Seed-OSS 36B Instruct here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LMArena (text)	1341.8 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 179/379	avg 1268	▇ 3.4%	blunt
IFEval (aggregation unspecified) lab	88.9% ✎	▁▁▃▁▂▄▄▆█▃▂▁ beats 105/141	very easy · avg 87%	▇ 0.6%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #203 of 398 · 15% of its score · 6 tests

Beats DeepSeek V3 (Dec '24), GPT-4o (2024-08-06), Command A (03-2025) here, though they rank higher overall. Trails Olmo 3 7B Instruct, Phi-4-mini-reasoning 3.8B, DeepSeek R1 Distill Qwen 7B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
GSM8K +1	89.7% ✎	█····█··▄·▄█ beats 7/7	very easy · avg 86%	▇▇ 5.4%	sharp · several runs
MATH (subset unspecified) +1	83.8% ✎	█▄··▄█▄····▄ beats 4/7	easy · avg 81%	▇▇ 4.1%	several runs
PolyMath (Marco evaluation)	35.5% ✎	▄··▄██··▄··▄ beats 6/7	hard · avg 30%	▇ 1.6%	at its level
OTIS mock AIME 2024–2025	16.7% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 56/193	very hard · avg 8.5%	▇ 1.4%	sharp · noisy
MGSM (Marco evaluation)	84.0% ✎	▄·▄·▄▄▄·▄··█ beats 6/7	easy · avg 70%	▇ 1.4%	at its level
MathVista (testmini) lab	62.9% ✎	▁··▁▄▄▇█▅▆█▄ beats 19/47	medium · avg 61%	▇ 1.4%	at its level · small run

vision & documents▁▁▃▃▁▃▄▂▆▇█▄ #124 of 147 · 7.2% of its score · 9 tests

Trails Qwen3.5 2B, Pixtral 12B (2409), Qwen3-VL 2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
VisEditBench strict pass rate (1,395 visualization-code editing tasks)	35.0% ✎	▄··█·▄··▄█·▄ beats 1/7	hard · avg 34%	▇ 1.8%	at its level
MathVista (testmini) lab	62.9% ✎	▁··▁▄▄▇█▅▆█▄ beats 19/47	medium · avg 61%	▇ 1.4%	at its level · small run
MMMU (val) lab	59.6% ✎	▁····▁▁▄█▇▆▆ beats 22/52	medium · avg 58%	▇ 1.4%	at its level · small run
InfographicVQA lab	64.9% ✎	▃·▃▅█▃▃▃▃▃▃▃ beats 2/13	easy · avg 68%	▇ 1.0%	at its level · small run
VQAv2 (val) lab	71.6% ✎	▃▅▃█·▃▅·▃·▃▅ beats 6/13	easy · avg 73%	▇ 0.5%	blunt · at its level · small run
ChartQA lab	75.7% ✎	▁··▁▁▁▁▁▃▅▅█ beats 9/44	easy · avg 79%	▇ 0.4%	blunt · small run
DocVQA lab	87.1% ✎	▁··▃▁▁▁▇█▇▄▅ beats 2/28	very easy · avg 90%	▇ 0.3%	blunt · small run
TextVQA lab	67.7% ✎	▁▁▁▄▅▁▃·▃▃▃█ beats 4/24	easy · avg 75%	▇ 0.2%	blunt · small run

1 more test

test	result	vs the others in its run	difficulty	counts	why
AI2D test lab	84.2% ✎	▁▁▅▇▃▇·▄·█·▃ beats 15/28	easy · avg 84%	▇ 0.1%	blunt · small run

other tests28 not in a task · 65% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond +2	39.5% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 37/208	hard · avg 38%	▇▇▇▇▇▇ 14%	sharp · several runs
MMLU +1	76.2% ✎	▄···▄▄·▄█··█ beats 4/7	easy · avg 75%	▇▇▇ 6.7%	at its level · several runs
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)	0.4379 ✎	▁▄▃▃▇▄█▃▅▁▁▁ beats 8/30	avg 0.423	▇▇ 5.3%	sharp
MMLU-Pro +1	55.8% ✎	▄····█·▄█·▄▄ beats 3/7	medium · avg 56%	▇▇ 4.5%	at its level · several runs
BIG-Bench Hard lab	85.7% ✎	▁▁▁▁▂▄▅▅█▃▄▁ beats 120/129	easy · avg 81%	▇▇ 3.9%	sharp
Medmarks-V weighted mean win rate (61 models, 71 configurations)	43.8% ✎	▂▃▅▂▆▃█▅█▅▃▆ beats 8/34	medium · avg 42%	▇ 2.6%	blunt · at its level
MMLU-ProX-Lite (Marco)	51.3% ✎	█··▄··█··█·▄ beats 5/7	medium · avg 43%	▇ 2.5%	at its level
Global-MMLU-Lite lab	69.5% ✎	▂···▂·▃▂▂·██ beats 2/12	medium · avg 65%	▇ 2.4%	at its level

20 more tests

test	result	vs the others in its run	difficulty	counts	why
SimpleQA lab	6.3% ✎	▄▄▄▆█▆███·▂█ beats 6/32	very hard · avg 7.4%	▇ 2.3%	typical
Polish medical board exams, bias-mitigated MCQ protocol (7-exam average)	53.3% ✎	▄·▂▂▂··█▂▄·▄ beats 5/13	hard · avg 48%	▇ 2.3%	at its level
MIMIC-CDM full information (4 pathologies)	74.4% ✎	▄·······▄█·▄ beats 2/4	easy · avg 73%	▇ 1.8%	typical
GlobalMMLU (Marco evaluation)	69.2% ✎	▃··▃▃··▅···█ beats 6/7	medium · avg 57%	▇ 1.8%	at its level
MIMIC-CDM clinical decision making (4 pathologies)	66.2% ✎	█······█··██ beats 2/3	medium · avg 62%	▇ 1.8%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	37.3% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 56/97	hard · avg 32%	▇ 1.8%	blunt · at its level
MMMLU (Marco evaluation)	69.4% ✎	▄···▄·█·▄·▄█ beats 6/7	medium · avg 60%	▇ 1.5%	at its level
Big Bench Extra Hard micro avg lab	16.3% ✎	▄▄·█▄·▄·▄·▄▄ beats 2/8	hard · avg 19%	▇ 1.5%	small run
Open Persian LLM Leaderboard v2 average (21 tasks)	61.7% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 55/81	medium · avg 58%	▇ 1.5%	blunt · at its level
AGIEval (Marco suite unspecified)	43.6% ✎	▄··▄█▄··▄··█ beats 1/7	medium · avg 47%	▇ 1.2%	blunt · at its level
RULER 128k lab	57.1% ✎	▃··▅▄▇▅█▇·▃▃ beats 8/32	medium · avg 61%	▇ 1.2%	at its level · small run
FACTS Grounding lab	75.8% ✎	▄········▄·█ beats 3/3	easy · avg 73%	▇ 1.2%	typical
Open PT-LLM Leaderboard average (9 Portuguese tasks)	76.4% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 233/276	medium · avg 73%	▇ 0.7%	at its level · noisy
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	80.9% ✎	▃·▁▁▂▁▂▅█▇▄▁ beats 75/100	medium · avg 68%	▇ 0.6%	at its level · noisy
WMT24++ (chrF) lab	51.6% ✎	█······█··██ beats 2/3	medium · avg 53%	▇ 0.6%	blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	85.0% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 99/146	easy · avg 83%	▇ 0.4%	noisy
WMT24++ lab	81.6% ✎	█···█·█··███ beats 2/5	easy · avg 72%	▇ 0.2%	blunt · at its level
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	97.2% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 110/140	very easy · avg 94%	▇ 0.2%	blunt · maxed out · noisy
Chess Puzzles (Epoch)	0.0% ✎	█▁▁▁▁▂▁▄▂▂▁▁ beats 28/135	very hard · avg 5.9%	▇ 0.2%	sharp · too hard for it · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)	95.6% ✎	▁▂▁▂▅█▂▃▄▃▃▁ beats 94/98	very easy · avg 90%	▇ 0.0%	blunt

$ where to run it

Through an API

api · DeepInfra · $0.075/M · $0.05 in / $0.15 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceUSD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-3-12b-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Gemma 3 12B · needs ~7.3 GB · from huggingface.co/ggml-org
chat in the terminal
llama cli -hf ggml-org/gemma-3-12b-it-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf ggml-org/gemma-3-12b-it-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Gemma 3 12B · needs ~7.3 GB · from huggingface.co/ggml-org
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/ggml-org/gemma-3-12b-it-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Gemma 3 12B · needs ~7.82 GB · from ollama.com
chat in the terminal
ollama run gemma3:12b

MLX (Mac)

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Gemma 3 12B · needs ~7.2 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/gemma-3-12b-it-4bit-DWQ
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/gemma-3-12b-it-4bit-DWQ

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Google DeepMind Gemma 3 270M… 32%, Marco-MoE: Open Multilingual… 28%, Medmarks: A Comprehensive Ope… 8% and 13 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Google DeepMind Gemma 3 270M IT model card · lab-reported	19	▇▇▇▇▇▇▇▇▇▇ 32%	CC-BY-4.0
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling	11	▇▇▇▇▇▇▇▇▇ 28%	CC BY 4.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models	2	▇▇ 8%	CC BY 4.0
Gemma Team — Gemma 3 Technical Report · lab-reported	6	▇▇ 7%	CC BY 4.0
Epoch AI — internal evaluations	3	▇▇ 6%	CC BY 4.0
MIMIC Clinical Decision Making leaderboard	2	▇ 4%	Apache-2.0
LMArena leaderboard dataset	1	▇ 3%	CC BY 4.0
Berkeley Function Calling Leaderboard result archive	2	▇ 2%	Apache-2.0
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? (arXiv:2606.12250v1)	1	▇ 2%	CC BY-SA 4.0
VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? (arXiv:2608.10408v1)	1	▇ 2%	CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 2%	MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 1%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
TranslateGemma Technical Report (Google), arXiv 2601.09012v2 · lab-reported	1	▇ 0%	CC BY 4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	2	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 51.6

$ quantization evidence · 4 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
NVFP4 (weights and activations; checkpoint not named)	GSM8K chain-of-thought exact match (1,319 problems; lm-eval v0.4.9, vLLM 0.12)	0.825 → 0.789	95.7%	source ↗
NVFP4 (weights and activations; checkpoint not named)	MMLU 5-shot (500-example subset; lm-eval v0.4.9, vLLM 0.12)	0.62 → 0.58	93.4%	source ↗
W4A16 (RedHatAI gemma-3-12b-it-quantized.w4a16)	GSM8K chain-of-thought exact match (1,319 problems; lm-eval v0.4.9, vLLM 0.12)	0.825 → 0.82	99.4%	source ↗
W4A16 (RedHatAI gemma-3-12b-it-quantized.w4a16)	MMLU 5-shot (500-example subset; lm-eval v0.4.9, vLLM 0.12)	0.62 → 0.59	95.2%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json