whatmodel.sh / Gemma 4 12B

Gemma 4 12B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Gemma 4 12B is an open-weights language model from Google, released June 2026. Its published size is 11.9B parameters and a 262k-token context window. Licence: apache-2.0.

62.4±0.8#172 of 719 LLMsfrom 30 benchmark results across 3 sources

$ how it compares

The LLMs ranked either side of Gemma 4 12B, out of 828 with a score, then more from Google.

#modelscorewhereRAMreleased
200Magistral Medium 1.2 (2509) Mistral63±2comparison only2025-09
201MiniMax-M1 (80k) MiniMax62±1api $0.96/M +local~959GB2025-06
202Gemma 4 12B Google62±1local~8.5GB2026-06
203GPT-4.1 OpenAI62±1api $3.50/M2025-04
204Nemotron 3.5 Lightning 30B-A3B NVIDIA62±2api $0.087/M +local~20GB2026-08
more from Google
9Gemini 4 Argon (limited preview) Google82±1limited preview2026-09
23Gemini 3.8 Flash Google77±1api $1.50/M2026-09
32Gemini 3.7 Flash Google76±1api $1.50/M2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #151 of 300 · 23% of its score · 8 tests

Beats o1 (2024-12-17), o3-mini, DeepSeek R1 0528 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, Jamba 1.5 Large here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Terminal-Bench Hard lab18.0% ✎██··█·█·█··█ beats 4/5very hard · avg 4.7%▇▇ 5.9%sharp
τ²-bench Telecom lab54.4% ✎▄····█··▄·▄▄ beats 4/5very hard · avg 14%▇▇ 4.7%sharp · at its level · noisy
Tau2 – airline lab75.0% ✎▃·▃··▃·····█ beats 3/5medium · avg 42%▇ 3.6%at its level
τ²-bench aggregate lab69.0% ✎▄·▄··▄····█▄ beats 4/5hard · avg 32%▇ 3.1%at its level
BFCL V4 (December 2025 archive)37.4% ✎▁▁▁▂▇▆▆▆█▂▄▂ beats 48/111medium · avg 35%▇ 1.4%at its level
BrowseComp10.0% ✎▅···█·▃···▃▃ beats 4/7very hard · avg 10%▇ 1.4%at its level · noisy
τ³-bench13.3% ✎▃·▃·█·▃··▃·▃ beats 5/7very hard · avg 14%▇ 1.4%blunt · at its level
Tau2 – retail lab77.6% ✎▄····▄··▄·▄█ beats 3/5medium · avg 40%▇ 1.2%sharp · at its level · noisy
coding▁▁▁▃▆██▇▆▄▃▁ #127 of 458 · 6.6% of its score · 3 tests

Beats Qwen3.5 Plus, Gemini 2.5 Pro, Gemini 3.1 Flash-Lite here, though they rank higher overall. Trails Llama 3.1 Nemotron Ultra 253B v1, Nemotron 3.5 Lightning 30B-A3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
SciCode lab38.0% ✎█▄·······▄▄▄ beats 3/5hard · avg 21%▇ 2.7%at its level
LiveCodeBench v6 (Feb–May 2025) lab72.0% ✎▄··▄·▄···▄·█ beats 3/5medium · avg 43%▇ 2.0%at its level
SciCode (level unspecified)39.8% ✎█·····██████ beats 6/6hard · avg 17%▇ 1.9%at its level
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #74 of 393 * · 0.9% of its score · 2 tests

Beats LongCat 2.0 (1.6T-A48B), Mistral Large 4 (public preview), Inkling Small (266B-A12B) here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
IFEval lab97.2% ✎▄··▄··█···▄▄ beats 3/5very easy · avg 94%▇ 0.7%maxed out
IFEval (aggregation unspecified)94.8% ✎▁▁▃▁▂▄▄▆█▃▂▃ beats 138/141easy · avg 75%▇ 0.2%noisy
math▁▂▃▅▆▅▄█▄▁▁▁ #161 of 398 · 20% of its score · 4 tests

Beats MiMo-V2.5 (310B-A15B), MiniMax-M2.7, MiniMax-M2.1 here, though they rank higher overall. Trails Olmo 3 7B Think, Qwen3-VL 4B Thinking, Granite 4.2 3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HMMT February 202665.6% ✎█·▄··▄··▄█·▄ beats 4/7hard · avg 32%▇▇▇▇ 9.4%sharp · at its level
IMO-AnswerBench57.2% ✎▄▄···▄·▄··█▄ beats 3/6hard · avg 32%▇▇ 4.9%at its level
MATH-Vision lab79.7% ✎██·█·····███ beats 3/5medium · avg 53%▇ 2.8%at its level
AIME 2026 lab77.5% ✎▄·▄▄····▄··█ beats 3/5medium · avg 41%▇ 2.6%sharp · noisy
vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #71 of 147 · 10% of its score · 4 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-5.4 nano, GPT-5 mini here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
MMMU Pro lab69.1% ✎██·█····█·██ beats 3/5medium · avg 49%▇ 3.6%at its level
MATH-Vision lab79.7% ✎██·█·····███ beats 3/5medium · avg 53%▇ 2.8%at its level
InfographicVQA lab88.4% ✎▄█·······█·▄ beats 3/5easy · avg 71%▇ 2.5%typical
OmniDocBench 1.5 ↓ lab16.4% ✎█·█·····████ beats 3/5avg 31%▇ 1.4%blunt
web search█·█▄██·▄▄·▄▄ #5 of 13 * · 1.4% of its score · 1 test

Beats Qwen3.5 9B here, though they rank higher overall. Trails Spark-X2.5 1.7B, Qwen3.5 4B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BrowseComp10.0% ✎▅···█·▃···▃▃ beats 4/7very hard · avg 10%▇ 1.4%at its level · noisy
other tests10 not in a task · 42% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond lab78.8% ✎█···▄·····▄█ beats 3/5hard · avg 47%▇▇▇ 7.1%sharp · at its level
Humanity's Last Exam13.1% ✎▄▄·▄··▄·▄·▄█ beats 6/7very hard · avg 3.8%▇▇▇ 7.1%typical
MedXPertQA MM lab48.7% ✎▂···▂·▃··▆▂█ beats 5/13very hard · avg 30%▇▇ 5.9%sharp · at its level
GPQA (subset unspecified)72.8% ✎█···▃···▅·▃▃ beats 6/7hard · avg 45%▇▇ 5.1%sharp · at its level
MMLU-Pro lab77.2% ✎▄··█···▄··▄▄ beats 3/5medium · avg 61%▇ 3.6%typical
MRCR v2 8-needle, 128k lab43.4% ✎▄▄·▄···█···▄ beats 3/5hard · avg 22%▇ 3.4%at its level
MMMLU lab83.4% ✎██··█···█·██ beats 3/5medium · avg 69%▇ 3.4%typical
IFBench lab74.0% ✎▄▄·▄······▄█ beats 4/5medium · avg 40%▇ 2.9%at its level
2 more tests
testresultvs the others in its rundifficultycountswhy
Big Bench Extra Hard micro avg lab53.0% ✎█··▄···▄·▄·▄ beats 3/5hard · avg 28%▇ 2.7%at its level
HLE lab5.2% ✎█···█······█ beats 0/2very hard · avg 3.9%▇ 1.1%small run

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Gemma 4 12B · needs ~7.4 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/gemma-4-12B-it-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/gemma-4-12B-it-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Gemma 4 12B · needs ~7.4 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/gemma-4-12B-it-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Gemma 4 12B · needs ~7.64 GB · from ollama.com
    chat in the terminal
    ollama run gemma4:12b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Gemma 4 12B · needs ~6.7 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/gemma-4-12B-it-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/gemma-4-12B-it-MLX-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Gemma Team — Gemma 4 Technica… 64%, XHToken/Spark-X2.5-4B model c… 33%, google/gemma-4-31B-it model c… 3%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Gemma Team — Gemma 4 Technical Report · lab-reported20▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 64%CC BY 4.0
XHToken/Spark-X2.5-4B model card9▇▇▇▇▇▇▇▇▇▇ 33%Apache-2.0
google/gemma-4-31B-it model card · lab-reported1▇ 3%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 62.4
$ published speed measurements · 1

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GBGemma QAT GGUF UD-Q4_K_XL29.34 tok/sllama.cpp ac4cddeb0 (build_number 9592 as recorded), Vulkan (RADV) · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence; file label r10 repetitions

Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics (RADV STRIX_HALO). Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 31 (2026-06-11, measured-local): Current Google Gemma 4 QAT direct route on upstream llama.cpp; useful current-model evidence, but not a speed candidate versus Qwen/LFM MoE rows.

source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json