whatmodel.sh / Gemma 3 4B

Gemma 3 4B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Gemma 3 4B is an open-weights language model from Google, released March 2025. Its published size is 4.3B parameters and a 131k-token context window. Licence: gemma.

44.1±0.7#455 of 719 LLMsfrom 57 benchmark results across 17 sources

$ how it compares

The LLMs ranked either side of Gemma 3 4B, out of 828 with a score, then more from Google.

#modelscorewhereRAMreleased
542Command R7B (12-2024) Cohere44±1local~6.2GB2024-12
543Qwen2-Audio 7B Instruct Alibaba44±3local~6.5GB2024-07
544Gemma 3 4B Google44±1api $0.063/M +local~3.8GB2025-03
545Eurus-2-7B-PRIME PRIME-RL44±2local~6.4GB2024-12
546Llama 3.1 Tulu 3 8B Ai244±1local~6.2GB2024-11
more from Google
9Gemini 4 Argon (limited preview) Google82±1limited preview2026-09
23Gemini 3.8 Flash Google77±1api $1.50/M2026-09
32Gemini 3.7 Flash Google76±1api $1.50/M2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #285 of 300 * · 2.3% of its score · 2 tests

Beats DBRX Instruct, Olmo 3 7B Instruct, QwQ-32B-Preview here, though they rank higher overall. Trails Granite 4.0 H 350M, ChatGLM3-6B, Qwen3 0.6B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
BFCL V4 (December 2025 archive)19.6% ✎▁▁▁▃██▅▅▆▂▂▂ beats 5/77hard · avg 32%▇ 1.4%typical
BFCL v339.6% ✎▁▁▁▁▃▃▅█▅▄▁▁ beats 14/114medium · avg 52%▇ 0.9%blunt · at its level
coding▁▁▁▃▆██▇▆▄▃▁ #361 of 458 · 11% of its score · 5 tests

Beats LFM2.5 8B-A1B, Qwen3-VL 2B Thinking, DeepSeek R1 Distill Qwen 7B here, though they rank higher overall. Trails Qwen2.5 1.5B Instruct, Phi-3-mini-128k-instruct (Apr 2024), OpenHermes 2.5 Mistral 7B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HumanEval lab71.3% ✎▁▁▁▁▁▃▅▅█▃▂▁ beats 126/190easy · avg 85%▇▇ 4.6%sharp · at its level
MBPP lab63.2% ✎▁▁▁▄▅█▄▅▃▁▂▁ beats 54/81easy · avg 72%▇ 2.4%at its level
LiveCodeBench (version unspecified) lab12.6% ✎▁···▃▁▃▅▅█▄▅ beats 1/37hard · avg 25%▇ 1.8%typical
VisEditBench strict pass rate (1,395 visualization-code editing tasks)21.3% ✎▄··█·▄··▄█·▄ beats 0/7hard · avg 34%▇ 1.7%at its level
BIRD-SQL (dev) lab36.3% ✎▂·······▃▂█▆ beats 1/11medium · avg 44%▇ 0.8%at its level
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #219 of 393 · 7.0% of its score · 3 tests

Beats Granite 4.2 8B, Mistral Small 4 (119B-A6.5B), Olmo 3.1 32B Think here, though they rank higher overall. Trails LFM2.5 1.2B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LMArena (text)1303.2 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 130/379avg 1268▇▇ 3.9%blunt
IFEval mean of four measures76.8% ✎█··▄▄·▄····▄ beats 2/5easy · avg 85%▇ 2.2%typical
IFEval (aggregation unspecified) lab90.2% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 117/141very easy · avg 87%▇ 0.9%typical
math▁▂▃▅▆▅▄█▄▁▁▁ #254 of 398 · 11% of its score · 6 tests

Beats GPT-4 Turbo (2024-04-09), Molmo 2 8B, Llama 3.3 70B Instruct here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), Tulu 2 DPO 70B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
GSM8K lab89.2% ✎▂▂▂▅▅▆▅▇█▇▄▃ beats 123/146very easy · avg 93%▇▇ 4.1%sharp
MATH (subset unspecified) lab75.6% ✎▂▄▄▃█▇▅▆▄▃▄▁ beats 48/80easy · avg 83%▇ 3.1%at its level
MATH-50073.2% ✎▃····▃·█···▃ beats 2/5easy · avg 81%▇ 1.9%at its level
MathVista (testmini) lab50.0% ✎▁··▁▄▄▇█▅▆█▄ beats 8/47medium · avg 61%▇ 1.6%at its level · small run
OTIS mock AIME 2024–20257.5% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 43/193very hard · avg 8.5%▇ 0.4%sharp · too hard for it · noisy
GSM8K loose substring accuracy89.9% ✎█·█··███···█ beats 5/5easy · avg 84%▇ 0.2%blunt
vision & documents▃▁▃▂▁▃▄▂▆▇█▄ #146 of 147 · 8.5% of its score · 10 tests

Trails Qwen2-VL 2B Instruct, Phi-3.5-vision-instruct, Qwen3.5 0.8B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
VisEditBench strict pass rate (1,395 visualization-code editing tasks)21.3% ✎▄··█·▄··▄█·▄ beats 0/7hard · avg 34%▇ 1.7%at its level
MathVista (testmini) lab50.0% ✎▁··▁▄▄▇█▅▆█▄ beats 8/47medium · avg 61%▇ 1.6%at its level · small run
MMMU (val) lab48.8% ✎▁····▁▁▄█▇▆▆ beats 7/52medium · avg 58%▇ 1.5%at its level · small run
InfographicVQA lab50.0% ✎▃·▃▅█▃▃▃▃▃▃▃ beats 0/13easy · avg 68%▇ 1.2%at its level · small run
VQAv2 (val) lab62.4% ✎▃▅▃█·▃▅·▃·▃▅ beats 0/13easy · avg 73%▇ 0.7%blunt · at its level · small run
ChartQA lab68.8% ✎▁··▁▁▁▁▃▂▅▅█ beats 5/44easy · avg 79%▇ 0.5%blunt · at its level · small run
VMMU (Vietnamese multimodal understanding and reasoning, mean accuracy over 7 tasks)25.4% ✎▅······▃·▅·█ beats 0/7very hard · avg 36%▇ 0.5%noisy
DocVQA lab75.8% ✎▃··▃▁▁▁▇█▇▄▅ beats 0/28very easy · avg 90%▇ 0.5%blunt · small run
2 more tests
testresultvs the others in its rundifficultycountswhy
TextVQA lab57.8% ✎▃▁▁▄▅▁▃·▃▃▃█ beats 0/24easy · avg 75%▇ 0.2%blunt · at its level · small run
AI2D test lab74.8% ✎▁▁▅▇▃▇·▄·█·▃ beats 7/28easy · avg 84%▇ 0.2%blunt · small run
other tests29 not in a task · 63% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond +223.2% ✎▃▁▁▁▂▄▆▄▅▇█▅ beats 4/208hard · avg 38%▇▇▇ 8.5%sharp · small run · several runs
MMLU +158.4% ✎▄█···█·····▄ beats 0/5medium · avg 71%▇▇▇ 6.6%at its level · several runs
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)0.3214 ✎▃▄▃▃▇▄█▃▅▁▁▁ beats 0/30avg 0.423▇▇ 5.9%sharp
BIG-Bench Hard lab72.2% ✎▁▁▁▁▂▄▅▅█▃▄▁ beats 99/129easy · avg 81%▇▇ 5.6%sharp · at its level
MMLU-Pro +134.8% ✎███···██···█ beats 3/5medium · avg 45%▇▇ 3.8%at its level · several runs
Global-MMLU-Lite lab54.5% ✎▂···▃·▂▂▂·██ beats 1/12medium · avg 65%▇ 2.7%at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)32.1% ✎▃▃▅▂▆▃█▅█▅▃▆ beats 0/34medium · avg 42%▇ 2.7%blunt · at its level
MIMIC-CDM full information (4 pathologies)69.4% ✎▄·······▄█·▄ beats 1/4easy · avg 73%▇ 2.5%at its level
21 more tests
testresultvs the others in its rundifficultycountswhy
EQ-Bench 3 (emotional intelligence, judge Elo)1118.2 ✎▁·▂▁▄▅▅▃█▄▁▁ beats 15/62avg 1044▇ 2.4%typical
MIMIC-CDM clinical decision making (4 pathologies)50.8% ✎█······█··██ beats 1/3medium · avg 62%▇ 2.1%at its level
IFBench (aggregation unspecified)23.5% ✎█··███··█··█ beats 3/5hard · avg 33%▇ 2.0%typical
Multi-IF three-turn average66.6% ✎▄··█▄··▄···▄ beats 4/5easy · avg 71%▇ 1.9%at its level
BRIDGE clinical text, zero-shot average (87 tasks)28.6% ✎▁▁▁▁▁▂▂▃▄▆█▃ beats 25/97hard · avg 32%▇ 1.8%blunt · at its level
Open Persian LLM Leaderboard v2 average (21 tasks)49.1% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 27/81medium · avg 58%▇ 1.7%blunt · at its level
SimpleQA lab4.0% ✎▄▄▄▆█▆███·▂█ beats 3/32very hard · avg 7.4%▇ 1.7%too hard for it
MMMLU seven languages (Liquid)50.1% ✎▄·█····█···▄ beats 2/5medium · avg 60%▇ 1.6%at its level
FACTS Grounding lab70.1% ✎▄········▄·█ beats 1/3easy · avg 73%▇ 1.5%at its level
RULER 128k lab46.8% ✎▃··▅▄▇▅█▇·▃▃ beats 4/32medium · avg 61%▇ 1.4%at its level · small run
Big Bench Extra Hard micro avg lab11.0% ✎▄▄·█▄·▄·▄·▄▄ beats 1/8hard · avg 19%▇ 1.4%small run
MGSM six languages, loose (Liquid)87.3% ✎▄·▄·█···▄··▄ beats 5/5easy · avg 82%▇ 1.2%at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)67.3% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 134/276medium · avg 73%▇ 0.9%at its level · noisy
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy57.9% ✎▃·▁▁▂▁▂▅█▇▄▁ beats 30/100medium · avg 68%▇ 0.7%at its level · noisy
WMT24++ (chrF) lab46.8% ✎█······█··██ beats 1/3medium · avg 53%▇ 0.6%blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy77.7% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 59/146easy · avg 83%▇ 0.5%at its level · noisy
GSM-Plus68.4% ✎█·····███·██ beats 5/5easy · avg 74%▇ 0.4%blunt · at its level
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)96.3% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 92/140very easy · avg 94%▇ 0.3%blunt · maxed out · noisy
WMT24++ lab77.2% ✎█···█·█··███ beats 0/5easy · avg 72%▇ 0.3%blunt · at its level
Chess Puzzles (Epoch)0.0% ✎█▁▁▁▁▂▁▄▂▂▁▁ beats 30/135very hard · avg 5.9%▇ 0.1%sharp · too hard for it · noisy
Vectara hallucination leaderboard (HHEM-2.3, factual consistency rate)93.6% ✎▁▂▁▂▅█▂▃▄▃▁▁ beats 74/98very easy · avg 90%▇ 0.0%blunt

$ where to run it

Through an API

api · DeepInfra · $0.063/M · $0.05 in / $0.1 out per 1M tokens · 131k context · deepinfra.com ↗ · price sourceUSD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-3-4b-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD.

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Gemma 3 4B · needs ~2.5 GB · from huggingface.co/ggml-org
    chat in the terminal
    llama cli -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Gemma 3 4B · needs ~2.5 GB · from huggingface.co/ggml-org

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Gemma 3 4B · needs ~3.08 GB · from ollama.com
    chat in the terminal
    ollama run gemma3:4b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Gemma 3 4B · needs ~2.2 GB, Mac only · from huggingface.co/Outlier-Ai
    chat in the terminal
    mlx_lm.chat --model Outlier-Ai/gemma-3-4b-it-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model Outlier-Ai/gemma-3-4b-it-MLX-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Google DeepMind Gemma 3 270M… 40%, LFM2 Technical Report — Liqui… 19%, Medmarks: A Comprehensive Ope… 9% and 14 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Google DeepMind Gemma 3 270M IT model card · lab-reported19▇▇▇▇▇▇▇▇▇▇▇▇ 40%CC-BY-4.0
LFM2 Technical Report — Liquid AI own SmolLM3 evaluation11▇▇▇▇▇▇ 19%CC BY 4.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models2▇▇▇ 9%CC BY 4.0
Gemma Team — Gemma 3 Technical Report · lab-reported6▇▇ 8%CC BY 4.0
MIMIC Clinical Decision Making leaderboard2▇ 5%Apache-2.0
LMArena leaderboard dataset1▇ 4%CC BY 4.0
Epoch AI — internal evaluations3▇ 3%CC BY 4.0
EQ-Bench 3 canonical leaderboard results1▇ 2%MIT
Berkeley Function Calling Leaderboard result archive2▇ 2%Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 2%MIT
VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? (arXiv:2608.10408v1)1▇ 2%CC BY 4.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 2%Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 1%MIT
Open Portuguese LLM Leaderboard (archived)1▇ 1%Apache-2.0
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark (arXiv:2508.13680v5)1▇ 0%CC-BY-4.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)2▇ 0%Apache-2.0
TranslateGemma Technical Report (Google), arXiv 2601.09012v2 · lab-reported1▇ 0%CC BY 4.0
prior · pull toward the middle▇ 0%
weighted average= 44.1
$ published speed measurements · 10

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

NVIDIA DGX Spark (GB10)QAT Q4_081.05 tok/sllama.cpp 11fb327bf (7941), CUDA · llama-bench; tg32; 32 generated tokens; context depth 0; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
NVIDIA DGX Spark (GB10)QAT Q4_076.37 tok/sllama.cpp 11fb327bf (7941), CUDA · llama-bench; tg32 @ d4096; 32 generated tokens; context depth 4096; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
NVIDIA DGX Spark (GB10)QAT Q4_062.28 tok/sllama.cpp 11fb327bf (7941), CUDA · llama-bench; tg32 @ d32768; 32 generated tokens; context depth 32768; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
Apple M2 Ultra (GPU core count and RAM not specified)QAT Q4_0134.28 tok/sllama.cpp b828e18c7 (7948), Metal/BLAS · llama-bench; tg32; 32 generated tokens; context depth 0; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
Apple M2 Ultra (GPU core count and RAM not specified)QAT Q4_0133.11 tok/sllama.cpp b828e18c7 (7948), Metal/BLAS · llama-bench; tg32 @ d4096; 32 generated tokens; context depth 4096; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
Apple M2 Ultra (GPU core count and RAM not specified)QAT Q4_0113.14 tok/sllama.cpp b828e18c7 (7948), Metal/BLAS · llama-bench; tg32 @ d32768; 32 generated tokens; context depth 32768; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
NVIDIA DGX Spark (GB10)QAT Q4_071.61 tok/sllama.cpp 11fb327bf (7941), CUDA · llama-bench; tg32 @ d8192; 32 generated tokens; context depth 8192; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
NVIDIA DGX Spark (GB10)QAT Q4_070.89 tok/sllama.cpp 11fb327bf (7941), CUDA · llama-bench; tg32 @ d16384; 32 generated tokens; context depth 16384; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
Apple M2 Ultra (GPU core count and RAM not specified)QAT Q4_0125.85 tok/sllama.cpp b828e18c7 (7948), Metal/BLAS · llama-bench; tg32 @ d8192; 32 generated tokens; context depth 8192; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗
Apple M2 Ultra (GPU core count and RAM not specified)QAT Q4_0125 tok/sllama.cpp b828e18c7 (7948), Metal/BLAS · llama-bench; tg32 @ d16384; 32 generated tokens; context depth 16384; n_ubatch=2048; flash attention=1; one sequence

Published token-generation mean; not prompt processing or aggregate batch throughput. Hardware core count and installed RAM are not inferred.

source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json