Gemma 4 E4B

+ contribute

Gemma 4 E4B is an open-weights language model from Google, released April 2026. Its published size is 8B parameters (4.5B active per token) and a 131k-token context window. Licence: apache-2.0.

52.7±0.6#305 of 719 LLMsfrom 75 benchmark results across 13 sources

$ how it compares

The LLMs ranked either side of Gemma 4 E4B, out of 828 with a score, then more from Google.

#	model	score	where	RAM	released
364	Gemma 3 27B Google	53±1	api $0.10/M +local	~18GB	2025-03
365	Phi-4 14B Microsoft	53±1	api $0.087/M +local	~10GB	2024-12
366	Gemma 4 E4B Google	53±1	local	~6.9GB	2026-04
367	Ministral 3 8B Mistral	53±1	api $0.15/M +local	~6.5GB	2025-12
368	Qwen3-VL 4B Instruct Alibaba	53±1	local	~4.1GB	2025-10
	more from Google
9	Gemini 4 Argon (limited preview) Google	82±1	limited preview		2026-09
23	Gemini 3.8 Flash Google	77±1	api $1.50/M		2026-09
32	Gemini 3.7 Flash Google	76±1	api $1.50/M		2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #226 of 300 · 19% of its score · 15 tests

Beats o3-mini, Nova 2 Lite, Ministral 3 14B Reasoning 2512 here, though they rank higher overall. Trails SmolLM3 3B, Jamba 1.5 Mini, LFM2.5 1.2B Thinking here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BFCL V4 (December 2025 archive) +3	47.0% ✎	▄·█·▄▄▄▄·▄·▄ beats 3/8	medium · avg 43%	▇ 3.4%	at its level · several runs
τ²-bench Telecom lab +1	18.4% ✎	▄····█··▄·▄▄ beats 1/5	very hard · avg 14%	▇ 1.9%	sharp · noisy · several runs
Tau2 – airline lab	52.0% ✎	▃·▃··▃·····█ beats 2/5	medium · avg 42%	▇ 1.9%	at its level
SWE-bench Verified +1	15.0% ✎	▄··▄▄█··▄·▄█ beats 5/8	hard · avg 17%	▇ 1.8%	sharp · noisy · several runs
ContractBench success rate (33 observation-contract tool tasks)	17.2% ✎	▆··▆▄▂▄▂▆▆█▄ beats 3/23	very hard · avg 14%	▇ 1.6%	sharp
τ²-bench aggregate lab	42.2% ✎	▄·▄··▄····█▄ beats 2/5	hard · avg 32%	▇ 1.5%	at its level
Terminal-Bench Hard lab	8.0% ✎	██··█·█·█··█ beats 2/5	very hard · avg 4.7%	▇ 1.3%	sharp · too hard for it
SWE-bench Pro (split unspecified) +1	3.3% ✎	█▂···▃···▄·▂ beats 5/8	very hard · avg 7.5%	▇ 1.3%	sharp · noisy · several runs

7 more tests

test	result	vs the others in its run	difficulty	counts	why
τ³-bench Banking +1	4.1% ✎	▂▂·▅█▂▂▃▂▅▂▃ beats 4/20	very hard · avg 4.0%	▇ 1.0%	blunt · too hard for it · several runs
Terminal-Bench 2.1	3.4% ✎	▁·▃·▁▁▄▃▄█▃▇ beats 1/56	very hard · avg 6.5%	▇ 1.0%	typical
BrowseComp-ZH	7.0% ✎	▄▄·▄▄▄·█···█ beats 2/8	very hard · avg 13%	▇ 0.9%	typical
Tau2 – retail lab	67.1% ✎	▄····▄··▄·▄█ beats 2/5	medium · avg 40%	▇ 0.6%	sharp · at its level · noisy
τ³-bench	10.1% ✎	▃·▃·█·▃··▃·▃ beats 4/7	very hard · avg 14%	▇ 0.5%	blunt
BrowseComp	8.3% ✎	▅···█·▃···▃▃ beats 2/7	very hard · avg 10%	▇ 0.4%	noisy
T1-Bench multi-domain conversational agent pass^3	35.6% ✎	▂▂··▂·▂·▂▂█▄ beats 9/11	hard · avg 25%	▇ 0.0%	blunt · at its level · noisy

coding▁▁▁▃▆██▇▆▄▃▁ #258 of 458 · 16% of its score · 10 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, QwQ-32B-Preview, Phi-3 Medium 128K Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
LiveCodeBench v6 (window unspecified) +2	53.9% ✎	▃····█·▃▅▃·▃ beats 5/8	medium · avg 39%	▇▇ 5.3%	at its level · several runs
HumanEval+	89.1% ✎	▁·▁▁▂▃▄▄█▅▄▃ beats 129/145	easy · avg 80%	▇ 1.8%	sharp
SWE-bench Verified +1	15.0% ✎	▄··▄▄█··▄·▄█ beats 5/8	hard · avg 17%	▇ 1.8%	sharp · noisy · several runs
LiveCodeBench Pro 2025Q2 (Easy)	45.8% ✎	▄···█▄··▄█▄▄ beats 4/8	hard · avg 33%	▇ 1.7%	at its level
SWE-bench Pro (split unspecified) +1	3.3% ✎	█▂···▃···▄·▂ beats 5/8	very hard · avg 7.5%	▇ 1.3%	sharp · noisy · several runs
LiveCodeBench Pro 2025Q2 (Medium)	1.8% ✎	█···▃··▄▂··▂ beats 4/8	very hard · avg 2.0%	▇ 1.0%	sharp · too hard for it · noisy
SciCode lab	24.0% ✎	█▄·······▄▄▄ beats 2/5	hard · avg 21%	▇ 1.0%	typical
LiveCodeBench v6 (Feb–May 2025) lab	52.0% ✎	▄··▄·▄···▄·█ beats 2/5	medium · avg 43%	▇ 1.0%	at its level

2 more tests

test	result	vs the others in its run	difficulty	counts	why
MBPP+	70.9% ✎	▁··▁▁▃▄█▆▄▁▁ beats 86/120	easy · avg 66%	▇ 0.7%	at its level
SciCode (level unspecified)	27.5% ✎	█·····██████ beats 4/6	hard · avg 17%	▇ 0.7%	typical

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #323 of 393 · 1.4% of its score · 2 tests

Trails LFM2 350M, Gemma 3 1B, Flan-T5 XXL here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
IFEval (aggregation unspecified) +2	44.4% ✎	▄▄····▄··███ beats 1/8	easy · avg 81%	▇ 0.8%	several runs
IFEval lab	96.7% ✎	▄··▄··█···▄▄ beats 2/5	very easy · avg 94%	▇ 0.6%	maxed out

math▁▂▃▅▆▅▄█▄▁▁▁ #244 of 398 · 24% of its score · 6 tests

Beats GPT-4o (2024-11-20), GPT-4 Turbo (2024-04-09), Mistral Large 2 (2411) here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 70B, OpenThinker 7B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HMMT February 2026 +1	30.1% ✎	▅··▃▃·▃··▃·█ beats 2/8	hard · avg 32%	▇▇▇ 8.6%	sharp · at its level · several runs
AIME 2025 +1	37.1% ✎	█▄▄▄▄····█·▄ beats 2/8	medium · avg 42%	▇▇ 4.9%	sharp · at its level · several runs
AIME 2026 +1	45.0% ✎	▄·▄█·▄▄···█▄ beats 2/8	medium · avg 48%	▇▇ 4.4%	sharp · at its level · several runs
IMO-AnswerBench	26.9% ✎	▄▄···▄·▄··█▄ beats 1/6	hard · avg 32%	▇ 2.4%	at its level
MATH-500	88.2% ✎	██▄▄▄··▄···▄ beats 2/8	very easy · avg 89%	▇ 1.7%	typical
MATH-Vision lab	59.5% ✎	██·█·····███ beats 2/5	medium · avg 53%	▇ 1.6%	at its level

vision & documents▁▁▃▂▁▃▄▂▆▇█▄ #111 of 147 · 5.7% of its score · 4 tests

Beats DeepSeek V4 Flash Vision (Exp), GPT-4 Turbo (2024-04-09), Pixtral Large (2411) here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
MMMU Pro lab	52.6% ✎	██·█····█·██ beats 2/5	medium · avg 49%	▇ 1.8%	at its level
InfographicVQA lab	70.0% ✎	▄█·······█·▄ beats 1/5	easy · avg 71%	▇ 1.7%	at its level
MATH-Vision lab	59.5% ✎	██·█·····███ beats 2/5	medium · avg 53%	▇ 1.6%	at its level
OmniDocBench 1.5 ↓ lab	18.1% ✎	█·█·····████ beats 2/5	avg 31%	▇ 0.6%	blunt

web search█·█▄██·▄▄·▄▄ #11 of 13 * · 1.3% of its score · 2 tests

Beats Nemotron 3 Nano 4B here, though they rank higher overall. Trails Qwen3.5 2B, LFM2.5 8B-A1B, LFM2.5 2.6B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
BrowseComp-ZH	7.0% ✎	▄▄·▄▄▄·█···█ beats 2/8	very hard · avg 13%	▇ 0.9%	typical
BrowseComp	8.3% ✎	▅···█·▃···▃▃ beats 2/7	very hard · avg 10%	▇ 0.4%	noisy

other tests22 not in a task · 39% of its score

test	result	vs the others in its run	difficulty	counts	why
GPQA Diamond lab +1	58.6% ✎	█···▄·····▄█ beats 2/5	hard · avg 47%	▇▇▇ 6.9%	sharp · at its level · several runs
MMLU-Pro +1	68.3% ✎	▂··▂█·▃▂···▂ beats 6/8	medium · avg 61%	▇▇ 4.0%	at its level · several runs
MMLU-Redux +1	83.7% ✎	▂···█··▃▂··▂ beats 6/8	easy · avg 79%	▇ 3.2%	several runs
GPQA (subset unspecified)	54.5% ✎	█···▃···▅·▃▃ beats 3/7	hard · avg 45%	▇ 2.2%	sharp · at its level
EuroEval Life in the UK (English knowledge MC), validation split, 5-shot, accuracy	81.5% ✎	▂·▂▂·▃·▃▃·█▃ beats 3/14	easy · avg 80%	▇ 2.2%	typical
MMMLU lab	76.6% ✎	██··█···█·██ beats 2/5	medium · avg 69%	▇ 2.1%	at its level
MedXPertQA MM lab	28.7% ✎	▂···▂·▃··▆▂█ beats 3/13	very hard · avg 30%	▇ 1.8%	sharp
Humanity's Last Exam	3.9% ✎	▄▄·▄··▄·▄·▄█ beats 2/7	very hard · avg 3.8%	▇ 1.7%	too hard for it

14 more tests

test	result	vs the others in its run	difficulty	counts	why
IFBench (aggregation unspecified) +1	28.3% ✎	▅····▃▃·█·▃▃ beats 1/8	medium · avg 39%	▇ 1.5%	at its level · several runs
IFBench lab	44.0% ✎	▄▄·▄······▄█ beats 2/5	medium · avg 40%	▇ 1.5%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	39.2% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 65/97	hard · avg 32%	▇ 1.4%	blunt · at its level
MRCR v2 8-needle, 128k lab	25.4% ✎	▄▄·▄···█···▄ beats 2/5	hard · avg 22%	▇ 1.3%	at its level
EuroEval HellaSwag (English), validation split, 5-shot generative MC, accuracy	70.0% ✎	▂··▂▂·▂▆▆█·▄ beats 4/15	medium · avg 63%	▇ 1.2%	at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	59.0% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 48/81	medium · avg 58%	▇ 1.1%	blunt · at its level
Big Bench Extra Hard micro avg lab	33.1% ✎	█··▄···▄·▄·▄ beats 2/5	hard · avg 28%	▇ 1.1%	at its level
OJBench	19.0% ✎	▃····▃▅··█▃▃ beats 4/8	very hard · avg 12%	▇ 1.1%	typical
SuperGPQA	38.7% ✎	▃·▃·▃▅█····▃ beats 5/8	hard · avg 32%	▇ 1.0%	at its level
LongBench v2	42.7% ✎	▃······▅▅▃·█ beats 6/8	very hard · avg 32%	▇ 0.8%	typical
Multi-IF (aggregation unspecified) +1	45.9% ✎	▄▄···▄·▄·▄██ beats 1/8	medium · avg 60%	▇ 0.7%	blunt · at its level · several runs
Claw-Eval average score	58.0% ✎	▄·▄··█▄▄▄▄▄▄ beats 1/9	medium · avg 53%	▇ 0.6%	blunt · at its level
NTCIR SciClaimEval Subtask 1 pair-accuracy (dev, single model)	52.6% ✎	▃▃···▃·▃▃█·▃ beats 1/8	medium · avg 45%	▇ 0.5%	blunt · at its level
GSM-Plus	87.4% ✎	▁··▁▁▁·▃▁▄█▁ beats 11/16	easy · avg 83%	▇ 0.5%	blunt

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Gemma 4 E4B · needs ~5.3 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/gemma-4-E4B-it-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/gemma-4-E4B-it-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Gemma 4 E4B · needs ~5.3 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/gemma-4-E4B-it-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Gemma 4 E4B · needs ~5.3 GB · from ollama.com
chat in the terminal
ollama run gemma4:e4b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Gemma 4 E4B · needs ~5.1 GB, Mac only · from huggingface.co/AtomicChat
chat in the terminal
mlx_lm.chat --model AtomicChat/gemma-4-E4B-it-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model AtomicChat/gemma-4-E4B-it-MLX-4bit

Official weights and documentation: huggingface.co/google ↗

$ how the score was worked out · Gemma Team — Gemma 4 Technica… 29%, openbmb/MiniCPM5-2B model card 28%, JetBrains/Mellum2.1-12B-A2.5B… 14% and 10 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Gemma Team — Gemma 4 Technical Report · lab-reported	19	▇▇▇▇▇▇▇▇▇ 29%	CC BY 4.0
openbmb/MiniCPM5-2B model card	20	▇▇▇▇▇▇▇▇ 28%	Apache-2.0
JetBrains/Mellum2.1-12B-A2.5B-Thinking model card	11	▇▇▇▇ 14%	Apache-2.0
XHToken/Spark-X2.5-4B model card	9	▇▇▇▇ 13%	Apache-2.0
LiquidAI/LFM2.5-2.6B model card	7	▇▇ 6%	LFM Open License v1.0
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 3%	MIT
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)	1	▇ 2%	CC BY 4.0
google/gemma-4-31B-it model card · lab-reported	1	▇ 1%	Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task — authors' own single-model dev evaluation	1	▇ 0%	CC BY 4.0
LiquidAI/LFM2.5-8B-A1B model card	1	▇ 0%	LFM Open License v1.0
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains (arXiv:2606.11070v1)	1	▇ 0%	CC BY 4.0
prior · pull toward the middle		▇ 0%
weighted average		= 52.7

$ published speed measurements · 14

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	24.3 tok/s	vllm-metal (Metal; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	24.1 tok/s	vllm-mlx (MLX; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	22.6 tok/s	mlx_lm server (MLX; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	22.4 tok/s	llama.cpp server (Metal; --parallel 4 slots; build not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	14.1 tok/s	Hugging Face Transformers (PyTorch MPS; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	21.6 tok/s	omlx (MLX; SSD prefix cache, in-memory tier disabled; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 94/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	21.1 tok/s	vllm-metal (Metal; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 93/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	19.4 tok/s	vllm-mlx (MLX; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 94/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
Apple M5 Pro (64GB; CPU/GPU core counts not stated)	BF16 (paper's main-track/fidelity protocol; Appendix E caption does not restate dtype)	18.3 tok/s	llama.cpp server (Metal; --parallel 4 slots; build not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench harness (benchmark commit 616aa51c); 100 prompts per level after 3 warmup requests; closed loop at concurrency 1 (one request in flight); temperature 0, non-thinking generation; output throughput = total generated tokens / measured sweep wall time, so it includes prefill/TTFT and HTTP overhead (not a pure decode rate) Concurrency-1 cell; 100/100 requests succeeded. Appendix E speed tables do not restate dtype; the paper's main track and cross-model fidelity protocol are BF16.	source ↗
NVIDIA DGX Spark (GB10)	BF16	20.1 tok/s	llama.cpp (CUDA build, GGUF BF16 weights; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 Chat split, concurrency 1.	source ↗
NVIDIA DGX Spark (GB10)	BF16	18.1 tok/s	llama.cpp (CUDA build, GGUF BF16 weights; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 Agent split, concurrency 1.	source ↗
NVIDIA DGX Spark (GB10)	BF16	17.2 tok/s	vLLM (CUDA, Safetensors BF16; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 Chat split, concurrency 1.	source ↗
NVIDIA DGX Spark (GB10)	BF16	16.1 tok/s	vLLM (CUDA, Safetensors BF16; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 Agent split, concurrency 1. Dagger in source: 94/100 requests completed (six agent prompts rejected); throughput over completed requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	16 tok/s	SGLang (CUDA, Safetensors BF16; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 Agent split, concurrency 1. Dagger in source: 94/100 requests completed (six agent prompts rejected); throughput over completed requests.	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json