whatmodel.sh / SmolLM3 3B

SmolLM3 3B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

SmolLM3 3B is an open-weights language model from Hugging Face, released July 2025. Its published size is 3.08B parameters and a 66k-token context window. Licence: apache-2.0.

42.7±1.0#487 of 719 LLMsfrom 25 benchmark results across 6 sources

$ how it compares

The LLMs ranked either side of SmolLM3 3B, out of 828 with a score, then more from Hugging Face.

#modelscorewhereRAMreleased
575Qwen2.5 Coder 1.5B Instruct Alibaba43±1local~2.4GB2024-09
576LFM2 2.6B Liquid AI43±2local~3GB2025-09
577SmolLM3 3B Hugging Face43±1local~3.3GB2025-07
578Granite 3.1 8B Instruct IBM43±1local~6.6GB2024-12
579Llama 3.2 11B Vision Instruct Meta43±2local~8.8GB2024-09
more from Hugging Face
451StarChat2 15B v0.1 Hugging Face48±1local~12GB2024-03
532Zephyr ORPO 141B-A35B v0.1 Hugging Face45±2local~56GB2024-04
678Zephyr 7B alpha Hugging Face36±1local~6.1GB2023-10

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #98 of 300 * · 1.8% of its score · 1 test

Beats Grok 4.20 (0309 reasoning), Grok 4.3, o3-pro here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
BFCL (version not stated) lab88.8% ✎▄▆▄▂·█▆·▄··▂ beats 16/17very easy · avg 86%▇ 1.8%blunt · small run
coding▁▁▁▃▆██▇▆▄▃▁ #376 of 458 * · 2.6% of its score · 1 test

Beats LFM2.5 8B-A1B, DeepSeek R1 Distill Qwen 7B, Qwen3.5 2B here, though they rank higher overall. Trails Qwen2.5 1.5B Instruct, WizardLM 13B V1.2, Falcon3 3B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench v4 lab30.0% ✎█·█········█ beats 0/2medium · avg 47%▇ 2.6%at its level · small run
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #282 of 393 * · 7.6% of its score · 2 tests

Beats Ministral 3 14B Reasoning 2512, Gemma 4 E4B, Ministral 3 8B here, though they rank higher overall. Trails Flan-T5 XXL, LFM2 700M, EXAONE 3.5 2.4B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
IFEval mean of four measures72.4% ✎█··▄▄·▄····▄ beats 1/5easy · avg 85%▇▇ 5.3%typical
IFEval (aggregation unspecified) lab71.2% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 33/141easy · avg 80%▇ 2.3%at its level · small run
math▁▂▃▅▆▅▄█▄▁▁▁ #253 of 398 · 13% of its score · 3 tests

Beats GPT-4 Turbo (2024-04-09), Molmo 2 8B, Llama 3.3 70B Instruct here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), Tulu 2 DPO 70B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AIME 2025 lab36.7% ✎▁·▁▁▁▂▅▆█▇▅▁ beats 23/133medium · avg 53%▇▇▇ 7.8%sharp · at its level · small run
MATH-50073.6% ✎▃····▃·█···▃ beats 3/5easy · avg 81%▇▇ 4.7%at its level
GSM8K loose substring accuracy81.1% ✎█·█··███···█ beats 2/5easy · avg 84%▇ 0.5%blunt
other tests16 not in a task · 75% of its score
testresultvs the others in its rundifficultycountswhy
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)0.3548 ✎▁▄▃▃▇▄█▃▅▁▁▁ beats 2/30avg 0.423▇▇▇▇▇ 14%sharp
GPQA Diamond +126.3% ✎██····███··█ beats 0/5medium · avg 53%▇▇▇▇▇ 13%sharp · small run · several runs
MMLU59.8% ✎▄█···█·····▄ beats 1/5easy · avg 74%▇▇▇ 7.2%at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)35.5% ✎▂▃▅▂▆▃█▅█▅▃▆ beats 2/34medium · avg 42%▇▇ 6.1%blunt · at its level
MMLU-Pro23.9% ✎███···██···█ beats 1/5medium · avg 45%▇▇ 4.8%typical
IFBench (aggregation unspecified)17.9% ✎█··███··█··█ beats 0/5hard · avg 33%▇▇ 4.3%typical
Multi-IF three-turn average58.9% ✎▄··█▄··▄···▄ beats 2/5easy · avg 71%▇▇ 4.3%at its level
MMMLU seven languages (Liquid)50.0% ✎▄·█····█···▄ beats 1/5medium · avg 60%▇ 3.7%at its level
8 more tests
testresultvs the others in its rundifficultycountswhy
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)16.2% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 14/66hard · avg 31%▇ 2.9%typical
MGSM six languages, loose (Liquid)68.7% ✎▄·▄·█···▄··▄ beats 1/5easy · avg 82%▇ 2.9%at its level
Global-MMLU lab64.1% ✎▃▅▃·▃···▅·▅█ beats 3/11medium · avg 70%▇ 2.7%at its level · small run
GSM-Plus +158.9% ✎█·····███·██ beats 2/5easy · avg 83%▇ 2.1%blunt · small run · several runs
Open PT-LLM Leaderboard average (9 Portuguese tasks)67.5% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 135/276medium · avg 73%▇ 2.0%at its level · noisy
MixEval-Hard lab30.8% ✎█····█·····█ beats 0/2medium · avg 40%▇ 1.9%at its level · small run
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy59.0% ✎▃·▁▁▂▁▂▅█▇▄▁ beats 32/100medium · avg 68%▇ 1.5%at its level · noisy
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy75.7% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 51/146easy · avg 83%▇ 1.3%at its level · noisy

$ where to run it

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run SmolLM3 3B · needs ~2.35 GB · from huggingface.co/ggml-org
    chat in the terminal
    llama cli -hf ggml-org/SmolLM3-3B-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf ggml-org/SmolLM3-3B-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run SmolLM3 3B · needs ~2.35 GB · from huggingface.co/ggml-org

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/ggml-org/SmolLM3-3B-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run SmolLM3 3B · needs ~2.35 GB · from huggingface.co/ggml-org
    chat in the terminal
    ollama run hf.co/ggml-org/SmolLM3-3B-GGUF:Q4_K_M

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run SmolLM3 3B · needs ~1.7 GB, Mac only · from huggingface.co/lmstudio-community
    chat in the terminal
    mlx_lm.chat --model lmstudio-community/SmolLM3-3B-MLX-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model lmstudio-community/SmolLM3-3B-MLX-4bit

Weights and documentation: huggingface.co/HuggingFaceTB ↗

$ how the score was worked out · LFM2 Technical Report — Liqui… 43%, HuggingFaceTB/SmolLM3-3B mode… 29%, Medmarks: A Comprehensive Ope… 20% and 3 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
LFM2 Technical Report — Liquid AI own SmolLM3 evaluation11▇▇▇▇▇▇▇▇▇▇▇▇▇ 43%CC BY 4.0
HuggingFaceTB/SmolLM3-3B model card (Evaluation section) · lab-reported8▇▇▇▇▇▇▇▇▇ 29%Apache-2.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models2▇▇▇▇▇▇ 20%CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)1▇ 3%Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard2▇ 3%MIT
Open Portuguese LLM Leaderboard (archived)1▇ 2%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 42.7
$ quantization evidence · 8 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token)AIME 2024 pass@1 (64 samples; LightEval, vLLM)0.453 → 0.475104.8%source ↗
FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token)Reasoning average of 3 tasks (AIME 2024 pass@1 over 64 samples, MATH-500 pass@1 over 4, GPQA Diamond pass@1 over 8; LightEval, vLLM)0.586 → 0.589100.5%source ↗
FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token)GPQA Diamond pass@1 (8 samples; LightEval, vLLM)0.412 → 0.40999.2%source ↗
FP8 W8A8 dynamic (weights symmetric static per-channel, activations symmetric dynamic per-token)MATH-500 pass@1 (4 samples; LightEval, vLLM)0.893 → 0.88398.9%source ↗
INT4 W4A16 (weights only, group size 128; GPTQ)GPQA Diamond pass@1 (8 samples; LightEval, vLLM)0.412 → 0.419101.6%source ↗
INT4 W4A16 (weights only, group size 128; GPTQ)MATH-500 pass@1 (4 samples; LightEval, vLLM)0.893 → 0.87598.0%source ↗
INT4 W4A16 (weights only, group size 128; GPTQ)Reasoning average of 3 tasks (AIME 2024 pass@1 over 64 samples, MATH-500 pass@1 over 4, GPQA Diamond pass@1 over 8; LightEval, vLLM)0.586 → 0.56295.9%source ↗
INT4 W4A16 (weights only, group size 128; GPTQ)AIME 2024 pass@1 (64 samples; LightEval, vLLM)0.453 → 0.39386.7%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json