whatmodel.sh / Llama 3.1 8B Instruct

Llama 3.1 8B Instruct

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Llama 3.1 8B Instruct is an open-weights language model from Meta, released July 2024. Its published size is 8.03B parameters and a 128k-token context window. Licence: llama3.1.

44.6±0.4#447 of 719 LLMsfrom 87 benchmark results across 31 sources

$ how it compares

The LLMs ranked either side of Llama 3.1 8B Instruct, out of 828 with a score, then more from Meta.

#modelscorewhereRAMreleased
534Qwen2-Math 1.5B Instruct Alibaba45±2local~2.4GB2024-08
535Mistral Small (2402) Mistral45±1comparison only2024-02
536Llama 3.1 8B Instruct Meta45±0api $0.22/M +local~6.2GB2024-07
537Command R Plus (04-2024) Cohere45±1local~43GB2024-04
538Mistral NeMo Instruct (2407) Mistral44±1local~8.7GB2024-07
more from Meta
24Muse Spark 1.3 Meta77±1api $2.00/M2026-09
40Muse Spark 1.2 Meta75±1api $2.00/M2026-08
43Muse Spark 1.1 Meta75±1api $2.00/M2026-07

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #260 of 300 · 1.5% of its score · 4 tests

Beats DeepSeek R1 0528 Qwen3 8B, Magistral Small 1.0 (2506), Llama 4 Scout (109B-A17B) here, though they rank higher overall. Trails Granite 4.0 H 1B, Granite 4.0 1B, Qwen2.5 3B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MASSIVE-Agents AST (10k, zero-shot)9.1% ✎█····███·███ beats 3/6hard · avg 16%▇ 0.6%sharp
BFCL V4 (December 2025 archive)25.8% ✎▁▁▁▂██▅▅▆▂▂▂ beats 16/77hard · avg 32%▇ 0.4%at its level
Galileo Agent Leaderboard v1 (tool selection quality)67.8% ✎▃▃▃▃▆▅▆▅█▂▅▂ beats 2/31easy · avg 79%▇ 0.2%blunt · at its level
BFCL v349.6% ✎▁▁▁▁▁▃▅█▅▄▁▁ beats 33/114medium · avg 52%▇ 0.2%blunt · at its level
coding▁▁▁▃▆██▇▆▄▃▁ #316 of 458 · 29% of its score · 13 tests

Beats GLM-4.7-Flash, Qwen3-VL 8B Instruct, Qwen3-VL 4B Instruct here, though they rank higher overall. Trails DeepSeek LLM 67B Chat, WizardLM 70B V1.0, Phi-4-mini 3.8B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
HumanEval +169.5% ✎▁·▁▁▃▄▆▅█▃▁▁ beats 57/99easy · avg 79%▇▇▇▇ 11%sharp · at its level · several runs
HumanEval+62.8% ✎▁·▁▁▁▃▄▄▄█▃▁ beats 57/99easy · avg 74%▇▇▇ 7.5%sharp · at its level
CRUXEval (ZeroEval, 800 problems)39.9% ✎▁▂▆▃▆█▆▃▂·▁▁ beats 5/47medium · avg 61%▇ 3.2%sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)37.6% ✎▁·▁▂▄▁▆▄█▆▂▂ beats 5/62medium · avg 54%▇ 2.0%at its level
LiveCodeBench official (Jan 1–Nov 1, 2024)16.1% ✎▁▁·▁▃▄▆█▅▂·▂ beats 3/31hard · avg 26%▇ 1.7%sharp
MBPP+55.6% ✎▁···▁▁▂▅▇█▆▂ beats 29/75easy · avg 68%▇ 1.3%at its level
MBPP52.2% ✎▄···▄·▄▄█··█ beats 7/7medium · avg 64%▇ 0.7%at its level
WebApp1K-React pass@1 (onekq-ai)25.1% ✎▃▁▁█▁▄▅▃▄▇▁▄ beats 4/31medium · avg 44%▇ 0.4%sharp · noisy
5 more tests
testresultvs the others in its rundifficultycountswhy
Luau Leetcode fully solved (Torpedo Software)8.2% ✎▃▁▁▁·▁▂▃▅▃█▇ beats 0/49hard · avg 18%▇ 0.4%typical
CodeReviewQA: solution identification (hard), invariant accuracy54.3% ✎▃··▁▂▂▃▂█▃▄▂ beats 29/47medium · avg 54%▇ 0.3%sharp · at its level · noisy
CodeReviewQA: change localisation (hard), invariant accuracy29.8% ✎█▂▁▁▂▂▂▃▅▅▅▇ beats 25/47medium · avg 36%▇ 0.3%sharp · noisy
CodeReviewQA: automated code refinement, exact match rate26.2% ✎▁▁▁▁▁▁▂▂▃█▅▄ beats 17/47hard · avg 33%▇ 0.2%at its level
CodeReviewQA: change type recognition, invariant accuracy71.8% ✎▁▁▁······▁▃█ beats 21/47easy · avg 67%▇ 0.0%blunt · at its level · noisy
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #297 of 393 · 5.9% of its score · 6 tests

Beats Ministral 3 14B Reasoning 2512, Gemma 4 E4B, OpenThinker3 7B here, though they rank higher overall. Trails LFM2 350M, Flan-T5 XXL, LFM2 700M here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Arena-Hard (v0.1, GPT-4-1106 judge)21.3% ✎▂▁▁▅█▅▄▃▃▅▃▁ beats 16/60medium · avg 52%▇ 2.6%sharp · at its level
LMArena (text)1211.2 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 73/379avg 1268▇ 1.1%blunt
Arabic IFEval prompt-level (Inception)41.6% ✎▁·▁▁▂▁▄▄▆▄█▄ beats 23/38medium · avg 49%▇ 0.9%sharp · at its level
AlpacaEval 2 length-controlled25.8% ✎▃▃·▁▃█▇▅·▁·▁ beats 5/22medium · avg 42%▇ 0.7%at its level
IFEval (aggregation unspecified)50.4% ✎▄····█·▄··██ beats 4/7easy · avg 68%▇ 0.4%at its level
IFEval prompt-level loose80.6% ✎▁▃▁▃▁▄▁▃█▃▃▁ beats 11/22easy · avg 80%▇ 0.3%at its level
math▁▂▃▅▆▅▄█▄▁▁▁ #319 of 398 · 12% of its score · 8 tests

Beats Athene-70B, Gemma 2 9B IT, GPT-3.5 Turbo (0125) here, though they rank higher overall. Trails Gemma 7B (base), DeepSeek R1 Distill Qwen 1.5B, Tulu 2 DPO 13B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
GSM8K +283.4% ✎▂▃··▃▂▅█▆▃▂▃ beats 10/22very easy · avg 91%▇ 3.5%sharp · several runs
GSM8K (ZeroEval, zero-shot)84.0% ✎▁▄·▁▄█▅▅▅███ beats 16/43very easy · avg 89%▇ 2.1%sharp
MATH (subset unspecified) +142.5% ✎▃▃█▂▂▆·····▁ beats 11/22medium · avg 53%▇ 1.8%at its level · several runs
MATH level 522.9% ✎▂▂▅▆▄█▆▃▅▂▃▄ beats 20/92hard · avg 32%▇ 1.4%sharp
MGSM56.7% ✎▅▃▃█·▅···▃·▃ beats 4/10easy · avg 71%▇ 1.3%at its level
MATH Level 5 (ZeroEval, 721 problems)22.2% ✎▁▁▁▆█▂▇▇▄··▂ beats 11/38hard · avg 34%▇ 1.1%sharp
MATH47.6% ✎█▄██···▄▄▄·▄ beats 4/10medium · avg 64%▇ 0.8%at its level
OTIS mock AIME 2024–20251.7% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 12/193very hard · avg 8.5%▇ 0.1%sharp · too hard for it · noisy
other tests38 not in a task · 51% of its score
testresultvs the others in its rundifficultycountswhy
BIG-Bench Hard +271.9% ✎▁▄·▁▁·▅█·▄▄▁ beats 14/22medium · avg 67%▇▇ 6.0%sharp · at its level · several runs
MMLU-Redux (ZeroEval, 2,778 questions)67.2% ✎▁·▁▁▄▄▄▅█▄▂▁ beats 11/53easy · avg 77%▇▇ 5.7%at its level
MMLU +271.3% ✎▁▂·▁·▁▁█·▃▃▅ beats 12/22easy · avg 77%▇▇ 4.4%at its level · several runs
LiveBench 2024-07-2628.0% ✎▁▁▁▂▂▃▂█▆▃▅▃ beats 22/61medium · avg 41%▇▇ 4.2%at its level
MMLU-Pro +244.2% ✎▃▂▅█▃·█▃▅··▂ beats 10/23medium · avg 57%▇▇ 4.0%at its level · several runs
ARC Challenge +183.1% ✎▂▂▂█▂···▄··▂ beats 5/10very easy · avg 91%▇ 3.0%sharp · several runs
GPQA (subset unspecified) +126.3% ✎█··▄▂··▂▂··▄ beats 4/10hard · avg 39%▇ 2.3%sharp · several runs
OpenBookQA +184.8% ✎▂··▄█▂·▃···▄ beats 8/10easy · avg 87%▇ 2.2%at its level · several runs
30 more tests
testresultvs the others in its rundifficultycountswhy
WinoGrande +164.7% ✎▃····▃▃█▃·▅▅ beats 4/10easy · avg 84%▇ 2.0%at its level · several runs
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)0.3967 ✎▁▄▃▃▇▄█▃▅▁▁▁ beats 5/30avg 0.423▇ 1.7%sharp
CLIcK Korean (Daekeun Kim)47.8% ✎▆▄▄▄█▄▂·▄█·▄ beats 2/23medium · avg 58%▇ 1.6%at its level
HAE-RAE 1.0 (Daekeun Kim)53.9% ✎▄▄▄▂▂█▂▄▂█▂▄ beats 3/22medium · avg 63%▇ 1.5%at its level
HellaSwag +172.8% ✎▃▅█·▃▃▃···▃▃ beats 2/10easy · avg 87%▇ 1.2%several runs
BoolQ +182.8% ✎▃··▃··██▃·▃▃ beats 8/10easy · avg 91%▇ 1.1%several runs
ZebraLogic (ZeroEval, 1,000 puzzles)12.8% ✎▁▁▃▆██▁▁·▁▁▁ beats 17/57hard · avg 18%▇ 1.0%sharp
Multilingual MMLU (Microsoft)56.2% ✎█▄▄██··█···▄ beats 7/10easy · avg 65%▇ 0.9%at its level
RobloxQA v2.0 multiple choice (Torpedo Software)78.7% ✎▃▁▃▁█▄▃▃▆▂▂▂ beats 1/49easy · avg 78%▇ 0.8%at its level
EvalPerf correctness (118 tasks, 20240328)64.3% ✎▂·▂▂▃▆▅▆▆▆▃█ beats 3/29easy · avg 76%▇ 0.8%at its level
CPTUB average (SpeakLeash, 0-5 judge scale)60.2% ✎▁▂▂▂▃█▃▆▅▄▂▃ beats 33/95easy · avg 66%▇ 0.8%at its level
GPQA Diamond27.0% ✎▁▁▃▁▂▄▆▄▅▇█▅ beats 8/208hard · avg 40%▇ 0.8%sharp
Medmarks-V weighted mean win rate (61 models, 71 configurations)39.7% ✎▂▃▅▂▆▃█▅█▅▃▆ beats 5/34medium · avg 42%▇ 0.8%blunt · at its level
Social IQA +171.8% ✎▃·▃▅▅█·▃···▃ beats 1/10easy · avg 78%▇ 0.7%blunt · at its level · several runs
BRIDGE clinical text, zero-shot average (87 tasks)29.0% ✎▁▁▁▁▁▂▂▃▄▆█▃ beats 27/97hard · avg 32%▇ 0.5%blunt · at its level
Open Persian LLM Leaderboard v2 average (21 tasks)50.3% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 29/81medium · avg 58%▇ 0.5%blunt · at its level
Judgemark v2 (LLM-as-judge for creative writing)35.7% ✎▃▃··▃▅▅···█▃ beats 0/10medium · avg 55%▇ 0.4%at its level
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)24.9% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 20/66hard · avg 31%▇ 0.4%typical
TruthfulQA MC2 +155.1% ✎▃▃▂·▃▃·▅█▂▅▂ beats 6/21medium · avg 63%▇ 0.4%blunt · at its level · several runs
RULER 64k84.7% ✎▁▁·▁·▂▃▄█▂·▁ beats 14/22easy · avg 83%▇ 0.3%typical
PopQA28.4% ✎▁▁▁▃▃▃▁█▁▃·▄ beats 11/21hard · avg 32%▇ 0.3%blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)70.2% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 166/276medium · avg 73%▇ 0.2%at its level · noisy
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)38.7% ✎▃··▃▁▁▁▅█▅▁▅ beats 0/24medium · avg 52%▇ 0.2%blunt · at its level
KMMLU-HARD (Daekeun Kim, zero-shot)24.0% ✎█···▁▁▁▂▃·▆▂ beats 7/23very hard · avg 29%▇ 0.2%sharp · too hard for it
PIQA83.2% ✎▃·▅·▃█·▃▅··▃ beats 8/10easy · avg 83%▇ 0.2%blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy84.4% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 95/146easy · avg 83%▇ 0.2%at its level · noisy
RULER 128k77.0% ✎▄··▂▄▂▆██▄▄▂ beats 15/21easy · avg 68%▇ 0.1%at its level · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)94.6% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 57/140very easy · avg 94%▇ 0.1%blunt · maxed out · noisy
LIBRA Mini (Russian long context, May 2026 version)43.9% ✎▄···▄█·██··▄ beats 7/8medium · avg 38%▇ 0.1%blunt · at its level
Chess Puzzles (Epoch)0.0% ✎█▁▁▁▁▂▁▄▂▂▁▁ beats 23/135very hard · avg 5.9%▇ 0.0%sharp · too hard for it · noisy

$ where to run it

Through an API

api · Amazon Bedrock · $0.22/M · $0.22 in / $0.22 out per 1M tokens · 128k context · docs.aws.amazon.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Llama 3.1 8B Instruct · needs ~4.9 GB · from huggingface.co/lmstudio-community
    chat in the terminal
    llama cli -hf lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Llama 3.1 8B Instruct · needs ~4.9 GB · from huggingface.co/lmstudio-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF@Q4_K_M -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Llama 3.1 8B Instruct · needs ~5.32 GB · from ollama.com
    chat in the terminal
    ollama run llama3.1:8b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Llama 3.1 8B Instruct · needs ~4.5 GB, Mac only · from huggingface.co/mlx-community
    chat in the terminal
    mlx_lm.chat --model mlx-community/Llama-3.1-8B-Instruct-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model mlx-community/Llama-3.1-8B-Instruct-4bit

Weights and documentation: huggingface.co/meta-llama ↗

$ how the score was worked out · EvalPlus / EvalPerf published… 19%, IBM Granite 3.0 Language Mode… 16%, microsoft/Phi-4-mini-instruct… 16% and 28 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
EvalPlus / EvalPerf published evaluation runs4▇▇▇▇▇▇ 19%Apache-2.0
IBM Granite 3.0 Language Models technical report (GitHub)15▇▇▇▇▇ 16%Apache-2.0
microsoft/Phi-4-mini-instruct model card16▇▇▇▇▇ 16%MIT
WildEval ZeroEval published results5▇▇▇▇ 13%Apache-2.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models8▇▇ 6%CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets1▇ 4%Apache-2.0 (stated in the LiveBench datasheet)
Daekeun Kim Korean LLM benchmark runs3▇ 3%MIT
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)1▇ 3%Apache-2.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models2▇ 2%CC BY 4.0
Epoch AI — internal evaluations4▇ 2%CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)1▇ 2%Apache-2.0
MMLU-Pro author-released evaluation runs1▇ 2%Apache-2.0
LiveCodeBench official website results (2024-12-26)1▇ 2%CC-BY-SA-4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)2▇ 1%Apache-2.0
Roblox LLM Leaderboard (Torpedo Software): RobloxQA v2.0 and Luau Leetcode2▇ 1%MIT
LMArena leaderboard dataset1▇ 1%CC BY 4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 1%MIT
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)4▇ 1%CC-BY-4.0
Berkeley Function Calling Leaderboard result archive2▇ 1%Apache-2.0
MASSIVE-Agents — Kulkarni et al. (2025)1▇ 1%CC BY 4.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 1%MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 0%Apache-2.0
Judgemark v2 results (EQ-Bench)1▇ 0%MIT
WebApp1K models leaderboard (onekq-ai)1▇ 0%MIT
NVIDIA RULER repository README results table2▇ 0%Apache-2.0
Open Portuguese LLM Leaderboard (archived)1▇ 0%Apache-2.0
Galileo Agent Leaderboard (v1 and v2 results)1▇ 0%Apache-2.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)1▇ 0%CC BY 4.0
EuroEval leaderboards (results collection), English generative leaderboard1▇ 0%MIT
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
ai-forever LIBRA leaderboard (Russian long-context benchmark)1▇ 0%MIT
prior · pull toward the middle▇ 0%
weighted average= 44.6
$ quantization evidence · 18 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.46663.2%source ↗
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.35751.5%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.6898.1%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.70495.5%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.67697.5%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.70996.1%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.68198.1%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.794.9%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.69794.5%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.55680.2%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.694 → 0.68398.4%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.737 → 0.72398.0%source ↗
MXFP4 via AutoRound 'OPT-RNT(-1,1) (Our's default)' (iters 0; scheme as defined by auto-round; activation handling not restated in this table)Average of 11 lm-eval tasks (ARC-c, ARC-e, BoolQ, HellaSwag, LAMBADA, MMLU, OBQA, PIQA, TruthfulQA, WinoGrande, GSM8K strict)0.652 → 0.59190.7%source ↗
MXFP4 via AutoRound 'OPT-RTN(-2,-1,1,2)' (iters 0; scheme as defined by auto-round; activation handling not restated in this table)Average of 11 lm-eval tasks (ARC-c, ARC-e, BoolQ, HellaSwag, LAMBADA, MMLU, OBQA, PIQA, TruthfulQA, WinoGrande, GSM8K strict)0.652 → 0.58589.7%source ↗
MXFP4 via AutoRound 'orig-RTN' (iters 0; scheme as defined by auto-round; activation handling not restated in this table)Average of 11 lm-eval tasks (ARC-c, ARC-e, BoolQ, HellaSwag, LAMBADA, MMLU, OBQA, PIQA, TruthfulQA, WinoGrande, GSM8K strict)0.652 → 0.57287.8%source ↗
NVFP4 via AutoRound 'opt-rtn' (iters 0; scheme as defined by auto-round; activation handling not restated in this table)Average of 11 lm-eval tasks (ARC-c, ARC-e, BoolQ, HellaSwag, LAMBADA, MMLU, OBQA, PIQA, TruthfulQA, WinoGrande, GSM8K strict)0.652 → 0.63998.1%source ↗
NVFP4 via AutoRound 'RTN' (iters 0; scheme as defined by auto-round; activation handling not restated in this table)Average of 11 lm-eval tasks (ARC-c, ARC-e, BoolQ, HellaSwag, LAMBADA, MMLU, OBQA, PIQA, TruthfulQA, WinoGrande, GSM8K strict)0.652 → 0.62696.1%source ↗
ONNX INT4 AWQ (NVIDIA ModelOpt-Windows v0.19.0, quantizing the FP16 ONNX model)MMLU 5-shot (ModelOpt-Windows accuracy_benchmark; onnxruntime-genai-directml 0.4, transformers 4.44)0.684 → 0.66196.6%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json