whatmodel.sh / Llama 3.3 70B Instruct

Llama 3.3 70B Instruct

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Llama 3.3 70B Instruct is an open-weights language model from Meta, released December 2024. Its published size is 70B parameters and a 128k-token context window. Licence: llama-3.3.

54.3±0.6#279 of 719 LLMsfrom 60 benchmark results across 27 sources

$ how it compares

The LLMs ranked either side of Llama 3.3 70B Instruct, out of 828 with a score, then more from Meta.

#modelscorewhereRAMreleased
331OpenThinker2 32B Open Thoughts54±1local~21GB2025-04
332Falcon-H1 34B Instruct TII54±3local~23GB2025-05
333Llama 3.3 70B Instruct Meta54±1api $0.72/M +local~24GB2024-12
334Llama 4 Scout (109B-A17B) Meta54±1api $0.15/M +local~36GB2025-04
335Nemotron 3 Nano 4B NVIDIA54±1local~3.8GB2026-03
more from Meta
24Muse Spark 1.3 Meta77±1api $2.00/M2026-09
40Muse Spark 1.2 Meta75±1api $2.00/M2026-08
43Muse Spark 1.1 Meta75±1api $2.00/M2026-07

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #246 of 300 · 11% of its score · 10 tests

Beats DeepSeek R1 0528 Qwen3 8B, Magistral Small 1.0 (2506) here, though they rank higher overall. Trails Granite 4.0 H 1B, Granite 4.0 1B, SmolLM3 3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
SWE-bench Verified29.4% ✎▃···▃·▃·▃█·▃ beats 4/7hard · avg 19%▇ 3.1%sharp · at its level
ContractBench success rate (33 observation-contract tool tasks)7.1% ✎▆··▆▄▂▄▂▆▆█▄ beats 1/23very hard · avg 14%▇ 1.8%sharp
BFCL V4 (December 2025 archive)31.9% ✎▁▁▁▂██▅▅▆▂▂▂ beats 34/77hard · avg 32%▇ 1.3%at its level
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)2.8% ✎█·▃▃·█▅▅▅▅▃▅ beats 1/18very hard · avg 5.8%▇ 1.0%sharp
τ-bench Airline35.3% ✎▁▁▁·▅▄▁█▇▁▁▇ beats 7/28hard · avg 23%▇ 0.8%at its level
EEGAgentBench overall (tool-based EEG analysis agents, 6 tasks)42.9% ✎▂·▁▁··▇▇█▂·▁ beats 5/25hard · avg 29%▇ 0.7%blunt · at its level
BFCL v352.2% ✎▁▁▁▁▁▃▅█▅▄▁▁ beats 44/114medium · avg 52%▇ 0.6%blunt · at its level
τ-bench Retail6.2% ✎▃····▃·▂▇█▂▆ beats 0/28hard · avg 29%▇ 0.6%at its level · noisy
2 more tests
testresultvs the others in its rundifficultycountswhy
Galileo Agent Leaderboard v2 (action completion)20.0% ✎▅▃·▃▃▅▅█▃█▅▅ beats 2/19hard · avg 27%▇ 0.5%at its level
Galileo Agent Leaderboard v1 (tool selection quality)77.4% ✎▃▃▃▃▆▅▆▅█▂▅▂ beats 7/31easy · avg 79%▇ 0.5%blunt
coding▁▁▁▃▆██▇▆▄▃▁ #234 of 458 · 30% of its score · 14 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails QwQ-32B-Preview, InternLM2.5 20B Chat, Gemma 2 27B IT here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LiveCodeBench official (Jan 1–Nov 1, 2024)31.8% ✎▁▁·▁▂▄▆█▅▂·▂ beats 15/31hard · avg 26%▇▇▇ 7.2%sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)59.4% ✎▁·▁▂▄▁▆▄█▆▂▂ beats 29/62medium · avg 54%▇▇ 4.9%at its level
SWE-bench Verified29.4% ✎▃···▃·▃·▃█·▃ beats 4/7hard · avg 19%▇ 3.1%sharp · at its level
Aider polyglot (225 exercises, two attempts)8.4% ✎█▄··█·█····▄ beats 4/7very hard · avg 6.7%▇ 2.9%sharp · noisy
HumanEval+77.9% ✎▁·▁▁▂▃▄▄█▅▄▃ beats 103/145easy · avg 74%▇ 2.6%sharp
LiveCodeBench v5 (Oct 2024–Feb 2025)32.9% ✎▄▄·▄·····█▄█ beats 6/7hard · avg 19%▇ 2.0%at its level
MBPP+81.0% ✎██·██···████ beats 3/7easy · avg 79%▇ 1.3%typical
WebApp1K-React pass@1 (onekq-ai)33.4% ✎▃▁▁█▁▄▅▃▄▇▁▄ beats 9/31medium · avg 44%▇ 1.3%sharp · at its level · noisy
6 more tests
testresultvs the others in its rundifficultycountswhy
BIRD-SQL (dev)58.0% ✎▄··▄·▄·█··▄█ beats 5/7medium · avg 49%▇ 1.0%at its level
CodeReviewQA: change localisation (hard), invariant accuracy66.6% ✎█▂▁▁▂▂▂▃▅▅▅▇ beats 45/47medium · avg 36%▇ 1.0%sharp · at its level · noisy
HumanEval (simple-evals)78.9% ✎▂▅▆█▅▃··▂··▂ beats 1/19easy · avg 78%▇ 0.8%typical
CodeReviewQA: solution identification (hard), invariant accuracy78.0% ✎▃··▁▂▂▃▂█▃▄▂ beats 43/47medium · avg 54%▇ 0.7%sharp · at its level · noisy
CodeReviewQA: automated code refinement, exact match rate32.9% ✎▁▁▁▁▁▁▂▂▃█▅▄ beats 29/47hard · avg 33%▇ 0.7%at its level
CodeReviewQA: change type recognition, invariant accuracy66.0% ✎▁▁▁······▁▃█ beats 14/47easy · avg 67%▇ 0.0%blunt · at its level · noisy
chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #207 of 393 · 9.2% of its score · 6 tests

Beats Granite 4.2 8B, Qwen3-VL 30B-A3B Thinking, Mistral Small 4 (119B-A6.5B) here, though they rank higher overall. Trails LFM2.5 1.2B Instruct, Granite 4.0 H Micro 3B, Gemma 3n E4B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
LMArena (text)1317.8 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 150/379avg 1268▇ 2.8%blunt
Arabic IFEval prompt-level (Inception)58.2% ✎▁·▁▁▂▁▄▄▆▄█▄ beats 32/38medium · avg 49%▇ 2.3%sharp · at its level
AlpacaEval 2 length-controlled36.5% ✎▃▃·▁▁█▇▅·▁·▁ beats 12/22medium · avg 42%▇ 2.0%at its level
Arena-Hard (version/judge unspecified)65.5% ✎▃▃▂▅▃▆▅█▅▅▂▃ beats 16/30easy · avg 66%▇ 1.3%at its level
IFEval prompt-level loose90.8% ✎▁▃▁▃▁▄▁▁█▃▃▃ beats 22/22easy · avg 80%▇ 0.5%typical
IFEval (aggregation unspecified)89.3% ✎▁▁▃▁▂▄▄▆█▃▂▁ beats 109/141easy · avg 77%▇ 0.4%typical
math▁▂▃▅▆▅▄█▄▁▁▁ #268 of 398 · 14% of its score · 7 tests

Beats Molmo 2 8B here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Mixtral 8x7B v0.1 (base), InternVL2-4B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
MATH level 541.6% ✎▂▂▅▆▄█▆▃▅▂▃▄ beats 34/92hard · avg 32%▇▇▇ 7.0%sharp · at its level
AIME 202420.0% ✎▁▁▁▂▁▃▁▅███▃ beats 12/85hard · avg 16%▇ 2.6%sharp · small run
OTIS mock AIME 2024–20255.1% ✎▃▃▄▃▃▃▇▇█▃▂▅ beats 32/193very hard · avg 8.5%▇ 1.8%sharp · noisy
GSM8K93.6% ✎▂▃··▃▂▅█▆▃▃▃ beats 20/22very easy · avg 88%▇ 0.9%sharp · maxed out
MATH (subset unspecified)71.8% ✎▃▃█▂▂▆·····▁ beats 18/22medium · avg 61%▇ 0.9%at its level
MGSM (simple-evals)89.1% ✎▁·▁▁·▁▇▃█▁▃▁ beats 10/20easy · avg 82%▇ 0.7%blunt
MathConstraint full release, no tools (329 instances)8.5% ✎▄··▄▄·▄▄█▄·▄ beats 0/8very hard · avg 11%▇ 0.4%typical
web search▂▁▂▆▄█▃▃▁▁·▁ #55 of 80 * · 1.0% of its score · 1 test

Beats Kimi K2.6, GPT-5 Pro, DeepSeek V4 Flash (Preview, Apr 2026) here, though they rank higher overall. Trails GPT-4o mini (2024-07-18), Mistral Small 3.1 24B (2503), Llama 4 Scout (109B-A17B) here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Finance Agent Benchmark v1 (537 questions, class-balanced accuracy)2.8% ✎█·▃▃·█▅▅▅▅▃▅ beats 1/18very hard · avg 5.8%▇ 1.0%sharp
other tests24 not in a task · 39% of its score
testresultvs the others in its rundifficultycountswhy
GPQA Diamond47.4% ✎▁▁▁▁▂▄▆▄▅▇█▅ beats 59/208hard · avg 40%▇▇ 4.4%sharp · at its level
Medmarks-V mean win rate (19 verifiable medical benchmarks, 71 configurations)0.5364 ✎▁▄▃▃▇▄█▃▅▁▁▁ beats 22/30avg 0.423▇▇ 4.3%sharp
BIG-Bench Hard85.8% ✎▁▄·▁▁·▅█·▄▄▁ beats 21/22easy · avg 75%▇ 3.0%sharp
MMLU85.9% ✎▁▂·▁·▁▁█·▃▃▅ beats 21/22easy · avg 77%▇ 2.9%typical
GPQA Diamond (simple-evals)49.1% ✎▁·····▄█▂▂▃▃ beats 7/22hard · avg 41%▇ 2.7%sharp · at its level
Medmarks-V weighted mean win rate (61 models, 71 configurations)53.6% ✎▂▃▅▂▆▃█▅█▅▃▆ beats 22/34medium · avg 42%▇ 2.1%blunt · at its level
Polish medical board exams, bias-mitigated MCQ protocol (7-exam average)65.8% ✎▄·▂▂▂··█▂▄·▄ beats 11/13hard · avg 48%▇ 1.9%at its level
MMLU-Pro64.4% ✎▁▁·▁▁▁▃▂▄▅█▁ beats 72/188medium · avg 59%▇ 1.9%at its level
16 more tests
testresultvs the others in its rundifficultycountswhy
RepoQA (32K, threshold 0.8)85.6% ✎▄▄▄··▄▄··▄·█ beats 4/7easy · avg 81%▇ 1.8%typical
CPTUB average (SpeakLeash, 0-5 judge scale)72.9% ✎▁▂▂▂▃█▃▆▅▄▂▃ beats 62/95easy · avg 66%▇ 1.7%at its level
BRIDGE clinical text, zero-shot average (87 tasks)39.9% ✎▁▁▁▁▁▂▂▃▄▆█▃ beats 72/97hard · avg 32%▇ 1.5%blunt · at its level
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)52.1% ✎▁▁·▁▂▂▄█▄▅▄▄ beats 48/66hard · avg 31%▇ 1.5%at its level
BigCodeBench (Instruct)41.9% ✎▄··▄▄···▄▄█▄ beats 3/7medium · avg 36%▇ 1.4%at its level
Polish EQ-Bench v2 (SpeakLeash)72.9% ✎▁··▁▁▂▂▆▅▆█▆ beats 63/70medium · avg 64%▇ 1.3%at its level
MIMIC-CDM full information (4 pathologies)83.9% ✎▄·······▄█·▄ beats 4/4easy · avg 73%▇ 1.3%typical
Open Persian LLM Leaderboard v2 average (21 tasks)68.5% ✎▂▄▂▃█▄▃▆▃▅▂▂ beats 68/81medium · avg 58%▇ 1.1%blunt · at its level
MMLU (simple-evals, zero-shot CoT)86.3% ✎▁···▂▁▅█▂▂▃▁ beats 11/22easy · avg 80%▇ 0.9%typical
PopQA48.2% ✎▁▁▁▃▃▃▁█▁▃·▄ beats 21/21hard · avg 32%▇ 0.8%blunt · at its level
TruthfulQA MC266.1% ✎▃▃▂·▃▃·▅█▂▅▂ beats 16/21medium · avg 63%▇ 0.6%blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)79.2% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 252/276medium · avg 73%▇ 0.5%at its level · noisy
DiagnosticIQ industrial maintenance MCQ (macro accuracy, 4 options)61.7% ✎▁··▃▁▁▁▅█▅▁▅ beats 12/24medium · avg 52%▇ 0.5%blunt · at its level
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy94.4% ✎▁▁▁▁▁▁▁▂▄█▄▃ beats 137/146easy · avg 83%▇ 0.3%noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)96.0% ✎▁▁▂▃▄▄█▆▃▂▂▁ beats 87/140very easy · avg 94%▇ 0.1%blunt · maxed out · noisy
DROP F1, 3-shot (simple-evals)90.2% ✎▅▂▆█▆·▃·▂▅·▂ beats 21/23easy · avg 80%▇ 0.1%blunt

$ where to run it

Through an API

api · Amazon Bedrock · $0.72/M · $0.72 in / $0.72 out per 1M tokens · 128k context · docs.aws.amazon.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run Llama 3.3 70B Instruct · needs ~21.5 GB · from huggingface.co/unsloth
    chat in the terminal
    llama cli -hf unsloth/Llama-3.3-70B-Instruct-GGUF:UD-Q2_K_XL
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf unsloth/Llama-3.3-70B-Instruct-GGUF:UD-Q2_K_XL

LM Studio

version

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run Llama 3.3 70B Instruct · needs ~39.7 GB, Mac only · from huggingface.co/mlx-community

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/mlx-community/Llama-3.3-70B-Instruct-4bit -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Llama 3.3 70B Instruct · needs ~42.5 GB · from ollama.com
    chat in the terminal
    ollama run llama3.3:70b

MLX (Mac)

version

  1. Install MLX (Mac) · Apple silicon Macs only, text only.
    with uv
    uv tool install mlx-lm
    or
    pip install mlx-lm
  2. Run Llama 3.3 70B Instruct · needs ~39.7 GB, Mac only · from huggingface.co/mlx-community
    chat in the terminal
    mlx_lm.chat --model mlx-community/Llama-3.3-70B-Instruct-4bit
  3. Or start an OpenAI-compatible server at http://localhost:8080/v1
    mlx_lm.server --model mlx-community/Llama-3.3-70B-Instruct-4bit

Weights and documentation: huggingface.co/meta-llama ↗

$ how the score was worked out · Cohere — Command A technical… 18%, Epoch AI — internal evaluatio… 13%, Microsoft Research — Phi-4 Te… 12% and 24 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Cohere — Command A technical report10▇▇▇▇▇ 18%CC BY 4.0
Epoch AI — internal evaluations3▇▇▇▇ 13%CC BY 4.0
Microsoft Research — Phi-4 Technical Report9▇▇▇ 12%CC BY 4.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models8▇▇▇ 12%CC BY 4.0
LiveCodeBench official website results (2024-12-26)1▇▇ 7%CC-BY-SA-4.0
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks — authors' own runs of 61 models2▇▇ 6%CC BY 4.0
Aider code editing benchmark runs (old edit leaderboard)1▇ 5%Apache-2.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)2▇ 4%Apache-2.0
LMArena leaderboard dataset1▇ 3%CC BY 4.0
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)4▇ 2%CC-BY-4.0
Berkeley Function Calling Leaderboard result archive2▇ 2%Apache-2.0
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? (arXiv:2606.12250v1)1▇ 2%CC BY-SA 4.0
ContractBench: Can LLM Agents Preserve Observation Contracts? (arXiv:2605.17281v2)1▇ 2%CC BY 4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)1▇ 2%MIT
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)1▇ 1%MIT
SpeakLeash Polish EQ-Bench leaderboard1▇ 1%Apache-2.0
MIMIC Clinical Decision Making leaderboard1▇ 1%Apache-2.0
WebApp1K models leaderboard (onekq-ai)1▇ 1%MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)1▇ 1%Apache-2.0
Finance Agent Benchmark (Vals AI, arXiv:2508.00828v1)1▇ 1%CC-BY-4.0
Galileo Agent Leaderboard (v1 and v2 results)2▇ 1%Apache-2.0
EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis (arXiv:2609.31632v1)1▇ 1%CC BY 4.0
Open Portuguese LLM Leaderboard (archived)1▇ 1%Apache-2.0
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules (arXiv:2605.08614v1)1▇ 1%CC BY 4.0
MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs — authors' own model runs1▇ 0%CC BY 4.0
EuroEval leaderboards (results collection), English generative leaderboard1▇ 0%MIT
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)1▇ 0%Apache-2.0
prior · pull toward the middle▇ 0%
weighted average= 54.3
$ published speed measurements · 4

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

NVIDIA GeForce RTX 5090 (32GB; host CPU/RAM not stated)GGUF IQ3_XXS (Unsloth repository)46.5 tok/sllama.cpp (GGUF; version not stated) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase

Model fully resident in GPU/unified memory as published.

source ↗
NVIDIA GeForce RTX 5090 (32GB; host CPU/RAM not stated)GGUF Q4_K_M (Unsloth repository) · layers offloaded2.3 tok/sllama.cpp (GGUF; version not stated) · Offload column: 25% (CPU); prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase

Partial CPU offload as published (share of model on CPU/system RAM); host CPU/RAM not stated.

source ↗
Apple M4 Pro (14-core CPU / 20-core GPU, 64GB)MLX 4-bit5.1 tok/sMLX v0.30.6 (macOS 26 Tahoe) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase

Model fully resident in GPU/unified memory as published.

source ↗
Apple M3 Ultra (28-core CPU / 60-core GPU, 96GB)MLX 4-bit13.1 tok/sMLX v0.30.6 (macOS 26 Tahoe) · Offload column: 0%; prompt-length variant not stated in Table 5 (study prompts ~16/~300/~496 tokens); fixed 256 output tokens, greedy decoding, fixed seed; two untimed warm-up generations; mean of three timed runs; generation throughput measured strictly during the decode phase

Model fully resident in GPU/unified memory as published.

source ↗
$ quantization evidence · 12 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precisiontestfull → quantizedkept
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.76796.5%source ↗
AutoRound 2-bit (group size 32; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.66786.0%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.795100.0%source ↗
AutoRound 4-bit (group size 128; 512 calibration samples, seqlen 4096, 512 iterations)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.77399.6%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.795100.0%source ↗
AWQ 4-bit (AutoAWQ, zero point, group size 128)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.77299.5%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.76298.2%source ↗
bitsandbytes NF4 (Transformers integration, nested/double quantization)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.76896.6%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.77697.6%source ↗
GGUF Q2_K with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.74395.8%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ X->English COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.795 → 0.79399.7%source ↗
GGUF Q4_K_M with imatrix (llama.cpp; imatrix from 20k WikiText samples)WMT24++ English->X COMET (wmt22-comet-da), unweighted mean over 6 language pairs (ja_JP, fr_FR, pl_PL, bn_IN, ml_IN, zu_ZA); greedy decoding0.776 → 0.77299.6%source ↗

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json