whatmodel.sh / Llama 3.1 70B Instruct

Llama 3.1 70B Instruct

+ contribute

Llama 3.1 70B Instruct is an open-weights language model from Meta, released July 2024. Its published size is 70B parameters and a 128k-token context window. Licence: llama-3.1.

52.4±0.5#310 of 719 LLMsfrom 58 benchmark results across 28 sources

$ how it compares

The LLMs ranked either side of Llama 3.1 70B Instruct, out of 828 with a score, then more from Meta.

#	model	score	where	RAM	released
370	LFM2.5 2.6B Liquid AI	53±1	local	~3.2GB	2026-07
371	Bielik 11B v2.1 Instruct SpeakLeash	53±3	local	~8.5GB	2024-08
372	Llama 3.1 70B Instruct Meta	52±1	api $0.72/M +local	~27GB	2024-07
373	Yi-Large Preview 01.AI	52±1	comparison only		2024-05
374	Qwen2.5-Math 7B Instruct Alibaba	52±2	local	~6GB	2024-09
	more from Meta
24	Muse Spark 1.3 Meta	77±1	api $2.00/M		2026-09
40	Muse Spark 1.2 Meta	75±1	api $2.00/M		2026-08
43	Muse Spark 1.1 Meta	75±1	api $2.00/M		2026-07

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #241 of 300 · 6.1% of its score · 7 tests

Beats DeepSeek R1 0528 Qwen3 8B, Sky-T1 32B Preview, Magistral Small 1.0 (2506) here, though they rank higher overall. Trails Granite 4.0 1B, SmolLM3 3B, Phi-3.5-mini-instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MASSIVE-Agents AST (10k, zero-shot)	18.8% ✎	█····███·███ beats 4/6	hard · avg 16%	▇ 1.8%	sharp
BrowserGym WebArena (AgentLab GenericAgent)	18.4% ✎	█··▄·▄█·▄··▄ beats 1/7	hard · avg 18%	▇ 1.1%	typical
BrowserGym WorkArena-L1 (AgentLab GenericAgent)	27.9% ✎	█···█▄▄·▄█·█ beats 1/10	hard · avg 29%	▇ 1.1%	at its level
BrowserGym MiniWoB (AgentLab GenericAgent)	57.6% ✎	▅····▅▅▃▃·▃█ beats 1/11	medium · avg 55%	▇ 0.7%	blunt · at its level
BFCL V4 (2025-08-26 archive, models absent from later archives)	17.6% ✎	▄▄····█▄···▄ beats 0/5	hard · avg 27%	▇ 0.6%	at its level
BFCL v3	53.6% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 52/114	medium · avg 52%	▇ 0.4%	blunt · at its level
BrowserGym WorkArena-L2 (AgentLab GenericAgent)	2.1% ✎	▃▅▃·▅▃▃··█·▃ beats 1/11	very hard · avg 3.9%	▇ 0.4%	sharp · too hard for it · noisy

coding▁▁▁▃▆██▇▆▄▃▁ #244 of 458 · 19% of its score · 10 tests

Beats Qwen3-VL 32B Instruct, Solar Open 100B (102B-A12B), GLM-4.7-Flash here, though they rank higher overall. Trails QwQ-32B-Preview, InternLM2.5 20B Chat, Mistral Small Instruct 22B (2409) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
CRUXEval (ZeroEval, 800 problems)	64.3% ✎	▁▂▆▃▆█▆▃▂·▁▁ beats 23/47	medium · avg 61%	▇▇ 5.0%	sharp · at its level
LiveCodeBench official (Jan 1–Nov 1, 2024)	26.3% ✎	▁▁·▁▂▄▆█▅▂·▂ beats 14/31	hard · avg 26%	▇▇ 4.7%	sharp · at its level
Aider code editing (133 Exercism Python exercises, two attempts)	58.6% ✎	▁·▁▂▄▁▆▄█▆▂▂ beats 28/62	medium · avg 54%	▇ 3.6%	at its level
HumanEval (Vector Institute, Inspect Evals)	78.7% ✎	▄·▄·█·█·█·▄▄ beats 1/9	easy · avg 79%	▇ 2.0%	typical
WebApp1K-React pass@1 (onekq-ai)	10.3% ✎	▃▁▁█▁▄▅▃▄▇▁▄ beats 1/31	medium · avg 44%	▇ 0.9%	sharp · at its level · noisy
SciCode main problems (official leaderboard)	0.0% ✎	▃···█··▂·▃·▂ beats 2/17	very hard · avg 1.1%	▇ 0.7%	sharp · too hard for it
CodeReviewQA: change localisation (hard), invariant accuracy	69.0% ✎	█▂▁▁▂▂▂▃▅▅▅▇ beats 47/47	medium · avg 36%	▇ 0.7%	sharp · at its level · noisy
CodeReviewQA: solution identification (hard), invariant accuracy	76.7% ✎	▃··▁▂▂▃▂█▃▄▂ beats 40/47	medium · avg 54%	▇ 0.5%	sharp · at its level · noisy

2 more tests

test	result	vs the others in its run	difficulty	counts	why
CodeReviewQA: automated code refinement, exact match rate	50.3% ✎	▁▁▁▁▁▁▂▂▃█▅▄ beats 47/47	hard · avg 33%	▇ 0.4%	at its level
CodeReviewQA: change type recognition, invariant accuracy	68.4% ✎	▁▁▁······▁▃█ beats 16/47	easy · avg 67%	▇ 0.0%	blunt · at its level · noisy

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #238 of 393 · 10% of its score · 5 tests

Beats Mistral Small 4 (119B-A6.5B), Ministral 3 14B Reasoning 2512, Qwen2.5 Coder 32B Instruct here, though they rank higher overall. Trails Flan-T5 XXL, Granite 4.0 H 1B, LFM2.5 1.2B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Arena-Hard (v0.1, GPT-4-1106 judge)	55.7% ✎	▂▁▁▅█▅▄▃▃▅▃▁ beats 39/60	medium · avg 52%	▇▇ 5.1%	sharp · at its level
LMArena (text)	1293.3 ✎	▁▁▁▂▃▃▃▆▆▇█▃ beats 125/379	avg 1268	▇ 1.9%	blunt
Arabic IFEval prompt-level (Inception)	44.8% ✎	▁·▁▁▂▁▄▄▆▄█▄ beats 24/38	medium · avg 49%	▇ 1.7%	sharp · at its level
AlpacaEval 2 length-controlled	32.9% ✎	▃▃·▁▁█▇▅·▁·▁ beats 9/22	medium · avg 42%	▇ 1.4%	at its level
IFEval prompt-level loose	88.0% ✎	▁▃▁▃▁▄▁▁█▃▃▁ beats 21/22	easy · avg 80%	▇ 0.3%	typical

math▁▂▃▅▆▅▄█▄▁▁▁ #276 of 398 · 17% of its score · 7 tests

Beats Molmo 2 8B, DeepSeek-Coder-V2 Instruct (0724) here, though they rank higher overall. Trails DeepSeek R1 Distill Qwen 1.5B, Granite 8B Code Base (4K), Mixtral 8x7B v0.1 (base) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
MATH level 5	36.7% ✎	▂▂▅▆▄█▆▃▅▂▃▄ beats 28/92	hard · avg 32%	▇▇ 4.8%	sharp · at its level
MATH (Vector Institute, Inspect Evals)	60.0% ✎	▄··██▄··▄·▄█ beats 1/9	medium · avg 53%	▇▇ 3.9%	sharp · at its level
MATH Level 5 (ZeroEval, 721 problems)	43.1% ✎	▁▁▁▆█▂▇▇▄··▂ beats 21/38	hard · avg 34%	▇ 2.9%	sharp · at its level
GSM8K +1	94.5% ✎	▂▃··▃▂▅█▆▃▂▃ beats 21/22	very easy · avg 89%	▇ 1.9%	sharp · small run · several runs
GSM8K (ZeroEval, zero-shot)	94.2% ✎	▁▄·▁▄█▅▅▅███ beats 32/43	very easy · avg 89%	▇ 1.8%	sharp
OTIS mock AIME 2024–2025	3.6% ✎	▃▃▄▃▃▃▇▇█▃▂▅ beats 24/193	very hard · avg 8.5%	▇ 0.9%	sharp · noisy
MATH (subset unspecified)	56.2% ✎	▃▃█▃▂▆·····▁ beats 14/22	medium · avg 61%	▇ 0.7%	at its level

other tests27 not in a task · 48% of its score

test	result	vs the others in its run	difficulty	counts	why
LiveBench 2024-07-26	48.4% ✎	▁▁▁▂▂▃▂█▆▃▅▃ beats 51/61	medium · avg 41%	▇▇▇ 8.5%	at its level
MMLU-Redux (ZeroEval, 2,778 questions)	83.0% ✎	▁·▁▁▄▄▄▅█▄▂▁ beats 40/53	easy · avg 77%	▇▇▇ 7.7%	at its level
ZebraLogic (ZeroEval, 1,000 puzzles)	24.9% ✎	▁▁▃▆██▁▁·▁▁▁ beats 36/57	hard · avg 18%	▇ 3.6%	sharp
MMLU-Pro	62.8% ✎	▃▂▅█▃·█▃▅··▂ beats 14/23	medium · avg 57%	▇ 3.3%	at its level
MMLU +1	85.2% ✎	▁▂·▁·▁▁█·▃▃▅ beats 19/22	easy · avg 77%	▇ 3.1%	at its level · several runs
GPQA Diamond	44.2% ✎	▁▁▁▁▂▄▆▄▅▇█▅ beats 48/208	hard · avg 40%	▇ 2.7%	sharp · at its level
GPQA Diamond (Vector Institute, Inspect Evals)	43.2% ✎	▃··▃▃▃▃·▃·▃█ beats 2/9	hard · avg 41%	▇ 2.5%	sharp · at its level
BIG-Bench Hard	83.0% ✎	▁▄·▁▁·▅█·▄▄▁ beats 19/22	easy · avg 75%	▇ 2.3%	sharp · at its level

19 more tests

test	result	vs the others in its run	difficulty	counts	why
MMLU-Pro (Vector Institute, Inspect Evals)	66.9% ✎	▃····▃▅·▃▅·█ beats 2/9	medium · avg 60%	▇ 1.8%	at its level
ARC Challenge lab	70.4% ✎	▆▁▁▂▂▅▇█▄▃▃▂ beats 81/97	medium · avg 70%	▇ 1.3%	sharp · at its level · small run
CPTUB average (SpeakLeash, 0-5 judge scale)	72.5% ✎	▁▂▂▂▃█▃▆▅▄▂▃ beats 59/95	easy · avg 66%	▇ 1.2%	at its level
EvalPerf correctness (118 tasks, 20240328)	77.3% ✎	▂·▂▂▂▆▅▆▆▆▃█ beats 12/29	easy · avg 76%	▇ 1.1%	typical
HindiGen-v1 3C3H score (Inception, Claude 3.5 Sonnet judge)	70.5% ✎	▁▁▁▁·▂·█▆█▇▁ beats 19/31	medium · avg 48%	▇ 1.0%	at its level
BRIDGE clinical text, zero-shot average (87 tasks)	39.1% ✎	▁▁▁▁▁▂▂▃▄▆█▃ beats 64/97	hard · avg 32%	▇ 1.0%	blunt · at its level
Polish EQ-Bench v2 (SpeakLeash)	72.5% ✎	▁··▁▁▂▂▆▅▆█▆ beats 61/70	medium · avg 64%	▇ 1.0%	at its level
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)	50.0% ✎	▁▁·▁▂▂▄█▄▅▄▄ beats 42/66	hard · avg 31%	▇ 1.0%	at its level
Open Persian LLM Leaderboard v2 average (21 tasks)	68.0% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 67/81	medium · avg 58%	▇ 0.8%	blunt · at its level
WinoGrande lab	85.1% ✎	▁···▂▁▁▅█▇▃▃ beats 86/87	easy · avg 85%	▇ 0.8%	at its level · small run
HellaSwag lab	86.4% ✎	▁▁▁▁▂▂▃▇█▆▅▃ beats 80/85	easy · avg 86%	▇ 0.6%	small run
PopQA	46.5% ✎	▁▁▁▃▃▃▁█▁▃·▄ beats 20/21	hard · avg 32%	▇ 0.5%	blunt · at its level
LongSocialBench (full-context, 4-option MCQ over discussion threads)	38.0% ✎	▆▃·▄█·▂▂·▄▄▂ beats 3/16	very hard · avg 36%	▇ 0.4%	blunt
Open PT-LLM Leaderboard average (9 Portuguese tasks)	79.3% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 255/276	medium · avg 73%	▇ 0.4%	at its level · noisy
TruthfulQA MC2	66.8% ✎	▃▃▂·▃▃·▅█▃▅▂ beats 17/21	medium · avg 63%	▇ 0.4%	blunt · at its level
RULER 64k	88.4% ✎	▁▁·▁·▂▃▄█▃·▁ beats 21/22	easy · avg 83%	▇ 0.4%	typical
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	93.9% ✎	▁▁▁▁▁▁▁▂▄█▄▃ beats 133/146	easy · avg 83%	▇ 0.2%	noisy
RULER 128k	66.6% ✎	▄··▂▄▂▆██▄▄▂ beats 9/21	easy · avg 68%	▇ 0.1%	at its level · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	95.0% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 59/140	very easy · avg 94%	▇ 0.1%	blunt · maxed out · noisy

$ where to run it

Through an API

api · Amazon Bedrock · $0.72/M · $0.72 in / $0.72 out per 1M tokens · 128k context · docs.aws.amazon.com ↗ · price sourceStandard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ.

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Llama 3.1 70B Instruct · needs ~42.5 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Meta-Llama-3.1-70B-Instruct-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Meta-Llama-3.1-70B-Instruct-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Llama 3.1 70B Instruct · needs ~42.5 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Meta-Llama-3.1-70B-Instruct-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Llama 3.1 70B Instruct · needs ~42.5 GB · from ollama.com
chat in the terminal
ollama run llama3.1:70b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Llama 3.1 70B Instruct · needs ~39.7 GB, Mac only · from huggingface.co/mlx-community
chat in the terminal
mlx_lm.chat --model mlx-community/Meta-Llama-3.1-70B-Instruct-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model mlx-community/Meta-Llama-3.1-70B-Instruct-4bit

Weights and documentation: huggingface.co/meta-llama ↗

$ how the score was worked out · WildEval ZeroEval published r… 21%, Vector Institute Evaluation L… 10%, 2 OLMo 2 Furious (Team OLMo,… 9% and 25 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
WildEval ZeroEval published results	5	▇▇▇▇▇▇ 21%	Apache-2.0
Vector Institute Evaluation Leaderboard (Inspect Evals runs)	4	▇▇▇ 10%	Apache-2.0
2 OLMo 2 Furious (Team OLMo, Ai2) — authors' own evaluations of OLMo 2 Instruct and peer instruct models	8	▇▇▇ 9%	CC BY 4.0
LiveBench model judgments (HF livebench/model_judgment, 2024-10-22 revision) with the LiveBench question datasets	1	▇▇▇ 9%	Apache-2.0 (stated in the LiveBench datasheet)
Epoch AI — internal evaluations	3	▇▇▇ 8%	CC BY 4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)	1	▇▇ 5%	Apache-2.0
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (arXiv:2411.19146v5) · lab-reported	5	▇ 5%	CC-BY-4.0
LiveCodeBench official website results (2024-12-26)	1	▇ 5%	CC-BY-SA-4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	3	▇ 4%	Apache-2.0
Aider code editing benchmark runs (old edit leaderboard)	1	▇ 4%	Apache-2.0
MMLU-Pro author-released evaluation runs	1	▇ 3%	Apache-2.0
ServiceNow BrowserGym leaderboard (AgentLab GenericAgent runs)	4	▇ 3%	MIT
LMArena leaderboard dataset	1	▇ 2%	CC BY 4.0
MASSIVE-Agents — Kulkarni et al. (2025)	1	▇ 2%	CC BY 4.0
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (arXiv:2503.16167v2)	4	▇ 2%	CC-BY-4.0
SpeakLeash CPTUB (Complex Polish Text Understanding Benchmark)	1	▇ 1%	MIT
EvalPlus / EvalPerf published evaluation runs	1	▇ 1%	Apache-2.0
Berkeley Function Calling Leaderboard result archive	2	▇ 1%	Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 1%	MIT
SpeakLeash Polish EQ-Bench leaderboard	1	▇ 1%	Apache-2.0
WebApp1K models leaderboard (onekq-ai)	1	▇ 1%	MIT
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 1%	Apache-2.0
SciCode leaderboard (repository README)	1	▇ 1%	Apache-2.0
NVIDIA RULER repository README results table	2	▇ 1%	Apache-2.0
LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? (arXiv:2610.04118v1)	1	▇ 0%	CC BY 4.0
Open Portuguese LLM Leaderboard (archived)	1	▇ 0%	Apache-2.0
EuroEval leaderboards (results collection), English generative leaderboard	1	▇ 0%	MIT
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 0%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 52.4