Qwen3 0.6B

+ contribute

Qwen3 0.6B is an open-weights language model from Alibaba, released April 2025. Its published size is 0.75B parameters and a 33k-token context window. Licence: apache-2.0.

33.4±1.0#613 of 719 LLMsfrom 26 benchmark results across 10 sources

$ how it compares

The LLMs ranked either side of Qwen3 0.6B, out of 828 with a score, then more from Alibaba.

#	model	score	where	RAM	released
707	Zephyr 7B beta Hugging Face	34±1	local	~6.1GB	2023-10
708	PaLM 2 Chat (chat-bison@001) Google	34±2	comparison only		2023-05
709	Qwen3 0.6B Alibaba	33±1	local	~1.8GB	2025-04
710	Llama 2 13B Chat Meta	33±1	local	~9.8GB	2023-07
711	Flan-T5 XXL Google	33±1	local	~49GB	2022-10
	more from Alibaba
18	Qwen3.8 Max (2.4T-A95B) Alibaba	78±1	api $3.00/M +local	~380GB	2026-08
34	Qwen3.8 Max (0902) Alibaba	76±1	api $3.00/M		2026-09
37	Qwen3.8-Flash-Next Alibaba	75±1	local	~59GB	2026-08

see all LLMs · find the ones that fit your machine

$ what it's good at

agent tasks▁▁▃▆█▆▅▅▅▄▃▁ #266 of 300 * · 3.7% of its score · 2 tests

Beats Magistral Small 1.0 (2506), Llama 4 Scout (109B-A17B), Bielik 11B v2.3 Instruct here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
BFCL V4 (December 2025 archive)	23.9% ✎	▁▁▁▂██▅▅▆▂▂▂ beats 12/77	hard · avg 32%	▇ 2.0%	typical
BFCL v3	47.3% ✎	▁▁▁▁▁▃▅█▅▄▁▁ beats 29/114	medium · avg 52%	▇ 1.7%	blunt · at its level

coding▁▁▃▃▆██▇▆▄▃▁ #451 of 458 * · 9.3% of its score · 2 tests

Beats Molmo 7B-D (0924), DeepSeek R1 Distill Qwen 1.5B here, though they rank higher overall. Trails Phi-1 (1.3B), OpenAssistant SFT-4 Pythia 12B, Falcon3 1B Instruct here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval+	14.8% ✎	▃·······▂·▂█ beats 0/6	easy · avg 80%	▇▇▇ 6.4%	sharp · at its level
LiveCodeBench v3	34.4% ✎	▃▃·····▃·▃·█ beats 1/6	easy · avg 65%	▇ 2.9%	at its level

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #301 of 393 * · 4.0% of its score · 1 test

Beats Ministral 3 14B Reasoning 2512, Gemma 4 E4B, OpenThinker3 7B here, though they rank higher overall. Trails LFM2 350M, Gemma 3 1B, Flan-T5 XXL here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
IFEval (aggregation unspecified) +1	62.3% ✎	▄▄·····█▄▄·▄ beats 1/6	easy · avg 82%	▇▇ 4.0%	at its level · several runs

math▁▂▃▅▆▅▄█▄▁▁▁ #327 of 398 · 26% of its score · 4 tests

Beats GPT-3.5 Turbo (0125), Qwen2-VL 7B Instruct, DBRX Instruct here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
AIME 2024	11.2% ✎	▄·····▄··▄██ beats 0/6	medium · avg 59%	▇▇▇ 8.4%	sharp
MGSM	41.3% ✎	█···█▄··▄··▄ beats 2/6	easy · avg 77%	▇▇▇ 8.3%	at its level
AIME 2025	15.2% ✎	▄··▄····█▄▄▄ beats 0/6	medium · avg 51%	▇▇ 5.7%	sharp
GSM8K	36.5% ✎	▄·█···▄·▄··█ beats 2/6	easy · avg 83%	▇▇ 4.1%	sharp · at its level

other tests14 not in a task · 56% of its score

test	result	vs the others in its run	difficulty	counts	why
MMLU +1	55.8% ✎	▄···▄··█··▄█ beats 0/6	easy · avg 77%	▇▇▇▇▇▇ 15%	at its level · several runs
BIG-Bench Hard	41.5% ✎	▄·····▄·▄·██ beats 0/6	easy · avg 78%	▇▇▇▇▇ 14%	sharp
GPQA (subset unspecified) +1	27.2% ✎	▃······▃·▅·█ beats 0/6	hard · avg 43%	▇▇ 4.6%	sharp · too hard for it · several runs
MMMLU	30.8% ✎	▄···▄·▄▄··▄█ beats 0/6	medium · avg 59%	▇ 3.5%	typical
Open Persian LLM Leaderboard v2 average (21 tasks)	36.3% ✎	▂▄▂▃█▄▃▆▃▅▂▂ beats 7/81	medium · avg 58%	▇ 3.4%	blunt · at its level
IFBench (aggregation unspecified)	19.8% ✎	▄▄·▄····▄·█▄ beats 3/6	medium · avg 40%	▇ 3.2%	typical
BRIDGE clinical text, zero-shot average (87 tasks)	20.4% ✎	▁▁▁▁▁▃▂▃▄▆█▃ beats 11/97	hard · avg 32%	▇ 2.9%	blunt
ZebraLogic	29.8% ✎	█▄····▄·▄·▄▄ beats 0/6	medium · avg 53%	▇ 2.3%	typical

6 more tests

test	result	vs the others in its run	difficulty	counts	why
Open PT-LLM Leaderboard average (9 Portuguese tasks)	50.3% ✎	▃▁▁▁▁▁▁▃▂█▅▁ beats 68/276	medium · avg 73%	▇ 1.5%	at its level · noisy
KINA (Knowledge Index of Noah's Ark) accuracy, 899 ten-option items	14.5% ✎	▃▁▂▁▃▄█▆▄▅▂▃ beats 0/39	very hard · avg 22%	▇ 1.5%	blunt · too hard for it
AraGen-v3 3C3H score (Inception, Claude Sonnet 4 judge)	1.6% ✎	▁▃·▁▂▂▄█▄▅▄▄ beats 1/66	hard · avg 31%	▇ 1.4%	typical
EuroEval Life in the UK (English knowledge MC), test split, 5-shot, accuracy	61.9% ✎	▁▁▁▁▁▁▁▃▄█▄▃ beats 32/146	easy · avg 83%	▇ 1.3%	at its level · noisy
Vectara hallucination leaderboard, 2025 dataset (HHEM-2.3, factual consistency rate)	96.3% ✎	▁▁▂▃▄▄█▆▃▂▂▁ beats 91/140	very easy · avg 94%	▇ 1.0%	blunt · noisy
EuroEval HellaSwag (English), test split, 5-shot generative MC, accuracy	38.6% ✎	▃·▁▁▂▃▂▅█▇▄▁ beats 19/100	medium · avg 68%	▇ 0.9%	noisy

$ where to run it

llama.cpp · recommended

version

Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
Windows
irm https://llama.app/install.ps1 | iex
other ways to install ↗
Run Qwen3 0.6B · needs ~0.5 GB · from huggingface.co/lmstudio-community
chat in the terminal
llama cli -hf lmstudio-community/Qwen3-0.6B-GGUF:Q4_K_M
Or start a local server and open the address it prints (http://localhost:9931)
llama serve -hf lmstudio-community/Qwen3-0.6B-GGUF:Q4_K_M

LM Studio

version

Install LM Studio · Open the app once before using the lms command.
download it (macOS / Windows / Linux) ↗
Run Qwen3 0.6B · needs ~0.5 GB · from huggingface.co/lmstudio-community
open in LM Studio ↗
Or download it in the terminal
lms get https://huggingface.co/lmstudio-community/Qwen3-0.6B-GGUF@Q4_K_M -y
Then run lms ls to see its name, and lms chat followed by that name

Ollama

version

Install Ollama
download it (macOS / Windows) ↗
Linux
curl -fsSL https://ollama.com/install.sh | sh
Run Qwen3 0.6B · needs ~0.95 GB · from ollama.com
chat in the terminal
ollama run qwen3:0.6b

MLX (Mac)

version

Install MLX (Mac) · Apple silicon Macs only, text only.
with uv
uv tool install mlx-lm
or
pip install mlx-lm
Run Qwen3 0.6B · needs ~0.3 GB, Mac only · from huggingface.co/lmstudio-community
chat in the terminal
mlx_lm.chat --model lmstudio-community/Qwen3-0.6B-MLX-4bit
Or start an OpenAI-compatible server at http://localhost:8080/v1
mlx_lm.server --model lmstudio-community/Qwen3-0.6B-MLX-4bit

Official weights and documentation: huggingface.co/Qwen ↗

$ how the score was worked out · Olmo 3 51%, LiquidAI/LFM2-1.2B model card 31%, Berkeley Function Calling Lea… 4% and 7 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
Olmo 3	9	▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 51%	CC BY 4.0
LiquidAI/LFM2-1.2B model card	7	▇▇▇▇▇▇▇▇▇ 31%	LFM Open License v1.0
Berkeley Function Calling Leaderboard result archive	2	▇ 4%	Apache-2.0
Open Persian LLM Leaderboard v2 (Part DP AI and AUT NLP Lab)	1	▇ 3%	Apache-2.0
BRIDGE Medical Leaderboard (Harvard Medical School / Brigham and Women's YLab)	1	▇ 3%	MIT
EuroEval leaderboards (results collection), English generative leaderboard	2	▇ 2%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
Knowledge Index of Noah's Ark (arXiv:2606.05104v2)	1	▇ 1%	CC BY 4.0
Inception Arabic/Hindi leaderboards (AraGen-v3, HindiGen-v1, Arabic IFEval)	1	▇ 1%	Apache-2.0
Vectara hallucination leaderboard (vectara/hallucination-leaderboard repository)	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 33.4

$ published speed measurements · 19

Reference results for the hardware, engine and settings below. They aren't verified speeds of the downloads above or predictions for your machine.

AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU; Beelink GTR9 Pro 128GB	Q8_0	208.73 tok/s	llama.cpp 32e789fdf (build_number 10046 as recorded), ROCm/HIP · llama-bench; tg128; 128 generated tokens; context depth 0; n_prompt=0 row; n_batch=2048; n_ubatch=512; flash_attn=-1; type_k=f16; type_v=f16; n_gpu_layers=999; threads=16; one sequence Author's own llama-bench CSV row (avg_ts mean over repetitions, stddev_ts retained). GPU string as recorded: Radeon 8060S Graphics. Unified memory; the guide reports ~124GiB OS-visible. Kernel/Mesa/power profile per the run folder notes. Not prompt processing or multi-request throughput. Curated in the guide's data/benchmarks.csv line 9 (2026-07-16, measured-local): Official b10046 ROCm/HIP historical allocation sentinel. Full 120124MiB free UMA was detected and a CPU-heavy smoke logged ROCm_Host model output and compute buffers without HSA_OVERRIDE_GFX_VERSION. Open issue 26209 and PR 25863 mean this tiny smoke does not qualify long-context multimodal multi-slot or practical-model correctness. Requires the existing Ollama ROCm library path; not a speed headline or Vulkan comparison.	source ↗
NVIDIA DGX Spark (GB10)	BF16	119.4 tok/s	llama.cpp (CUDA build, GGUF BF16 weights; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	103.3 tok/s	vLLM (CUDA, Safetensors BF16; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	106.8 tok/s	SGLang (CUDA, Safetensors BF16; version not stated) · chat split (OpenOrca + CNN/DailyMail, 21-3.5K input tokens); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	55.3 tok/s	llama.cpp (CUDA build, GGUF BF16 weights; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	80.2 tok/s	vLLM (CUDA, Safetensors BF16; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
NVIDIA DGX Spark (GB10)	BF16	82.6 tok/s	SGLang (CUDA, Safetensors BF16; version not stated) · agent split (BFCL V3, Hermes, ClawsBench; 0.3K-8.9K input tokens; agent context raised to 16384 for llama.cpp/ollama/vllm-metal); SiliconBench DGX Spark reference track (--platform dgxspark); identical prompts and concurrency grid as the Apple track; n=100 per level; concurrency 1; output throughput = generated tokens / sweep wall time (includes prefill); runs of 2026-07-03 to 2026-07-05 All cells in this table completed 100/100 requests.	source ↗
Apple M4 Pro (16-core GPU, 24GB)	Q4 GGUF (exact llama.cpp type not stated)	297.4 tok/s	llama.cpp b9630 (June 2026), Metal backend, default settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M4 Pro (16-core GPU, 24GB)	Q4 MLX (4-bit; group size not stated)	343.6 tok/s	MLX v0.31.2 via mlx-lm, default generation settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M4 Pro (16-core GPU, 24GB)	Q8 GGUF (exact llama.cpp type not stated)	219.8 tok/s	llama.cpp b9630 (June 2026), Metal backend, default settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M4 Pro (16-core GPU, 24GB)	Q8 MLX (8-bit; group size not stated)	255.3 tok/s	MLX v0.31.2 via mlx-lm, default generation settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M3 (8-core GPU, 16GB)	Q4 GGUF (exact llama.cpp type not stated)	180.5 tok/s	llama.cpp b9630 (June 2026), Metal backend, default settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M3 (8-core GPU, 16GB)	Q4 MLX (4-bit; group size not stated)	182.3 tok/s	MLX v0.31.2 via mlx-lm, default generation settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M3 (8-core GPU, 16GB)	Q8 GGUF (exact llama.cpp type not stated)	99.8 tok/s	llama.cpp b9630 (June 2026), Metal backend, default settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M3 (8-core GPU, 16GB)	Q8 MLX (8-bit; group size not stated)	113.6 tok/s	MLX v0.31.2 via mlx-lm, default generation settings · llama-bench-style tg128: decode throughput over 128 generated tokens; std dev generally below 3% for decode (paper); baselines at default settings Baseline runtime column from the BaseRT paper; BaseRT's own runtime column is not recorded.	source ↗
Apple M5 Pro (48GB; CPU/GPU core counts not stated)	Q4 GGUF (4-bit; exact llama.cpp type not stated)	386.1 tok/s	llama.cpp b9960, Metal backend, default settings · tg128: token-generation throughput over 128 generated tokens; average of five repetitions; macOS (Darwin 25.4) Baseline runtime column from the BaseRT M5 paper; BaseRT's own column is not recorded.	source ↗
Apple M5 Pro (48GB; CPU/GPU core counts not stated)	MLX 4-bit (group size not stated)	398.1 tok/s	mlx-lm 0.31.3 (MLX 0.32.0), default generation settings · tg128: token-generation throughput over 128 generated tokens; average of five repetitions; macOS (Darwin 25.4) Baseline runtime column from the BaseRT M5 paper; BaseRT's own column is not recorded.	source ↗
Apple M5 Pro (48GB; CPU/GPU core counts not stated)	Q8 GGUF (8-bit; exact llama.cpp type not stated)	285.3 tok/s	llama.cpp b9960, Metal backend, default settings · tg128: token-generation throughput over 128 generated tokens; average of five repetitions; macOS (Darwin 25.4) Baseline runtime column from the BaseRT M5 paper; BaseRT's own column is not recorded.	source ↗
Apple M5 Pro (48GB; CPU/GPU core counts not stated)	MLX 8-bit (group size not stated)	295.5 tok/s	mlx-lm 0.31.3 (MLX 0.32.0), default generation settings · tg128: token-generation throughput over 128 generated tokens; average of five repetitions; macOS (Darwin 25.4) Baseline runtime column from the BaseRT M5 paper; BaseRT's own column is not recorded.	source ↗

$ quantization evidence · 12 measurements

How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent.

precision	test	full → quantized	kept
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)	OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)	0.378 → 0.378	100.1%	source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)	MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)	0.428 → 0.423	98.8%	source ↗
FP8 dynamic (weights FP8 per-channel static, activations FP8 per-token dynamic; llm-compressor QuantizationModifier)	GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768	0.293 → 0.258	87.9%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	GPQA (subset not stated in card), thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.279 → 0.293	105.0%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	GPQA (subset not stated in card), non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.229 → 0.232	101.3%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	MMLU-Redux, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.446 → 0.449	100.7%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	LiveBench 2024-11-25, non-thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.218 → 0.218	100.0%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	LiveBench 2024-11-25, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.303 → 0.302	99.7%	source ↗
GPTQ INT8 (w8, group_size 128, sym, desc_act=false; GPTQModel 2.2.0; checkpoint Qwen/Qwen3-0.6B-GPTQ-Int8)	MMLU-Redux, thinking mode (Qwen-reported in quantized model card; harness, sampling and number of runs not stated)	0.556 → 0.538	96.8%	source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)	OpenLLM v1 average of 6 tasks (MMLU (5-shot); ARC Challenge (25-shot); GSM-8K (5-shot, strict-match); Hellaswag (10-shot); Winogrande (5-shot); TruthfulQA (0-shot, mc2)) as published; lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)	0.378 → 0.362	95.8%	source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)	GPQA Diamond — published row 'GPQA diamond' (Reasoning section); lighteval + vLLM, temperature 0.6, top_p 0.95, top_k 20, max_new_tokens 32768	0.293 → 0.278	94.8%	source ↗
INT4 W4A16 (weight-only INT4, asymmetric, group 64; GPTQ, llm-compressor)	MMLU — published row 'MMLU (5-shot)' (OpenLLM v1 section); lm-evaluation-harness + vLLM (backend vllm, max_model_len 8192, chat template, few-shot as multiturn)	0.428 → 0.398	92.9%	source ↗

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json