GPT-Neo 2.7B

+ contribute

GPT-Neo 2.7B is an open-weights language model from EleutherAI, released March 2021. Its published size is 2.7B parameters and a 2k-token context window. Licence: mit.

18.2±1.0#694 of 719 LLMsfrom 25 benchmark results across 7 sources

$ how it compares

The LLMs ranked either side of GPT-Neo 2.7B, out of 828 with a score, then more from EleutherAI.

#	model	score	where	RAM	released
798	OPT-2.7B Meta	19±1	local	~7.1GB	2022-05
799	SmolLM2 360M Instruct Hugging Face	19±3	local	~1.7GB	2024-10
800	GPT-Neo 2.7B EleutherAI	18±1	local	~13GB	2021-03
801	GPT-3 babbage (API base model) OpenAI	18±1	comparison only		2020-06
802	Pythia 1.4B EleutherAI	18±2	local	~2.4GB	2023-02
	more from EleutherAI
756	GPT-NeoX-20B EleutherAI	28±1	local	~14GB	2022-02
777	Pythia 12B EleutherAI	25±1	local	~9.5GB	2023-02
780	GPT-J 6B EleutherAI	25±1	local	~14GB	2021-06

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▃▁▃▆██▇▆▄▃▁ #454 of 458 * · 34% of its score · 2 tests

Beats DeepSeek R1 Distill Qwen 1.5B, ChatGLM-6B here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
HumanEval	7.9% ✎	▁·▃▁▃▄▆▅█▃▁▁ beats 2/99	easy · avg 79%	▇▇▇▇▇▇▇▇ 20%	sharp · too hard for it
HumanEval+	6.7% ✎	▁·▃▁▁▃▄▄▄█▃▁ beats 2/99	easy · avg 74%	▇▇▇▇▇ 14%	sharp · too hard for it

math█···█······█ #1 of 3 * · 2.4% of its score · 1 test

test	result	vs the others in its run	difficulty	counts	why
MathQA lab	24.7% ✎	█···█··█···█ beats 3/3	hard · avg 46%	▇ 2.4%	too hard for it

other tests15 not in a task · 63% of its score

test	result	vs the others in its run	difficulty	counts	why
HellaSwag +2	42.7% ✎	▃▃▃··▅····▃█ beats 2/8	very easy · avg 92%	▇▇▇▇▇▇ 14%	at its level · several runs
WinoGrande +2	57.7% ✎	█··█···█▄▄·▄ beats 0/8	easy · avg 90%	▇▇▇▇▇ 14%	several runs
PIQA +2	72.9% ✎	▄▄··▄·▄·▄·██ beats 2/8	easy · avg 88%	▇▇ 6.0%	blunt · at its level · several runs
LAMBADA (OpenAI version), last-word accuracy lab +1	62.2% ✎	█·▄··█··█··▄ beats 3/7	very easy · avg 90%	▇▇ 5.3%	blunt · at its level · several runs
BoolQ	61.8% ✎	▄█▄···▄··█▄▄ beats 2/8	easy · avg 92%	▇▇ 5.1%	typical
Natural Questions (closed book)	5.7% ✎	▁·▁·▁▂▃▂·▅▂█ beats 6/28	medium · avg 58%	▇▇ 4.5%	too hard for it
ARC Easy	61.1% ✎	▅·▃▃··▃··▃·█ beats 2/8	very easy · avg 90%	▇▇ 4.3%	at its level
ARC Challenge	27.4% ✎	▅····▃▃···▅█ beats 2/8	easy · avg 79%	▇▇ 4.0%	sharp · too hard for it

7 more tests

test	result	vs the others in its run	difficulty	counts	why
Social IQA	40.0% ✎	█▄····█▄▄··▄ beats 1/7	medium · avg 63%	▇ 2.0%	blunt
OpenBookQA	23.2% ✎	█······▂·▆·▂ beats 1/8	medium · avg 54%	▇ 1.3%	too hard for it · noisy
TriviaQA	20.1% ✎	▁··▁█▃▃▁▄▅▄█ beats 11/28	easy · avg 79%	▇ 0.9%	noisy
Open PT-LLM Leaderboard average (9 Portuguese tasks)	28.0% ✎	▃▁▁▁▁▁▁▁▂█▅▁ beats 20/276	medium · avg 73%	▇ 0.7%	noisy
PubMedQA lab	57.5% ✎	▆·▄▆·▄▂▄▄▂█▂ beats 3/20	medium · avg 75%	▇ 0.7%	blunt · at its level
TruthfulQA MC2	39.9% ✎	▅▃█▆▁▂▅···▁▁ beats 9/31	medium · avg 64%	▇ 0.7%	blunt · at its level
HaluEval QA hallucination detection (accuracy)	43.2% ✎	█▁▁·▁▁·▁▁▁▁▁ beats 13/27	medium · avg 70%	▇ 0.2%	sharp · too hard for it · noisy

$ where to run it

Other ways to run it

Run it as published · needs ~10.67 GB · from huggingface.co/EleutherAI (official)
uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"

Official weights and documentation: huggingface.co/EleutherAI ↗

$ how the score was worked out · EvalPlus / EvalPerf published… 34%, Textbooks Are All You Need II… 28%, EleutherAI GPT-J-6B model car… 17% and 4 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
EvalPlus / EvalPerf published evaluation runs	2	▇▇▇▇▇▇▇▇▇▇ 34%	Apache-2.0
Textbooks Are All You Need II: phi-1.5 technical report — authors' own runs	8	▇▇▇▇▇▇▇▇ 28%	CC BY 4.0
EleutherAI GPT-J-6B model card — EleutherAI lm-evaluation-harness runs · lab-reported	4	▇▇▇▇▇ 17%	Apache-2.0
RedPajama — original zero-shot 3B comparison	4	▇▇▇ 10%	CC BY 4.0
Hallucinations Leaderboard results (HF dataset hallucinations-leaderboard/results)	4	▇▇ 6%	Apache-2.0
EleutherAI GPT-Neo 2.7B model card — EleutherAI evaluation-harness runs · lab-reported	2	▇ 3%	MIT
Open Portuguese LLM Leaderboard (archived)	1	▇ 1%	Apache-2.0
prior · pull toward the middle		▇ 0%
weighted average		= 18.2

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-10-11 02:20 UTC · sources and licences · json