whatmodel.sh / OpenAssistant Pythia 12B SFT v8 (7k steps)

OpenAssistant Pythia 12B SFT v8 (7k steps)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

OpenAssistant Pythia 12B SFT v8 (7k steps) is an open-weights language model from LAION (OpenAssistant), released May 2023. Licence: apache-2.0.

Catalogue identity notes

Maker repo OpenAssistant/pythia-12b-sft-v8-7k-steps @275c9b7: licence tag apache-2.0. Released 2023-05: month the maker's Hugging Face repo was created (createdAt 2023-05-07).

26.1±2.8#890 of 949 LLMsfrom 2 benchmark results across 1 source* thinly tested, treat the rank as provisional

$ how it compares

The LLMs ranked either side of OpenAssistant Pythia 12B SFT v8 (7k steps), out of 1132 with a score, then more from LAION (OpenAssistant).

#modelscorewhereRAMreleased
1065GPT4All-13B-Snoozy Nomic AI26±2local~56GB2023-04
1066OPT-13B (13 billion parameters) Meta26±1local~29GB2022-05
1067OpenAssistant Pythia 12B SFT v8 (7k steps) LAION (OpenAssistant)26±3local~27GB2023-05
1068Granite 4.0 350M IBM26±1local~1.7GB2025-10
1069GPT-3 curie (API base model) OpenAI26±1comparison only2020-06
more from LAION (OpenAssistant)
1087OpenAssistant SFT-4 Pythia 12B LAION (OpenAssistant)22±1local~9.5GB2023-04

see all LLMs · find the ones that fit your machine

$ what it's good at

chat & writing▁▃▁▄▆▅▆█▆▆▆▂ #458 of 466 * · 97% of its score · 2 tests

Beats Phi-2 (2.7B) here, though they rank higher overall. Trails Gemma 3 270M IT, SmolLM2 360M Instruct, Falcon 7B Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AlpacaEval 2 length-controlled4.2% ✎▁▃▂▃▆▆█▆▄▄▁▁ beats 4/122medium · avg 39%▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 53%too hard for it
AlpacaEval 1.0 win rate (vs text-davinci-003, GPT-4 judge)41.9% ✎▂▂·▆·▇▃▆█▆▄▂ beats 4/45very easy · avg 96%▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 43%sharp · at its level · noisy

$ where to run it

Other ways to run it

  1. Run it as published · needs ~23.84 GB · from huggingface.co/OpenAssistant
    uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='OpenAssistant/pythia-12b-sft-v8-7k-steps', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"

Weights and documentation: huggingface.co/OpenAssistant ↗

$ how the score was worked out · AlpacaEval 1.0 and 2.0 leader… 97%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
AlpacaEval 1.0 and 2.0 leaderboards (tatsu-lab/alpaca_eval repository)2▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 97%Apache-2.0
prior · pull toward the middle▇ 3%
weighted average= 26.1

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json