whatmodel.sh / OpenAssistant SFT-4 Pythia 12B

OpenAssistant SFT-4 Pythia 12B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

OpenAssistant SFT-4 Pythia 12B is an open-weights language model from LAION (OpenAssistant), released April 2023. Its published size is 12B parameters. Licence: apache-2.0.

Catalogue identity notes

Comparison-only. Arena entry 'oasst-pythia-12b' (text). LMArena's leaderboard table (arena-leaderboard space @6f2eeeb) labels 'oasst-pythia-12b' as 'OpenAssistant-Pythia-12B' (OpenAssistant, Apache 2.0), linking https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5. Maker repo OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 @626b8c1: licence tag apache-2.0. Parameters: nominal size from the card ('based on a Pythia 12B'). Arena key 'oasst-pythia-12b'; LMArena's table links OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5.

Model information sources

Release date ↗

21.8±1.4#682 of 719 LLMsfrom 9 benchmark results across 6 sources

$ how it compares

The LLMs ranked either side of OpenAssistant SFT-4 Pythia 12B, out of 828 with a score.

#modelscorewhereRAMreleased
786Qwen1.5 1.8B Chat Alibaba24±2local~2.7GB2024-01
787Pythia 6.9B EleutherAI23±1local~6.1GB2023-02
788OpenAssistant SFT-4 Pythia 12B LAION (OpenAssistant)22±1local~9.5GB2023-04
789Pythia 2.8B EleutherAI21±1local~3.4GB2023-02
790BLOOM 7.1B BigScience21±1local~7.1GB2022-07

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▁▁▃▃▆██▇▆▄▃▁ #447 of 458 * · 12% of its score · 1 test

Beats Molmo 7B-D (0924), DeepSeek R1 Distill Qwen 1.5B, Qwen3 0.6B here, though they rank higher overall.

testresultvs the others in its rundifficultycountswhy
HumanEval9.1% ✎▄··▄····▄▄█▄ beats 3/6easy · avg 71%▇▇▇▇▇ 12%sharp · too hard for it
chat & writing▁▃▁▃▅▄▅█▆▆▇▃ #390 of 393 · 71% of its score · 6 tests

Trails Gemma 3 270M IT, SmolLM2 360M Instruct here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)912.0 ✎▂▃▂▁▃▇█▆▇▄▂▂ beats 5/95avg 1201▇▇▇▇▇▇▇ 18%blunt
LMArena (text)1023.5 ✎▁▃▁▂▃▃▃▆▆▇█▃ beats 5/379avg 1268▇▇▇▇▇▇▇ 18%blunt
MT-Bench (GPT-4 single-answer grading, LMSYS runs)4.3200 MT-Bench score (1-10) ✎▆▂▂▆▆▄██▂▄▄▆ beats 5/28avg 8.819 MT-Bench score (1-10)▇▇▇▇▇▇ 16%sharp · noisy
AlpacaEval 1.0 win rate (vs text-davinci-003, GPT-4 judge)26.0% ✎▂▃·▆·▇▃▆█▆▄▂ beats 2/45very easy · avg 96%▇▇▇▇ 10%sharp · at its level · noisy
AlpacaEval 2 length-controlled3.3% ✎▁▃▂▃▆▆█▆▄▄▁▁ beats 2/122medium · avg 39%▇▇▇▇ 8.9%too hard for it
MixInstruct GPT-Rank (LLM-Blender, mean rank among 11 instruction-tuned LLMs, ChatGPT pairwise judge)3.9000 mean rank (1-11) ✎█···▄··▄···▄ beats 4/4avg 6.444 mean rank (1-11)▇ 0.0%blunt · noisy
other tests2 not in a task · 16% of its score
testresultvs the others in its rundifficultycountswhy
BIG-Bench Hard30.0% ✎███··█·█·█·█ beats 0/6easy · avg 76%▇▇▇▇ 11%sharp · too hard for it
MMLU27.0% ✎▄▄·▄···▄▄·██ beats 2/8easy · avg 77%▇▇ 5.3%typical

$ where to run it

llama.cpp · recommended

  1. Install llama.cpp · Homebrew and winget installs name the commands llama-cli and llama-server instead.
    macOS / Linux
    curl -LsSf https://llama.app/install.sh | sh
    Windows
    irm https://llama.app/install.ps1 | iex

    other ways to install ↗

  2. Run OpenAssistant SFT-4 Pythia 12B · needs ~7.6 GB · from huggingface.co/ArtyomSubDiv
    chat in the terminal
    llama cli -hf ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m
  3. Or start a local server and open the address it prints (http://localhost:9931)
    llama serve -hf ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m

LM Studio

  1. Install LM Studio · Open the app once before using the lms command.

    download it (macOS / Windows / Linux) ↗

  2. Run OpenAssistant SFT-4 Pythia 12B · needs ~7.6 GB · from huggingface.co/ArtyomSubDiv

    open in LM Studio ↗

  3. Or download it in the terminal
    lms get https://huggingface.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF@q4_k_m -y
  4. Then run lms ls to see its name, and lms chat followed by that name

Ollama

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run OpenAssistant SFT-4 Pythia 12B · needs ~7.6 GB · from huggingface.co/ArtyomSubDiv
    chat in the terminal
    ollama run hf.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m

Other ways to run it

  1. Run it as published · needs ~23.84 GB · from huggingface.co/OpenAssistant
    uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5', torch_dtype='auto', device_map='auto')('<|prompter|>What is a meme?<|endoftext|><|assistant|>', max_new_tokens=40)[0]['generated_text'])"

Weights and documentation: huggingface.co/OpenAssistant ↗

$ how the score was worked out · InstructEval (DeCLaRe Lab) RE… 28%, AlpacaEval 1.0 and 2.0 leader… 19%, Polyrating: A Cost-Effective… 18% and 3 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
InstructEval (DeCLaRe Lab) README results table3▇▇▇▇▇▇▇▇ 28%MIT / Apache-2.0
AlpacaEval 1.0 and 2.0 leaderboards (tatsu-lab/alpaca_eval repository)2▇▇▇▇▇▇ 19%Apache-2.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)1▇▇▇▇▇ 18%CC-BY-4.0
LMArena leaderboard dataset1▇▇▇▇▇ 18%CC BY 4.0
LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table1▇▇▇▇▇ 16%Apache-2.0
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (arXiv:2306.02561v3)1▇ 0%CC-BY-4.0
prior · pull toward the middle▇ 1%
weighted average= 21.8

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json