whatmodel.sh / StableLM-Base-Alpha 7B

StableLM-Base-Alpha 7B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

StableLM-Base-Alpha 7B is an open-weights language model from Stability AI, released April 2023. Its published size is 7.9B parameters and a 4k-token context window. Licence: cc-by-sa-4.0.

Catalogue identity notes

First StableLM alpha base release (April 2023), not the -v2 re-release.

9.3±1.2#708 of 719 LLMsfrom 16 benchmark results across 4 sources

$ how it compares

The LLMs ranked either side of StableLM-Base-Alpha 7B, out of 828 with a score, then more from Stability AI.

#modelscorewhereRAMreleased
815Pythia 410M EleutherAI10±2local~1.8GB2023-02
816GPT-3 ada (API base model) OpenAI10±2comparison only2020-06
817StableLM-Base-Alpha 7B Stability AI9±1local~35GB2023-04
818Zyte-1B venkycs (aihub-app)8±2local~2.2GB2024-01
819GPT-Neo 125M EleutherAI5±5local~2.1GB2021-03
more from Stability AI
718Stable Code 3B Stability AI33±1local~3.3GB2024-01
746StableLM-3B-4E1T Stability AI29±1local~3.3GB2023-09
750StableLM-Base-Alpha 7B v2 Stability AI29±1local~30GB2023-08

see all LLMs · find the ones that fit your machine

$ what it's good at

coding▃▁▁▃▆██▇▆▄▃▁ #457 of 458 · 20% of its score · 2 tests
testresultvs the others in its rundifficultycountswhy
HumanEval +12.4% ✎▃·▁▁▃▄▆▅█▃▁▁ beats 1/99easy · avg 79%▇▇▇▇▇ 13%sharp · too hard for it · several runs
HumanEval+2.4% ✎▃·▁▁▁▃▄▄▄█▃▁ beats 1/99easy · avg 74%▇▇▇ 7.5%sharp · too hard for it
other tests13 not in a task · 79% of its score
testresultvs the others in its rundifficultycountswhy
SciQ lab80.1% ✎▁·▃·▁·▃▅▃█▄▁ beats 1/26very easy · avg 98%▇▇▇▇▇▇▇▇ 19%at its level
ARC Challenge lab27.0% ✎▁·▃··▁▁▇▂█▆▃ beats 1/26easy · avg 85%▇▇▇▇ 11%sharp · too hard for it
ARC Easy lab44.9% ✎▁▃·▁▁▄▅▄█▅▁▃ beats 1/26easy · avg 88%▇▇▇▇ 10%at its level
HellaSwag lab41.2% ✎▃···▁▇▅█▇▃▁▁ beats 1/26very easy · avg 92%▇▇▇ 8.1%at its level
BoolQ lab60.1% ✎▂▃▂▅▆▂▅▆█▂▂▂ beats 1/26easy · avg 91%▇▇▇ 7.4%typical
WinoGrande lab50.1% ✎▃····▁··▁▂█▄ beats 0/26easy · avg 89%▇▇▇ 6.6%typical
PIQA lab66.8% ✎▁·▃··▁▁▃█▅█▂ beats 1/26easy · avg 88%▇▇▇ 6.5%blunt · at its level
LAMBADA (OpenAI version), last-word accuracy lab55.1% ✎▁··▂▃▁▁▂▆█▅▃ beats 3/26very easy · avg 89%▇▇ 5.2%blunt · at its level
5 more tests
testresultvs the others in its rundifficultycountswhy
Social IQA lab39.5% ✎▁···▃▃█▁▁▁·▁ beats 1/26medium · avg 59%▇ 1.8%blunt · too hard for it
MMLU26.2% ✎▄▄·▄···▄▄·██ beats 1/8easy · avg 77%▇ 1.3%too hard for it
OpenBookQA lab21.4% ✎█·▁▁▃·▁▃▅▇▃▄ beats 4/26medium · avg 53%▇ 1.0%too hard for it
TruthfulQA MC2 lab40.0% ✎█·█·█▄···▄·█ beats 8/9medium · avg 61%▇ 0.8%blunt · at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)25.4% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 1/276medium · avg 73%▇ 0.6%too hard for it · noisy

$ where to run it

Other ways to run it

  1. Run it as published · needs ~31.74 GB · from huggingface.co/stabilityai (official)
    uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='stabilityai/stablelm-base-alpha-7b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"

Official weights and documentation: huggingface.co/stabilityai ↗

$ how the score was worked out · Stability AI — original Stabl… 77%, EvalPlus / EvalPerf published… 18%, InstructEval (DeCLaRe Lab) RE… 3% and 1 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Stability AI — original StableLM zero-shot evaluation outputs · lab-reported11▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 77%Apache-2.0
EvalPlus / EvalPerf published evaluation runs2▇▇▇▇▇ 18%Apache-2.0
InstructEval (DeCLaRe Lab) README results table2▇ 3%MIT / Apache-2.0
Open Portuguese LLM Leaderboard (archived)1▇ 1%Apache-2.0
prior · pull toward the middle▇ 1%
weighted average= 9.3

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json