whatmodel.sh / Luminous Supreme (70B)

Luminous Supreme (70B)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Luminous Supreme (70B) is a closed-weights language model from Aleph Alpha, released February 2023. Its published size is 70B parameters. Licence: proprietary.

Catalogue identity notes

Aleph Alpha API model luminous-supreme (70B per Aleph Alpha's 'Luminous Performance Benchmarks' post dated 20/02/2023, which says the models are 'available on the Completion Playground and API client'). 'released' is the date of that first-party post; the Luminous series was sold earlier (2022) but no first-party launch page was fetched. API-only, later deprecated. Comparison-only.

Model information sources

Release date ↗

62.3±12.6comparison only: no current run option verifiedfrom 10 benchmark results across 1 source

Score within the ARC Challenge comparison set only.

$ how it compares

Scored LLMs released closest to Luminous Supreme (70B), out of 1182 with a score.

#modelscorewhereRAMreleased
931LLaMA 65B Meta40±1comparison only2023-02
986LLaMA 33B Meta37±1comparison only2023-02
1057LLaMA 13B Meta32±1comparison only2023-02
1077LLaMA 7B Meta30±1comparison only2023-02
1118Pythia 12B EleutherAI25±1local~9.5GB2023-02

see all LLMs · find the ones that fit your machine

$ what it's good at

other tests10 not in a task · 87% of its score
testresultvs the others in its rundifficultycountswhy
OpenBookQA lab33.0% ✎▅▁▁▁▂▃▃█▃▁▃▁ beats 34/107very hard · avg 29%▇▇▇▇▇▇▇ 17%sharp · small run
COPA lab90.0% ✎▁·▁▁▁▅▅█▆▂▁▁ beats 53/67easy · avg 86%▇▇▇▇▇ 13%sharp · small run
TriviaQA lab37.7% ✎▁▁▂▂▄▄▃▆▅█▅▂ beats 32/80hard · avg 28%▇▇▇▇▇ 11%at its level · small run
BoolQ lab74.3% ✎▁····▁▂█▆▅▆▄ beats 63/103medium · avg 70%▇▇▇▇ 10%at its level · small run
ARC Challenge lab44.2% ✎▄▁▁▂▁▄█▇▃▂▃▂ beats 72/140hard · avg 40%▇▇▇ 8.5%at its level · small run
WinoGrande lab70.0% ✎▁▁▁▁▂▁▁▅▇█▃▁ beats 90/131hard · avg 67%▇▇▇ 7.6%at its level · small run
HellaSwag lab59.2% ✎▁▁▁▂▂▂▄▆█▇▄▂ beats 37/135medium · avg 56%▇▇ 5.7%at its level · small run
ARC Easy lab76.9% ✎▁▃▂▄▆█▅▃▁··▁ beats 85/100easy · avg 74%▇▇ 5.6%at its level · small run
2 more tests
testresultvs the others in its rundifficultycountswhy
PIQA lab79.9% ✎▁▁▁▂▂▃▅█▅▃▁▁ beats 92/123medium · avg 79%▇▇ 4.8%at its level · small run
LAMBADA (OpenAI version), last-word accuracy lab74.0% ✎▁·····▁▁▃▃██ beats 71/86easy · avg 73%▇▇ 4.5%small run

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Aleph Alpha's own Luminous be… 87%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Aleph Alpha's own Luminous benchmark results (Luminous performance benchmarks post) · lab-reported10Factual vendor-reported score (not licensed content)
prior · pull toward the middle
weighted average= 62.3

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json