whatmodel.sh / Alpaca 7B

Alpaca 7B

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Alpaca 7B is an open-weights language model from Stanford CRFM, released March 2023. Its published size is 7B parameters. Licence: cc-by-nc-4.0.

Catalogue identity notes

Stanford Alpaca 7B (LLaMA 7B fine-tuned on 52K self-instruct demonstrations; announced 2023-03-13). Official weights published only as a weight diff against LLaMA 7B (tatsu-lab/alpaca-7b-wdiff, created 2023-04-13, CC BY-NC 4.0). Community re-trainings labelled 'Alpaca-7B' are not this checkpoint.

Model information sources

Release date ↗

30.6±1.4comparison only: no current run option verifiedfrom 10 benchmark results across 5 sources

Score within the LMArena (text) comparison set only.

$ how it compares

The LLMs ranked either side of Alpaca 7B, out of 877 with a score, then more from Stanford CRFM.

#modelscorewhereRAMreleased
775Gemma 1.1 2B IT Google31±1local~3.2GB2024-04
776Falcon 7B (base) TII31±1local~5.9GB2023-04
777Alpaca 7B Stanford CRFM31±1comparison only2023-03
778Baichuan2-7B-Base Baichuan30±1local~17GB2023-08
779NexusRaven V2 13B Nexusflow30±3local~9.3GB2023-12
more from Stanford CRFM
793Alpaca 13B Stanford CRFM29±2comparison only2023-03

see all LLMs · find the ones that fit your machine

$ what it's good at

math▁▂▃▆█▇▅▇▃▁▁▁ #466 of 523 * · 28% of its score · 1 test

Beats Yi 34B Chat, Llama 3 8B Instruct, Command R (03-2024) here, though they rank higher overall. Trails GPT-Neo 125M, GPT-2 XL (1.5B), GPT-Neo 1.3B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
GSM8K8.6% ✎█··█······▄▄ beats 2/5easy · avg 67%▇▇▇▇▇▇▇▇▇▇▇ 28%sharp
other tests8 not in a task · 71% of its score
testresultvs the others in its rundifficultycountswhy
MMLU +142.9% ✎▄···▄▄·█···▄ beats 2/5easy · avg 79%▇▇▇▇▇▇▇▇▇▇ 26%several runs
ARC Challenge52.6% ✎▄·▄···█·▄··▄ beats 2/5easy · avg 86%▇▇▇▇▇▇▇ 18%sharp · at its level
HellaSwag76.9% ✎▄·▄█··▄····▄ beats 2/5very easy · avg 92%▇▇▇▇ 9.1%at its level
Static HHH alignment (BIG-bench hhh_alignment)62.9% ✎█·█········█ beats 2/2medium · avg 77%▇▇ 5.7%small run
SciTab 3-class claim verification (zero-shot, macro-F1)27.6% ✎█···▄▄·▄▄▄·▄ beats 1/7medium · avg 51%▇▇ 4.1%blunt · at its level
API-Bank evaluation system, zero-shot API-call correctness (Total)15.2% ✎▄······█··▄▄ beats 1/4medium · avg 50%▇ 3.7%typical
TruthfulQA MC119.5% ✎█·····█····█ beats 1/2medium · avg 50%▇ 3.5%small run
TruthfulQA MC239.6% ✎▃···█·▃····▃ beats 1/5medium · avg 56%▇ 0.7%blunt · at its level

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · Enhancing Chat Language Model… 47%, INFORM : Information eNtropy… 28%, Aligning Large Language Model… 16% and 2 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations — authors' own runs4CC BY 4.0
INFORM : Information eNtropy based multi-step reasoning FOR large language Models — authors' own runs1CC BY 4.0
Aligning Large Language Models through Synthetic Feedback — authors' own runs3CC BY 4.0
SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables — authors' own runs1CC BY 4.0
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs — authors' own runs1CC BY 4.0
prior · pull toward the middle
weighted average= 30.6

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json