whatmodel.sh / Vicuna 7B v1.1
Vicuna 7B v1.1
↗ share
+ contribute
Vicuna 7B v1.1 is an open-weights language model from LMSYS, released April 2023. Its published size is 7B parameters.
Catalogue identity notes
LMSYS Vicuna 7B v1.1 delta weights lmsys/vicuna-7b-delta-v1.1 (created 2023-04-12). Not v0 or v1.3/v1.5.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Vicuna 7B v1.1, out of 877 with a score, then more from LMSYS.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 719 |  StarCoder2 7B | 37±1 | local | ||
| 720 |  DeepSeek LLM 7B Chat | 37±2 | local | ||
| 721 | Vicuna 7B v1.1 | 37±2 | comparison only | ||
| 722 | StarCoder 15.5B | 37±1 | local | ||
| 723 |  Eurus-2-7B-SFT | 37±3 | local | ||
| more from LMSYS | |||||
| 638 | Vicuna 13B v1.1 | 42±2 | comparison only | ||
| 643 | Vicuna 33B v1.3 | 41±1 | local | ||
| 693 | Vicuna 13B v1.5 | 38±1 | local | ||
$ what it's good at
other tests4 not in a task · 97% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| CommonsenseQA | 44.9% ✎ | █·····▂▂▃▂▄▄ beats 20/38 | 41% |
| ARC Challenge | 40.5% ✎ | ▄▁▁▂▁▄█▇▃▂▃▂ beats 51/140 | 29% |
| OpenBookQA | 37.2% ✎ | ▅▁▁▁▂▂▃█▃▁▃▁ beats 40/107 | 16% |
| This is not a Dataset (negation), zero-shot accuracy | 52.4% ✎ | █▁····▃▁·▁·▁ beats 9/14 | 11% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Search Augmented Instruction… 86%, This is not a Dataset: A Larg… 11%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Search Augmented Instruction Learning — authors' own runs | 3 | 86% |
| This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models — authors' own runs | 1 | 11% |
| prior · pull toward the middle | 3% | |
| weighted average | = 36.6 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json