whatmodel.sh / Vicuna 13B v1.1
Vicuna 13B v1.1
↗ share
+ contribute
Vicuna 13B v1.1 is an open-weights language model from LMSYS, released April 2023. Its published size is 13B parameters.
Catalogue identity notes
LMSYS Vicuna 13B v1.1 (LLaMA 13B fine-tuned on ShareGPT), released as delta weights lmsys/vicuna-13b-delta-v1.1 (created 2023-04-12). Not v0 (March 2023) or v1.3/v1.5.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Vicuna 13B v1.1, out of 877 with a score, then more from LMSYS.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 636 |  North Micro Vision Instruct (2.4B) | 42±2 | local | ||
| 637 |  Deita 7B v1.0 | 42±2 | local | ||
| 638 | Vicuna 13B v1.1 | 42±2 | comparison only | ||
| 639 |  OpenHermes 2.5 Mistral 7B | 42±1 | local | ||
| 640 |  Code Llama 34B Instruct | 42±1 | local | ||
| more from LMSYS | |||||
| 643 | Vicuna 33B v1.3 | 41±1 | local | ||
| 693 | Vicuna 13B v1.5 | 38±1 | local | ||
| 715 | Vicuna 13B v1.3 | 37±2 | local | ||
$ what it's good at
other tests5 not in a task · 98% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| CommonsenseQA | 50.6% ✎ | █·····▂▂▃▂▄▄ beats 23/38 | 35% |
| ARC Challenge | 53.2% ✎ | ▄▁▁▂▁▄█▇▃▂▃▂ beats 94/140 | 25% |
| This is not a Dataset (negation), zero-shot accuracy | 57.8% ✎ | █▁····▃▁·▃·▁ beats 13/14 | 15% |
| OpenBookQA | 49.0% ✎ | ▅▁▁▁▂▂▃█▃▁▃▁ beats 67/107 | 15% |
| SummEdits factual-consistency detection (zero-shot, balanced accuracy) | 56.1% ✎ | ▄····▄·█···▄ beats 1/4 | 8.2% |
$ where to run it
No verified API or download recorded yet · add one
$ how the score was worked out · Search Augmented Instruction… 75%, This is not a Dataset: A Larg… 15%, SummEdits: Measuring LLM Abil… 8%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Search Augmented Instruction Learning — authors' own runs | 3 | 75% |
| This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models — authors' own runs | 1 | 15% |
| SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization — authors' own runs | 1 | 8% |
| prior · pull toward the middle | 2% | |
| weighted average | = 41.8 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json