whatmodel.sh / Vicuna 33B v1.3

Vicuna 33B v1.3

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Vicuna 33B v1.3 is an open-weights language model from LMSYS, released June 2023. Its published size is 33B parameters and a 2k-token context window. Licence: non-commercial.

Catalogue identity notes

Arena entry 'vicuna-33b' (text). LMArena's leaderboard table (arena-leaderboard space @6f2eeeb) labels 'vicuna-33b' as 'Vicuna-33B' (LMSYS, Non-commercial), linking https://huggingface.co/lmsys/vicuna-33b-v1.3. Maker repo lmsys/vicuna-33b-v1.3 @ef8d6be: licence tag none tagged. Parameters: nominal size from the model name. Released 2023-06: month the maker's Hugging Face repo was created (createdAt 2023-06-21). Only one Vicuna 33B checkpoint (v1.3) was released; card licence 'Non-commercial license'. The ollama vicuna 33b tags are the only 33B build in that library.

39.1±1.1#543 of 719 LLMsfrom 10 benchmark results across 8 sources

$ how it compares

The LLMs ranked either side of Vicuna 33B v1.3, out of 828 with a score, then more from LMSYS.

#modelscorewhereRAMreleased
633Phi-3-mini-128k-instruct (Apr 2024) Microsoft39±2local~9.5GB2024-04
634Mistral 7B Instruct v0.3 Mistral39±1local~5.8GB2024-05
635Vicuna 33B v1.3 LMSYS39±1local~23GB2023-06
636DeepSeek-Coder 6.7B Base DeepSeek39±1local~5.8GB2023-10
637Starling-LM 7B alpha UC Berkeley (NEST)39±1local~6.1GB2023-11
more from LMSYS
679Vicuna 13B v1.5 LMSYS36±2local~9.8GB2023-08
716Vicuna 7B v1.5 LMSYS33±1local~5.8GB2023-08
804FastChat-T5 3B LMSYS17±2local~8.5GB2023-04

see all LLMs · find the ones that fit your machine

$ what it's good at

chat & writing▁▁▁▃▅▄▅█▆▆▇▃ #331 of 393 · 77% of its score · 6 tests

Beats OpenThinker3 7B, Gemma 4 E2B, Code Llama 70B Instruct here, though they rank higher overall. Trails Granite 4.0 H 350M, LFM2 350M, Gemma 3 1B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AlpacaEval 2 length-controlled17.6% ✎▁▂▂▃▆▆█▆▄▄▁▁ beats 44/122medium · avg 39%▇▇▇▇▇▇▇▇▇▇▇ 27%typical
Arena-Hard (v0.1, GPT-4-1106 judge)8.6% ✎▂▁▃▅█▅▄▃▃▅▃▁ beats 4/60medium · avg 52%▇▇▇▇▇▇ 15%sharp
Polyrating rating (bias-aware re-fit of Chatbot Arena votes, mid-2024)1105.0 ✎▂▁▂▁▃▇█▆▇▄▂▂ beats 49/95avg 1201▇▇▇▇ 11%blunt
LMArena (text)1172.9 ✎▁▁▁▂▃▃▃▆▆▇█▃ beats 52/379avg 1268▇▇▇▇ 11%blunt
MT-Bench (GPT-4 single-answer grading, LMSYS runs) lab7.1200 MT-Bench score (1-10) ✎▆▂▂▆▆▄██▃▄▄▆ beats 21/28avg 8.819 MT-Bench score (1-10)▇▇▇▇ 10%sharp · noisy
AlpacaEval 1.0 win rate (vs text-davinci-003, GPT-4 judge)89.0% ✎▂▂·▆·▇▃▆█▆▄▂ beats 28/45very easy · avg 96%▇ 3.6%sharp · noisy
other tests4 not in a task · 23% of its score
testresultvs the others in its rundifficultycountswhy
Factual knowledge recall, Premium2k exact match (zero-shot)34.3% ✎▅····▃▅·██·█ beats 11/13medium · avg 59%▇▇▇▇ 9.5%at its level
Factual knowledge recall, Premium2k exact match (10-shot)44.0% ✎▃▅▃··▅▃·█▃▃▅ beats 10/13easy · avg 67%▇▇ 6.0%at its level
EQ-Bench v1 (60 questions, paper run)36.5% ✎▂··▄·▂▆▄▂█·▂ beats 4/14medium · avg 57%▇▇ 4.9%at its level
Open PT-LLM Leaderboard average (9 Portuguese tasks)65.3% ✎▃▁▁▁▁▁▁▁▂█▅▁ beats 115/276medium · avg 73%▇ 2.3%at its level · noisy

$ where to run it

Ollama · recommended

version

  1. Install Ollama

    download it (macOS / Windows) ↗

    Linux
    curl -fsSL https://ollama.com/install.sh | sh
  2. Run Vicuna 33B v1.3 · needs ~20.3 GB · from ollama.com
    chat in the terminal
    ollama run vicuna:33b-q4_K_M

Official weights and documentation: huggingface.co/lmsys ↗

$ how the score was worked out · AlpacaEval 1.0 and 2.0 leader… 30%, Towards a Holistic Evaluation… 15%, Arena-Hard-Auto v0.1 leaderbo… 15% and 5 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
AlpacaEval 1.0 and 2.0 leaderboards (tatsu-lab/alpaca_eval repository)2▇▇▇▇▇▇▇▇▇ 30%Apache-2.0
Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall (arXiv:2404.16164v1)2▇▇▇▇▇ 15%CC-BY-4.0
Arena-Hard-Auto v0.1 leaderboard (repository README, pre-v2)1▇▇▇▇▇ 15%Apache-2.0
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (arXiv:2409.00696v3)1▇▇▇ 11%CC-BY-4.0
LMArena leaderboard dataset1▇▇▇ 11%CC BY 4.0
LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table · lab-reported1▇▇▇ 10%Apache-2.0
EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models — authors' own runs1▇ 5%CC BY 4.0
Open Portuguese LLM Leaderboard (archived)1▇ 2%Apache-2.0
prior · pull toward the middle▇ 1%
weighted average= 39.1

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json