whatmodel.sh / Tk-Instruct 11B (def)
Tk-Instruct 11B (def)
↗ share
+ contribute
Tk-Instruct 11B (def) is an open-weights language model from AllenAI / UW, released May 2022. Its published size is 11B parameters. Licence: apache-2.0.
Catalogue identity notes
Tk-Instruct 11B trained on Super-NaturalInstructions with task definitions only (Wang et al., EMNLP 2022); repo allenai/tk-instruct-11b-def created 2022-05-05.
Model information sources
Score within the LMArena (text) comparison set only.
$ how it compares
The LLMs ranked either side of Tk-Instruct 11B (def), out of 877 with a score.
| # | model | score | where |
|---|---|---|---|
| 808 |  BLOOMZ 3B | 27±2 | comparison only |
| 809 |  Qwen1.5 1.8B Chat | 26±2 | local |
| 810 | Tk-Instruct 11B (def) | 26±4 | comparison only |
| 811 |  Falcon3 1B Instruct | 26±4 | local |
| 812 |  OPT-13B (13 billion parameters) | 26±1 | local |
$ what it's good at
other tests3 not in a task · 92% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| BIG-Bench Hard | 11.8% ✎ | ▃▁▁▁▂▄▅▆█▃▃▁ beats 0/155 | 45% |
| Super-NaturalInstructions English test (definition only, zero-shot), ROUGE-L | 45.6% ✎ | ▄·▄······▄▄█ beats 5/5 | 36% |
| LMentry accuracy (zero-shot) | 48.0% ✎ | ▂▆·▄▄█▂·▂··▂ beats 9/14 | 12% |
$ where to run it
No verified API or download recorded yet · add one
Official weights and documentation: huggingface.co/allenai ↗
$ how the score was worked out · Unnatural Instructions: Tunin… 81%, LMentry: A Language Model Ben… 12%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor — authors' own runs | 2 | 81% |
| LMentry: A Language Model Benchmark of Elementary Language Tasks — authors' own runs | 1 | 12% |
| prior · pull toward the middle | 8% | |
| weighted average | = 26.4 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json