whatmodel.sh / decision / Bespoke-Nimble-9B (JevBench mini)
Bespoke-Nimble-9B (JevBench mini)
↗ share
+ contribute
Bespoke-Nimble-9B (JevBench mini) is a source-available decision implementation from Bespoke Labs.
Catalogue identity notes
JevBench 0.1.0 mini tested configuration bespoke-nimble-9b: Decoder, answer-letter logits; base Qwen3.5-9B; readout LoRA scoring one-letter answer tokens; whole schema in every prompt. JevBench shim (Nimble's own Modal server does not run locally); same request mapping as Nimble's compiler.py, PyTorch reference runner, current checkpoint Separate identity because the report does not establish equivalence to the newer Decision Index checkpoint or serving recipe. No latency imported.
$ how it compares
The decision models ranked either side of Bespoke-Nimble-9B (JevBench mini), out of 357 with a score, then more from Bespoke Labs.
| # | model | score | where | ||
|---|---|---|---|---|---|
| 184 | decider-2b (DecisionBench fa996cea) | 53±2 | model files | ||
| 185 |  cua-s1-4b-0.2 (DecisionBench 16818868) | 53±2 | model files | ||
| 186 | Bespoke-Nimble-9B (JevBench mini) | 53±2 | source code | ||
| 187 |  Hopper 12B (trained) | 52±1 | model files | ||
| 188 |  Jebadiah 9B v2 (JevBench mini) | 52±2 | source code | ||
| more from Bespoke Labs | |||||
| 15 | Bespoke Nimble v4 Max (27B) | 70±1 | model files | ||
| 135 | Bespoke Nimble 9B v3 | 58±1 | local | ||
| 172 | Bespoke-Nimble-9B (DecisionBench 594dfdcf) | 54±2 | model files | ||
see all decision models · find the ones that fit your machine
$ what it's good at
consistency▃·▂▂·▅▃·▃▅▃█ #6 of 21 · 68% of its score · 5 tests
| test | result | vs the others in its run | counts |
|---|---|---|---|
| JevBench mini 0.1.0 · Probability measure coherence (pass rate) | 50.5% ✎ | ▄·▁·▃▃▁·▃▄▁█ beats 17/20 | 51% |
| JevBench mini 0.1.0 · Representation consistency (pass rate) | 76.4% ✎ | ▃·▂·▂▂▂▃▅▅▃█ beats 9/20 | 13% |
| JevBench mini 0.1.0 · Logical coherence (pass rate) | 87.1% ✎ | ▃·▃·█▅▅██▃█▅ beats 19/20 | 2.3% |
| JevBench mini 0.1.0 · Choice-set coherence (pass rate) | 63.9% ✎ | ▁▃▄█▄▄·····▄ beats 3/20 | 1.2% |
| JevBench mini 0.1.0 · Batch independence (pass rate) | 68.3% ✎ | ▃··········█ beats 0/20 | 0.0% |
other tests1 not in a task · 30% of its score
| test | result | vs the others in its run | counts |
|---|---|---|---|
| JevBench mini 0.1.0 · unchanged-answer accuracy | 89.1% ✎ | ▁·▂·▂···▂▂▁█ beats 18/20 | 30% |
$ where to run it
Implementation and setup instructions ↗
Published implementation and setup documentation verified. This entry describes a decision readout or evaluated configuration, not necessarily separate trained weights. Follow the referenced version and model settings; hardware compatibility has not been verified. Source · checked 2026-10-31.
Model repository and documentation: github.com/bespokelabsai ↗
$ how the score was worked out · JevBench 0.1.0 mini evaluatio… 98%
Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works
| source | results | share of the score |
|---|---|---|
 JevBench 0.1.0 mini evaluation reports | 6 | 98% |
| prior · pull toward the middle | 2% | |
| weighted average | = 52.6 |
$ improve this page
Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.
data updated · sources and licences · json