Jev 1.13

+ contribute

Jev 1.13 is a closed-weights decision model from TypeSafe, released 15 September 2026. Licence: proprietary.

Catalogue identity notes

Exact Decision Index tested engine jev-1.13.0; reported model Jev 1.13. Type: hosted API; technique: None. Comparison entry; no current run command, API price or speed inferred from the benchmark.

Model information sources

Release date ↗

65.9±0.6#49 of 355 decision modelsfrom 180 benchmark results across 17 sources

$ how it compares

The decision models ranked either side of Jev 1.13, out of 357 with a score, then more from TypeSafe.

#	model	score	where	released
47	Standard One 35B SH (v1) Standard Thinking	66±1	model files
48	allec-oavg (Gemma 4 31B) plantwaterer	66±1	model files
49	Jev 1.13 TypeSafe	66±1	api $0.042/M input tokens	2026-09-15
50	Span-02 (Gemma 4 26B-A4B) Respan	66±1	api price unknown
51	RSI-Jev v6.1 VL 27B (effort auto) shgao	66±1	model files	2026-10-08
	more from TypeSafe
63	Jev 1.14 TypeSafe	64±1	api $0.042/M input tokens	2026-10-14

see all decision models · find the ones that fit your machine

$ what it's good at

tools & automation▁▃▁▁▁▂▃▄▆██▆ #103 of 250 · 2.7% of its score · 11 tests

Trails OneJev1.6-0.8B, TokenRhythm NeoHorse-Jev-4B, mmastrac diffgemma (decision readout) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
AgentDecisionBench: L1 BFCL v4 + When2Call (3,991 items)	86.4% ✎	█·····▄·▄▄·▄ beats 5/5	easy · avg 75%	▇ 0.5%	typical
AgentDecisionBench: L2 DecisionGym common states (200 tasks)	91.5% ✎	██···█·██··█ beats 5/5	medium · avg 57%	▇ 0.4%	sharp · noisy
Decision Index 0.4.1: API-Bank (accuracy) submitted	88.2% ✎	▁▁▁▁▁▁▁▂▃▂█▃ beats 221/236	easy · avg 70%	▇ 0.4%	sharp · noisy
DecideBench v1: agent routing	98.0% ✎	▄·····▂▄·█·▄ beats 6/10	very easy · avg 93%	▇ 0.3%	sharp · maxed out · noisy
DecideBench v1: action review	100.0% ✎	▃▃·▃·█···█·▅ beats 10/10	easy · avg 87%	▇ 0.3%	sharp · maxed out · noisy
Decision Index 0.4.1: ToolRet (nDCG@10) submitted	65.3% ✎	▁▁▁▁▁·▁▁▁▄█▁ beats 143/246	medium · avg 60%	▇ 0.2%	blunt · at its level
Decision Index 0.4.1: Home appliance simulator (case exact accuracy) submitted	52.3% ✎	█▁▁▃▄▄▆▄▁▂▁▃ beats 134/243	medium · avg 37%	▇ 0.2%	sharp · noisy
Decision Index 0.4.1: BFCL (case exact accuracy) submitted	95.8% ✎	▁▂▁▁▁▂▂▄▅▆█▁ beats 144/246	very easy · avg 92%	▇ 0.2%	sharp · maxed out · noisy

3 more tests

test	result	vs the others in its run	difficulty	counts	why
Decision Index 0.4.1: When2Call (accuracy) submitted	81.0% ✎	▁▁▁▂▃▅▅▆█▇█▃ beats 148/246	easy · avg 72%	▇ 0.2%	noisy
AgentDecisionBench: L2 DecisionGym-Shift (100 tasks)	91.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Decision Index 0.4.1: RouterBench (selected quality (quality objective)) submitted	79.9% ✎	no other model yet		—	failed run: Publisher excludes this protocol because its prompt reveals GPT-4 as the best route. Evidence: https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/c130faf65f82e81d3bdefb01e758c712dc5c5199/data/index.json

retrieval & classification▁▁▁·▁▁▁▁▃▃█▅ #93 of 258 · 8.7% of its score · 21 tests

Beats AutoTrust-JEV-27B v1.1, Decider 31B (NVFP4) here, though they rank higher overall. Trails Jev-Style 2B v4, Sieve 2B v3, StartLux-Decision v1.1 0.8B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
DecisionPivot v1.1 · paws (accuracy)	85.8% ✎	█·██·······█ beats 2/3	easy · avg 68%	▇ 1.2%	sharp
DecisionPivot v1.1 · nevir (pair_accuracy)	67.8% ✎	█··█··█····█ beats 1/3	medium · avg 49%	▇ 1.0%	sharp · at its level
System One Bench: MIRACL pairwise (Arabic)	90.5% ✎	▂···▁··▃▄▂█▇ beats 23/29	easy · avg 84%	▇ 0.8%	typical
System One Bench: MASSIVE intent (Arabic)	84.0% ✎	▄·▁·▁▃▃▃█▄▇▇ beats 12/29	easy · avg 76%	▇ 0.7%	typical
LeanMCP · scifact rank noul	75.0% ✎	▂····▂···▃▅█ beats 10/11	medium · avg 62%	▇ 0.6%	typical
Decision Index 0.4.1: BRIGHT (nDCG@10) submitted	47.5% ✎	▁▁▁▁▁▁▁▂▂▄▇█ beats 192/246	medium · avg 38%	▇ 0.6%	blunt · at its level
LeanMCP · nfcorpus rank noul	37.8% ✎	▃▃·····▃▃▅██ beats 9/11	hard · avg 29%	▇ 0.6%	blunt · at its level
Decision Index 0.4.1: Amazon ESCI (macro-F1) submitted	55.2% ✎	▁▁▁▁▁▁▂▂▃▃█▅ beats 155/246	medium · avg 47%	▇ 0.6%	at its level

13 more tests

test	result	vs the others in its run	difficulty	counts	why
System One Bench: MIRACL rerank (Arabic)	86.5% ✎	▁▂·▁▁·▂·▄▄█▇ beats 14/29	easy · avg 79%	▇ 0.5%	typical
DecisionPivot v1.1 · entity match amazon google (auroc)	0.9610 auroc ✎	█···█·█····█ beats 2/3	avg 0.290 auroc	▇ 0.5%	typical
DecisionPivot v1.1 · entity match abt buy (auroc)	0.9900 auroc ✎	█······█··██ beats 1/3	avg 0.320 auroc	▇ 0.5%	typical
System One Bench: MASSIVE scenario (Arabic)	63.0% ✎	▃▁·▄▃▂█▂▁▁·▃ beats 10/29	medium · avg 60%	▇ 0.3%	at its level
GEO Citation · listwise Spearman (October 2026)	0.5752 spearman-rho ✎	▃···▃······█ beats 3/4	avg 0.256 spearman-rho	▇ 0.2%	blunt
Decision Index 0.4.1: HoVer (accuracy) submitted	72.9% ✎	▆▆▇▅▄▄▃▇▄█▆▄ beats 111/246	easy · avg 74%	▇ 0.1%	blunt
Decision Index 0.4.1: PhishNChips phishing decisions (accuracy) submitted	62.5% ✎	█▅▅▄▃▂▁▁▁▁·▁ beats 98/245	easy · avg 70%	▇ 0.1%	noisy
DecideBench v1: support intent	100.0% ✎	▃····▂▂·▃··█ beats 10/10	very easy · avg 95%	▇ 0.1%	sharp · maxed out · noisy
System One Bench: AG News (English)	88.5% ✎	▁···▅▁▅▃▆█▂▅ beats 11/29	easy · avg 88%	▇ 0.1%	blunt
Decision Index 0.4.1: BANKING77 (macro-F1) submitted	79.7% ✎	▁·▁▁▁▁▁▁▂█▇█ beats 94/238	easy · avg 79%	▇ 0.1%	noisy
Decision Index 0.4.1: CLINC150+OOS (macro-F1) submitted	89.3% ✎	▁▁▁▁▁▁▁▂▃▆█▄ beats 113/238	easy · avg 83%	▇ 0.1%	sharp · maxed out · noisy
System One Bench: Banking77 (English)	79.0% ✎	█▅▆█▅▃▃▃···▂ beats 2/26	very easy · avg 87%	▇ 0.0%	blunt · noisy
Decision Index 0.4.1: SGD/SGD-X (macro-F1) submitted	43.0% ✎	no other model yet		—	failed run: Publisher reports a builder bug exposing dialogue turns after the classified turn; awaiting corrected rerun. Evidence: https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/c130faf65f82e81d3bdefb01e758c712dc5c5199/data/index.json

language understanding▂▂▂▃▂▅▆▃█▆▄▃ #67 of 256 · 7.9% of its score · 21 tests

Beats Turion Jev 31B, AJev lm6q (Gemma 4 31B), oenogarum (Gemma 4 31B) here, though they rank higher overall. Trails Sieve 4B, Decider 35B-A3B, Sieve 9B v2 here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Decision Index 0.4.1: ANLI (macro-F1) submitted	74.8% ✎	▁▂▃▅▅█▂····▁ beats 194/246	medium · avg 60%	▇ 1.1%	typical
Decision Index 0.4.1: NLI4CT (macro-F1) submitted	84.1% ✎	▁▁▁▂▂▃▃▄▇▇█▄ beats 201/246	easy · avg 74%	▇ 1.1%	typical
DecideBench v1: review sentiment	98.0% ✎	▂··▂···▂▂█▄▃ beats 10/10	easy · avg 85%	▇ 0.6%	sharp · maxed out
Decision Index 0.4.1: HellaSwag (accuracy) submitted	94.5% ✎	▂▄▂▂▄▇▃▇█▅▃▁ beats 187/246	very easy · avg 85%	▇ 0.6%	sharp · maxed out
System One Bench: BoolQ (English)	89.5% ✎	▁▁▁▃·▁▃▃▇▇█▅ beats 14/29	easy · avg 86%	▇ 0.6%	typical
Decision Index 0.4.1: iSarcasmEval (Sarcasm F1 · track A, English) submitted	50.5% ✎	▁·▁·▁▁▂▄▅█▇▄ beats 134/246	medium · avg 44%	▇ 0.6%	at its level
System One Bench: OSACT4 offensive (Arabic)	90.0% ✎	▁······▁▃█▇▄ beats 15/29	easy · avg 84%	▇ 0.5%	sharp
DecideBench v1: content moderation	98.0% ✎	▂··▂···▄▄▂·█ beats 9/10	easy · avg 83%	▇ 0.4%	sharp · maxed out · noisy

13 more tests

test	result	vs the others in its run	difficulty	counts	why
Decision Index 0.4.1: ACOS (per-review F1) submitted	29.5% ✎	▁··▂▃▃▅▇█▇▃▃ beats 163/246	hard · avg 22%	▇ 0.4%	at its level · noisy
System One Bench: SST-5 (English)	62.0% ✎	▃·▁▁▃·▇▄▄▄█▅ beats 28/29	medium · avg 51%	▇ 0.3%	blunt · at its level
System One Bench: SST-2 (English)	94.5% ✎	▁·▃▁▂▂▃▅█·▂▁ beats 14/29	easy · avg 92%	▇ 0.3%	typical
Decision Index 0.4.1: VAST (macro-F1) submitted	64.6% ✎	▁▁▁▂▅▄▄▂▄▆█▄ beats 119/246	medium · avg 62%	▇ 0.2%	at its level
Decision Index 0.4.1: ContractNLI (macro-F1) submitted	71.7% ✎	▁▁▁▁▁▁▁▂▂▇▇█ beats 84/246	easy · avg 71%	▇ 0.2%	noisy
DecideBench v1: claim support	96.0% ✎	▄··▄·▄·▄▄███ beats 7/10	easy · avg 82%	▇ 0.2%	sharp · maxed out · noisy
System One Bench: XNLI Arabic (Arabic)	69.3% ✎	▄▁▃▁▄▇█▅▃▁·▃ beats 9/29	medium · avg 68%	▇ 0.2%	at its level
Decision Index 0.4.1: RAGTruth (F1 on hallucinated class) submitted	76.5% ✎	▁▁▁▁▁▁▁▁▁▃▆█ beats 141/246	easy · avg 66%	▇ 0.2%	noisy
System One Bench: XNLI French (French)	76.7% ✎	▂▅·▁▁▅▅█▃▁·▃ beats 9/29	medium · avg 76%	▇ 0.1%	blunt · at its level
System One Bench: Tweet sentiment (Arabic)	76.0% ✎	▁······▁▁▄█▄ beats 28/29	medium · avg 62%	▇ 0.1%	blunt · at its level
Decision Index 0.4.1: FinEntity (macro-F1) submitted	87.0% ✎	▁···▁▁▁▁▃▆█▃ beats 111/246	easy · avg 84%	▇ 0.1%	maxed out · noisy
System One Bench: XNLI English (English)	86.0% ✎	▃·▁·▃▇█▄█▃▃▁ beats 12/29	easy · avg 85%	▇ 0.0%	blunt · noisy
System One Bench: DAIR Emotion (English)	56.0% ✎	██▃·····▁··▁ beats 10/29	medium · avg 59%	▇ 0.0%	blunt · at its level

arts & human taste▁▇▄▃▅▅█▅▄▂▁▁ #100 of 247 · 2.1% of its score · 6 tests

Trails vLLM-SR vela3-fast-vision (Vela 3.0, 0.25B), Crivo-400M, Amber 0.4B here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Decision Index 0.4.1: New Yorker (accuracy) submitted	70.1% ✎	▁▁▂▃▃▃▇▅█▅▃▁ beats 151/246	medium · avg 61%	▇ 0.8%	typical
Decision Index 0.4.1: Humicroedit (accuracy) submitted	61.9% ✎	▁·▁▃▂▂▅█▄▂▁▁ beats 145/246	medium · avg 60%	▇ 0.5%	blunt · at its level
Decision Index 0.4.1: POP909-CL (accuracy) submitted	18.1% ✎	▄▅▅▃▆▅▃▅▂▅▃█ beats 126/237	hard · avg 21%	▇ 0.3%	sharp · at its level · noisy
Decision Index 0.4.1: cfcolor (accuracy) submitted	64.7% ✎	▁········▁▇█ beats 172/246	medium · avg 60%	▇ 0.3%	blunt · at its level
Decision Index 0.4.1: BPoMP (accuracy) submitted	90.6% ✎	▁█▄▄▅█▆▅▄▁▁▁ beats 148/246	easy · avg 83%	▇ 0.2%	sharp · maxed out · noisy
Decision Index 0.4.1: Habermas Machine (accuracy) submitted	45.9% ✎	▁▂▄█▇▅▁▁▄▄▁▁ beats 147/246	medium · avg 47%	▇ 0.1%	blunt · at its level

knowledge & reasoning▁▂▂▂▂▃▃▅▅▆█▃ #1 of 263 · 28% of its score · 44 tests

Beats Jeffan-31b-1710, baselabs mammoth-1 (27B), Sieve 31B v3 here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Etsabary · 1342 text decisions	76.5% ✎	▁··▂·▃▅▁▃▂▂█ beats 13/24	medium · avg 47%	▇ 3.7%	sharp
Decision Index 0.4.1: MMLU-Pro (accuracy) submitted	82.7% ✎	▄▄▃▃▂▄▅▇██▅▃ beats 246/246	medium · avg 46%	▇ 2.3%	sharp · at its level
Hallucination judging · Context inconsistency	82.5% ✎	▄·▄···▄····█ beats 3/4	medium · avg 49%	▇ 1.3%	sharp
Hallucination judging · Instruction inconsistency	61.8% ✎	▄····█·▄···▄ beats 1/4	medium · avg 38%	▇ 1.3%	sharp · at its level
LeanMCP · scienceqa text	95.7% ✎	▄··▄█▄█▄·█▄▄ beats 9/11	very easy · avg 88%	▇ 1.2%	sharp · maxed out
LeanMCP · medqa usmle	87.3% ✎	▄▄▄·▄▄▄▄▄·██ beats 8/11	easy · avg 72%	▇ 1.2%	typical
Decision Index 0.4.1: BBH (accuracy) submitted	92.9% ✎	▄▆▄▄█▇▇▄▃▁▃▁ beats 240/246	easy · avg 66%	▇ 1.2%	typical
Decision Index 0.4.1: MMLU (accuracy) submitted	91.7% ✎	▁▄▄▃▃▂▅▇▅█▇▃ beats 244/246	easy · avg 70%	▇ 1.2%	sharp

36 more tests

test	result	vs the others in its run	difficulty	counts	why
Hallucination judging · Logical inconsistency	69.0% ✎	█·█···█·█··█ beats 3/4	medium · avg 43%	▇ 1.1%	sharp · at its level
CONFLICTS · source conflict types	61.8% ✎	▃▃··▅▃···▃▃█ beats 3/9	medium · avg 45%	▇ 1.0%	at its level
DecisionPivot v1.1 · folio validation (accuracy)	82.8% ✎	█·········▄▄ beats 2/3	medium · avg 56%	▇ 1.0%	sharp
DecisionPivot v1.1 · primevul pair (pair_accuracy)	4.4% ✎	▄·········█▄ beats 1/3	very hard · avg 1.2%	▇ 1.0%	sharp · too hard for it
MedQA · Freyer et al. (first-valid accuracy)	88.3% ✎	█··▃▃·▃▃▃·▃█ beats 9/11	easy · avg 72%	▇ 0.9%	typical
LeanMCP · pubmedqa	70.6% ✎	▂···▂·▂·▄▄▂█ beats 11/11	medium · avg 54%	▇ 0.9%	at its level
LeanMCP · medmcqa	78.1% ✎	▃·▃··▃█▃·▅▅▃ beats 11/11	medium · avg 63%	▇ 0.9%	typical
Hallucination judging · Factual contradiction	75.0% ✎	█···█··█··██ beats 2/4	medium · avg 55%	▇ 0.8%	typical
Decision Index 0.4.1: GPQA Diamond (accuracy) submitted	78.6% ✎	▂▆▅▅▆█▆▃▁··▃ beats 246/246	medium · avg 39%	▇ 0.8%	blunt · at its level
DecisionPivot v1.1 · cuad clause (accuracy)	81.9% ✎	▄·········█▄ beats 1/3	medium · avg 65%	▇ 0.8%	typical
Decision Index 0.4.1: GSM8K (accuracy (4- and 10-choice)) submitted	71.8% ✎	▁··▄█▆▆▆▅▅▃▁ beats 192/246	medium · avg 45%	▇ 0.8%	sharp · at its level
LeanMCP · mmlu pro	80.6% ✎	▄··▄▄████··▄ beats 11/11	medium · avg 54%	▇ 0.7%	at its level
Hallucination judging · Factual fabrication	80.7% ✎	█·······▄▄·▄ beats 3/4	medium · avg 59%	▇ 0.6%	typical
DecisionPivot v1.1 · legalbench mix (accuracy)	79.5% ✎	█····█···█·█ beats 3/3	medium · avg 62%	▇ 0.6%	typical
Decision Index 0.4.1: WinoGrande (accuracy) submitted	92.0% ✎	█▅█▅▄█▆▄▃▃▁▁ beats 210/246	easy · avg 78%	▇ 0.6%	typical
Decision Index 0.4.1: SATA-Bench (case exact accuracy) submitted	26.4% ✎	▁▁▁▁▁▁▁▂▅█▂▁ beats 113/246	hard · avg 22%	▇ 0.4%	at its level
Decision Index 0.4.1: ARC-Challenge (accuracy) submitted	97.8% ✎	▂▂▂▂▂▂▁▃▄▆█▆ beats 236/246	very easy · avg 90%	▇ 0.4%	sharp · maxed out
Decision Index 0.4.1: MuSR (accuracy) submitted	66.1% ✎	▁▁▂█▅▄▁▁▁▁▁▁ beats 189/246	medium · avg 60%	▇ 0.3%	blunt · at its level
Decision Index 0.4.1: CRUXEval (accuracy) submitted	73.0% ✎	██▄▃▃▄▁▁···▁ beats 176/246	medium · avg 58%	▇ 0.2%	at its level · noisy
Decision Index 0.4.1: CLadder (accuracy) submitted	72.6% ✎	▁███▆▅▃▃▁▁▁▁ beats 142/246	easy · avg 71%	▇ 0.2%	typical
Decision Index 0.4.1: ChessBench (accuracy) submitted	17.2% ✎	▄▇▄▄▄▆▄█▄▁▁▁ beats 135/246	hard · avg 16%	▇ 0.2%	blunt
Decision Index 0.4.1: ARC-Easy (accuracy) submitted	99.3% ✎	▁▁▂▁▂▂▂▂▃▆█▅ beats 239/246	very easy · avg 96%	▇ 0.2%	sharp · maxed out · noisy
Decision Index 0.4.1: HLE (accuracy) submitted	20.4% ✎	▁▁▂▃▆█▇▅▅▃▁▃ beats 246/246	very hard · avg 13%	▇ 0.0%	blunt
system1bench · boolq	92.6% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · xnli en	85.9% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
system1bench · xnli zh	73.6% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · turtlebench	74.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Hard-Decisions · ProofWriter CWA	89.3% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Hard-Decisions · ProofWriter OWA	83.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · ViNLI · Natural language inference	81.0% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · ViANLI · Natural language inference	53.5% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · ViMedNLI · Natural language inference	73.4% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · vnNLI · Natural language inference	85.9% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
JevResearch · MMLU-Pro native choice	82.7% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
JevResearch · GPQA Diamond native choice	76.5% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
JevResearch · HLE text multiple-choice	21.9% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with

vision▁·▃▂▃▃▇▄▄█▄▃ #3 of 79 · 7.1% of its score · 8 tests

Beats Fastino GLiDE 1.2, StartLux-Decision v1.1 27B, vLLM-SR d3-ultra (Decision 3.0, 27B) here, though they rank higher overall.

test	result	vs the others in its run	difficulty	counts	why
Etsabary · 200 image decisions	71.0% ✎	▂▂··▂▃▂·█▃▂▂ beats 8/14	medium · avg 45%	▇ 2.6%	sharp · at its level
System One Bench: Image object count	94.5% ✎	▃····▃█▃▅·█▃ beats 10/11	easy · avg 71%	▇ 1.0%	sharp
System One Bench: SST-2 screenshot	94.5% ✎	█···███·█·▄▄ beats 7/11	easy · avg 87%	▇ 0.9%	sharp
System One Bench: Visual yes/no questions	88.0% ✎	▂······▃█▃·▃ beats 9/11	medium · avg 80%	▇ 0.8%	typical
System One Bench: Arabic sentiment screenshot	59.0% ✎	▄··▄··▂···▆█ beats 6/11	very hard · avg 37%	▇ 0.8%	sharp · at its level · noisy
System One Bench: Oxford pets classification	83.5% ✎	███·█▄▄·▄··▄ beats 1/11	easy · avg 77%	▇ 0.7%	typical
Benchmark Heaven imagejevbench v0.3.0: intelligence (points)	59.9112 points ✎	▃·▃▃··▅█▅▅·▃ beats 12/12	avg 32.235 points	▇ 0.2%	blunt
Benchmark Heaven imagejevbench v0.3.0: calibration (points)	84.9654 points ✎	▃···▃·▅█▅▃▃▅ beats 12/12	avg 68.584 points	▇ 0.1%	blunt

workflows▁▁▁▂▁▂▂▆█▄▁▁ #9 of 114 · 41% of its score · 123 tests

Beats baselabs mammoth-1 (27B), Nace.AI Drex Max v1 (27B), Perplexity Decider v1.2 (27B) here, though they rank higher overall. Trails Microsoft-Decision-1, Mercury Decide (paid endpoint), Quyet-1.0-Large here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
DecisionBench v1 · action selection (all-request accuracy) submitted	65.3% ✎	▂▁▂▃▄▃▁▁▂█▇▄ beats 51/56	medium · avg 51%	▇ 2.6%	at its level
DecisionPivot v1.1 · mind2web element (accuracy)	80.6% ✎	█···█·█····█ beats 2/3	medium · avg 54%	▇ 1.5%	sharp
DecisionBench v1 · routing (all-request accuracy) submitted	77.0% ✎	▁▁▁▁▂▂▁▁▁▁▃█ beats 50/56	medium · avg 64%	▇ 1.4%	sharp
DecisionBench v1 · record classification (all-request accuracy) submitted	49.0% ✎	▃▂▁▂▂▂▄▁▃▆██ beats 50/56	medium · avg 39%	▇ 1.3%	at its level
DecisionPivot v1.1 · processbench (accuracy)	71.7% ✎	█·····█··█·█ beats 2/3	medium · avg 41%	▇ 1.3%	sharp · at its level
DecisionPivot v1.1 · judgebench (accuracy)	77.7% ✎	█······██··█ beats 2/3	medium · avg 52%	▇ 1.2%	sharp
DecisionPivot v1.1 · llmbar (accuracy)	82.5% ✎	██·····█···█ beats 2/3	medium · avg 58%	▇ 1.2%	sharp
DecisionBench v1 · navigation (all-request accuracy) submitted	72.0% ✎	▁▁·▃▁▁▁▁▁▁▃█ beats 43/56	medium · avg 58%	▇ 1.1%	sharp

115 more tests

test	result	vs the others in its run	difficulty	counts	why
LeanMCP · mcpbench server routing	95.5% ✎	█·███··▄···▄ beats 7/9	easy · avg 82%	▇ 1.1%	sharp · maxed out
DecisionPivot v1.1 · rewardbench2 (accuracy)	80.8% ✎	█·····█···██ beats 2/3	medium · avg 57%	▇ 1.1%	sharp
DecisionPivot v1.1 · squad2 answerable (accuracy)	83.9% ✎	███········█ beats 2/3	easy · avg 69%	▇ 1.0%	sharp
DecisionPivot v1.1 · browsesafe (accuracy)	80.1% ✎	▄·······▄··█ beats 2/3	medium · avg 56%	▇ 1.0%	sharp
DecisionPivot v1.1 · rewardbench (accuracy)	96.4% ✎	█······█·█·█ beats 3/3	easy · avg 83%	▇ 1.0%	sharp · maxed out
LeanMCP · mcpmark server routing	51.5% ✎	▂··▂··▄█▃·▂▄ beats 8/11	medium · avg 36%	▇ 0.9%	at its level
Support routing · DecisionModelHub	92.0% ✎	▄······█···▄ beats 2/3	easy · avg 82%	▇ 0.9%	sharp · maxed out
DecisionBench v1 · verification (all-request accuracy) submitted	61.0% ✎	▁▁▅▁▃▂▂▁▁▇▇█ beats 41/56	medium · avg 56%	▇ 0.9%	blunt · at its level
LeanMCP · rewardbench2	79.3% ✎	▄▄▄·▄▄▄▄█·█▄ beats 9/11	medium · avg 58%	▇ 0.8%	typical
System One Bench: Typed decisions (English)	72.7% ✎	▁▁·▁▁▁▅▄▇██▁ beats 20/29	medium · avg 63%	▇ 0.8%	blunt · at its level
DecisionBench v1 · document record classification (all-request accuracy) submitted	56.6% ✎	▂▂·▁▂▄▃▆▄█▁▁ beats 51/56	medium · avg 46%	▇ 0.7%	at its level
LeanMCP · aegis2 response	81.2% ✎	█··▃·▃·█··▅▅ beats 9/11	easy · avg 70%	▇ 0.7%	typical
DecisionBench v1 · rubric scoring prioritization (all-request accuracy) submitted	55.7% ✎	▁▁▅▅▃▃▅█▃▃·▁ beats 48/56	medium · avg 45%	▇ 0.6%	at its level
Prevalence study · TREC-6	92.2% ✎	█··█·······█ beats 2/2	easy · avg 84%	▇ 0.6%	sharp · maxed out · small run
Prevalence study · CLINC150	97.8% ✎	██·········█ beats 2/2	very easy · avg 95%	▇ 0.6%	sharp · maxed out · small run
DecisionPivot v1.1 · cap boundary record (MSE skill)	0.2390 skill ✎	█··█····█··█ beats 3/3	avg -0.616 skill	▇ 0.6%	typical
DecisionPivot v1.1 · cap boundary card (MSE skill)	0.0490 skill ✎	█··█····█··█ beats 2/3	avg -0.709 skill	▇ 0.6%	typical
Support urgency · DecisionModelHub	99.3% ✎	█···█·····██ beats 2/3	very easy · avg 94%	▇ 0.6%	sharp · maxed out · noisy
DecisionPivot v1.1 · faithbench (accuracy)	62.5% ✎	█·█·█······█ beats 3/3	medium · avg 38%	▇ 0.5%	at its level
Prevalence study · FPB	75.2% ✎	█·········██ beats 2/2	medium · avg 60%	▇ 0.5%	small run
LeanMCP · helpsteer2 helpfulness	0.9598 ordinal levels ✎	▄▄▄▄▄·▄·█▄▄▄ beats 6/10	avg 1.357 ordinal levels	▇ 0.5%	blunt
DecisionPivot v1.1 · freshqa premise (auroc)	0.9410 auroc ✎	█·······█·██ beats 2/3	avg 0.209 auroc	▇ 0.5%	typical
DecideBench v1: ticket triage	98.0% ✎	▄·····▂█·▆·▂ beats 9/10	very easy · avg 90%	▇ 0.5%	sharp · maxed out
DecisionPivot v1.1 · known unknown (accuracy)	77.5% ✎	█··█······██ beats 0/3	easy · avg 69%	▇ 0.5%	typical
Prevalence study · HateXplain	51.4% ✎	█·█········█ beats 2/2	medium · avg 37%	▇ 0.5%	at its level · small run
DecisionPivot v1.1 · agentreward success (accuracy)	69.4% ✎	█···██·····█ beats 0/3	medium · avg 58%	▇ 0.5%	typical
LeanMCP · sst5	0.5071 ordinal levels ✎	▂·····▂▂▄█▄▂ beats 10/11	avg 0.879 ordinal levels	▇ 0.5%	blunt
DecisionPivot v1.1 · contextual judge (accuracy)	64.3% ✎	█·██·······█ beats 3/3	medium · avg 43%	▇ 0.5%	at its level
DecisionPivot v1.1 · bipia injection (accuracy)	97.7% ✎	█····█····██ beats 3/3	easy · avg 76%	▇ 0.5%	sharp · maxed out · noisy
DecisionPivot v1.1 · aegis2 prompt (accuracy)	82.6% ✎	█·····█·█··█ beats 0/3	easy · avg 73%	▇ 0.5%	typical
DecisionPivot v1.1 · jbb judge (auroc)	0.9810 auroc ✎	█····█··█··█ beats 3/3	avg 0.299 auroc	▇ 0.5%	typical
DecisionPivot v1.1 · xstest (auroc)	0.9930 auroc ✎	█········▄·▄ beats 2/3	avg 0.320 auroc	▇ 0.5%	typical
LeanMCP · atbench500	93.0% ✎	▃··▅▅█··▅··▅ beats 10/11	medium · avg 63%	▇ 0.4%	sharp · noisy
Prevalence study · HWU64	90.8% ✎	█··········▄ beats 2/2	very easy · avg 86%	▇ 0.4%	maxed out · small run
DecisionPivot v1.1 · cosapien policy (accuracy)	89.9% ✎	▄·······█··▄ beats 3/3	easy · avg 75%	▇ 0.4%	typical
DecisionBench v1 · triage (all-request accuracy) submitted	93.0% ✎	▁·▁·▂▃▄▃▂█▄▃ beats 54/56	easy · avg 78%	▇ 0.4%	sharp
LeanMCP · mcpbench tool selection	100.0% ✎	▂··········█ beats 9/9	very easy · avg 89%	▇ 0.4%	sharp · maxed out · noisy
DecisionBench v1 · review scoring (all-request accuracy) submitted	62.0% ✎	▁······▂▃▅▇█ beats 46/56	medium · avg 52%	▇ 0.4%	at its level
DecisionPivot v1.1 · label errors (accuracy)	60.6% ✎	██·······█·█ beats 0/3	medium · avg 47%	▇ 0.4%	at its level
LeanMCP · prompt injections	73.3% ✎	▂▂▂▂█·▄▂···▂ beats 6/11	medium · avg 62%	▇ 0.3%	typical
DecisionBench v1 · target selection (all-request accuracy) submitted	50.0% ✎	▁▁·▁▁▁▃▁▂▁▁█ beats 53/56	hard · avg 29%	▇ 0.3%	at its level · noisy
DecideBench v1: returns policy	96.0% ✎	▃··▃··▃·▅▅█▃ beats 10/10	easy · avg 81%	▇ 0.3%	sharp · noisy
DecisionBench v1 · retrieval ranking (all-request accuracy) submitted	60.0% ✎	▁▁·▁▁▂▃▂▃▁▁█ beats 50/56	medium · avg 35%	▇ 0.3%	at its level · noisy
DecisionBench v1 · content classification (all-request accuracy) submitted	81.0% ✎	▁····▁▁▁▁▁▁█ beats 53/56	easy · avg 73%	▇ 0.3%	typical
DecisionPivot v1.1 · jailbreak wild (accuracy)	78.3% ✎	█······█·█·█ beats 2/3	easy · avg 65%	▇ 0.3%	typical
DecisionPivot v1.1 · bfcl relevance (accuracy)	91.5% ✎	█··█······██ beats 3/3	easy · avg 82%	▇ 0.3%	typical
DecisionBench v1 · completion assessment (all-request accuracy) submitted	94.0% ✎	▁▁▂▃▄·▂▁▁▆█▂ beats 49/56	very easy · avg 86%	▇ 0.3%	sharp · maxed out
Chao Ma · clean support routing	99.7% ✎	█····█·····█ beats 2/2	very easy · avg 96%	▇ 0.2%	sharp · maxed out · small run
DecisionBench v1 · risk scoring (all-request accuracy) submitted	66.0% ✎	▁▁·▁▂▂▁▃▃█▇▃ beats 39/56	medium · avg 62%	▇ 0.2%	blunt · at its level
DecisionPivot v1.1 · ambigqa (accuracy)	64.4% ✎	█·······▄··▄ beats 3/3	medium · avg 44%	▇ 0.2%	at its level
DecisionBench v1 · bounded extraction (all-request accuracy) submitted	95.6% ✎	▃▃▂·▁▁▁▁▁▁█▅ beats 53/56	very easy · avg 86%	▇ 0.2%	sharp · maxed out · noisy
LeanMCP · mcpmark tool selection	100.0% ✎	▂··········█ beats 11/11	very easy · avg 96%	▇ 0.2%	sharp · maxed out · noisy
DecisionBench v1 · reasoning (all-request accuracy) submitted	74.5% ✎	▂·▁▁▆█▅▃▃▃▁▁ beats 51/56	medium · avg 52%	▇ 0.2%	at its level
DecisionPivot v1.1 · selfaware (accuracy)	59.0% ✎	█····▄·····▄ beats 0/3	medium · avg 53%	▇ 0.2%	blunt · at its level
Perch semantic lint · private (Youden J)	0.5211 J ✎	█····▄█▄···█ beats 5/7	avg 0.380 J	▇ 0.1%	blunt
Perch semantic lint · public (Youden J)	0.4953 J ✎	█····▄█▄···█ beats 5/7	avg 0.363 J	▇ 0.1%	blunt
DecisionBench v1 · document workflows (all-request accuracy) submitted	72.9% ✎	█▄▃▁▁▃▂▂▁·▁▁ beats 40/56	easy · avg 66%	▇ 0.1%	blunt · at its level
DecisionBench v1 · retrieval verification (all-request accuracy) submitted	52.7% ✎	█▁▂▁▁▁▁▁▂▅▁▁ beats 19/56	easy · avg 74%	▇ 0.1%	noisy
DecisionBench v1 · guardrails moderation (all-request accuracy) submitted	39.1% ✎	▁▁▁▂▆▃▃█···▁ beats 20/56	medium · avg 52%	▇ 0.1%	blunt · at its level · noisy
DecisionBench v1 · document classification (all-request accuracy) submitted	39.0% ✎	▂▁·▂▁▁▁▄▁▂▂█ beats 55/56	hard · avg 21%	▇ 0.1%	blunt · at its level · noisy
DecisionBench v1 · safety gating (all-request accuracy) submitted	57.0% ✎	▂·····▆█▂▂▄▇ beats 18/56	easy · avg 77%	▇ 0.0%	noisy
DecisionBench v1 · semantic filtering (all-request accuracy) submitted	54.0% ✎	▁·▁▁·▁▁█▂▃█▂ beats 17/56	easy · avg 71%	▇ 0.0%	blunt · noisy
Perch security detection · public (Youden J)	0.1101 J ✎	█···▄▄···▄▄█ beats 6/7	avg 0.052 J	▇ 0.0%	blunt
DecisionBench v1 · context management (all-request accuracy) submitted	51.0% ✎	▁·····▁▇▆▁█▃ beats 8/56	easy · avg 73%	▇ 0.0%	blunt · noisy
Perch bug detection · private (Youden J)	0.0449 J ✎	▅··▃▃█·····▃ beats 5/7	avg 0.027 J	▇ 0.0%	blunt
DecisionBench v1 · content moderation (all-request accuracy) submitted	45.0% ✎	▁····▃▅▃▂▂·█ beats 3/56	easy · avg 82%	▇ 0.0%	noisy
DecisionBench v1 · entity alignment (all-request accuracy) submitted	100.0% ✎	▁·▁▄▇▄▅▄▂▁▄█ beats 47/56	easy · avg 66%	▇ 0.0%	noisy
Perch bug detection · public (Youden J)	0.0496 J ✎	█▄·█·▄▄····▄ beats 5/7	avg 0.026 J	▇ 0.0%	blunt
Perch security detection · private (Youden J)	0.0394 J ✎	█▃▃··▃·▃···▃ beats 5/7	avg 0.017 J	▇ 0.0%	blunt
DecisionBench v1 · function agent skill routing (all-request accuracy) submitted	99.0% ✎	▃▃▄▅▆▂▅▄▃▄█▃ beats 56/56	easy · avg 68%	▇ 0.0%	blunt · noisy
LeanMCP · jailbreak classification	97.5% ✎	▂·▄··▂···█▆▃ beats 11/11	very easy · avg 95%	▇ 0.0%	blunt · maxed out · noisy
DecisionBench v1 · routing triage (all-request accuracy) submitted	84.8% ✎	█▁···▁·▂▆▅▃▂ beats 52/56	medium · avg 42%	▇ 0.0%	blunt · at its level · noisy
system1bench · ag news	88.6% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
system1bench · emotion	59.0% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
system1bench · banking77	79.9% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · sst5	56.5% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · massive en	77.5% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · massive zh	76.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · prompt injections	76.7% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · typed decisions	73.9% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jevbench original	98.6% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jevbench easy	100.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jevbench hard	72.1% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · reflexbench reflex-public-choice-v1	93.7% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya triage	90.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya moderation	92.7% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya routing	88.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya claims	100.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya reviews	87.7% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya guard	93.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya multilingual	100.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · jev laya needle	95.2% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · clinc150 oos	89.5% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
system1bench · aegis2 prompt	82.4% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Prism-1 study · Banking20	87.0% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
Prism-1 study · Newsgroups	75.3% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Prism-1 study · BEV decision mix	76.4% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Prism-1 study · Typed decisions	72.8% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Prism-1 study · JevBench held-out	88.5% ✎	█··········█ beats 0/1		—	not counted yet: no other model in the same run to compare it with
Structured state reasoning · V3 probe	85.0% ✎	█··········█ beats 1/1		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · NEU-ESC · Topic classification	37.3% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · UIT-VSFC · Topic classification	74.4% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · UIT-VSMEC · Emotion recognition	50.0% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · ViHSD · Hate speech detection	59.3% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · UIT-ViCTSD · Hate speech detection	69.3% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · UIT-ViOCD · Hate speech detection	92.8% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · UIT-VSFC · Sentiment analysis	73.4% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · PhoATIS · Intent detection	79.3% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Vietnamese EN labels · ViMQ · Intent detection	80.2% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
PriorBench · French reference classification	95.9% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Morrenhale · typed-decisions	73.2% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Morrenhale · agnews	89.2% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Morrenhale · emotion	58.9% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Morrenhale · banking77	80.0% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with
Morrenhale · massive	67.7% ✎	no other model yet		—	not counted yet: no other model in the same run to compare it with

consistency█····█·····█ #3 of 3 * · 0.0% of its score · 1 test

Trails open-jev-deberta-v3-large (DecisionBench 19bf9a64), decider-2b (DecisionBench fa996cea) here, though they rank lower overall.

test	result	vs the others in its run	difficulty	counts	why
Chao Ma · confidence under authority injection	0.2610 auroc ✎	█····█·····█ beats 0/2	avg 0.426 auroc	▇ 0.0%	blunt · small run

other tests4 not in a task · 2.9% of its score

test	result	vs the others in its run	difficulty	counts	why
Decision Index 0.4.1: private same skills (equated points)	57.6600 points ✎	▂▂▂▃▃▃▄▅▅▇█▃ beats 191/247	avg 41.206 points	▇ 1.4%	blunt
Decision Index 0.4.1: private new domains (equated points)	61.5800 points ✎	▃█▃▃▂▆▃▃▄▄▄▄ beats 191/247	avg 38.861 points	▇ 0.8%	blunt
Benchmark Heaven jevbench v1.6.1: intelligence (points)	63.6208 points ✎	▃···▃·▃▃█▅▅█ beats 8/13	avg 32.166 points	▇ 0.6%	blunt
Benchmark Heaven jevbench v1.6.1: calibration (points)	90.6322 points ✎	▂······▂▆▂██ beats 10/13	avg 70.133 points	▇ 0.2%	blunt

$ where to run it

Through an API

api · $0.042/M input tokens · docs.typesafe.ai ↗ · price sourceObserved 2026-10-31: $0.042 per million input tokens; output tokens free. Versioned model ID jev-1.13.0.

Model repository and documentation: docs.typesafe.ai ↗

$ how the score was worked out · DecisionPivot · StartLux Labs 23%, Decision Index 0.4.1 20%, LeanMCP · JevBench 12% and 14 more

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

source	results	share of the score	licence
DecisionPivot · StartLux Labs	33	▇▇▇▇▇▇▇ 23%	CC BY 4.0
Decision Index 0.4.1 · submitted; independent execution unverified	42	▇▇▇▇▇▇ 20%	CC BY 4.0
LeanMCP · JevBench	18	▇▇▇▇ 12%	MIT
Hanno Labs DecisionBench v1 reviewed results · submitted; independent execution unverified	27	▇▇▇▇ 12%	CC0 1.0
System One Bench (Ali Alhashimi)	21	▇▇▇ 10%	CC-BY-4.0
Etsabary · decision benchmarks	2	▇▇ 6%	MIT
Decision Models as Judges for Hallucination Detection	5	▇▇ 5%	CC-BY-4.0
DecideBench (Cho Yin Yong)	8	▇ 3%	CC BY 4.0
Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors	5	▇ 3%	CC-BY-4.0
DecisionModelHub · support benchmarks	2	▇ 1%	CC-BY-4.0
Benchmark Heaven (own decision evaluations)	4	▇ 1%	CC BY 4.0
Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG	1	▇ 1%	CC-BY-4.0
AgentDecisionBench	2	▇ 1%	CC-BY-4.0
Freyer et al. · MedQA	1	▇ 1%	CC BY 4.0
Perch benchmark results	6	▇ 0%	CC BY 4.0
Chao Ma · independent injection study	2	▇ 0%	MIT
Citead GEO Citation Benchmark (October 2026)	1	▇ 0%	CC BY-SA 4.0
prior · pull toward the middle		▇ 0%
weighted average		= 65.9

not counted (59)

Decision Index 0.4.1	Decision Index 0.4.1: RouterBench (selected quality (quality objective)) [correct]	79.9%	failed run: Publisher excludes this protocol because its prompt reveals GPT-4 as the best route. Evidence: https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/c130faf65f82e81d3bdefb01e758c712dc5c5199/data/index.json
Decision Index 0.4.1	Decision Index 0.4.1: SGD/SGD-X (macro-F1) [correct]	43.0%	failed run: Publisher reports a builder bug exposing dialogue turns after the classified turn; awaiting corrected rerun. Evidence: https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/c130faf65f82e81d3bdefb01e758c712dc5c5199/data/index.json
AgentDecisionBench	AgentDecisionBench: L2 DecisionGym-Shift (100 tasks) [correct]	91.8%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · ag news [correct]	88.6%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · emotion [correct]	59.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · banking77 [correct]	79.9%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · boolq [correct]	92.6%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · sst5 [correct]	56.5%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · xnli en [correct]	85.9%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · xnli zh [correct]	73.6%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · massive en [correct]	77.5%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · massive zh [correct]	76.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · prompt injections [correct]	76.7%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · typed decisions [correct]	73.9%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jevbench original [correct]	98.6%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jevbench easy [correct]	100.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jevbench hard [correct]	72.1%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · reflexbench reflex-public-choice-v1 [correct]	93.7%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya triage [correct]	90.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya moderation [correct]	92.7%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya routing [correct]	88.8%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya claims [correct]	100.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya reviews [correct]	87.7%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya guard [correct]	93.8%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya multilingual [correct]	100.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · jev laya needle [correct]	95.2%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · clinc150 oos [correct]	89.5%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · turtlebench [correct]	74.0%	fewer than three models have this test yet, or this source ran it for this model alone
CYMCharming · system1bench	system1bench · aegis2 prompt [correct]	82.4%	fewer than three models have this test yet, or this source ran it for this model alone
Prism-1: Position- and Rival-Independent Scoring for System One Models	Prism-1 study · Banking20 [correct]	87.0%	fewer than three models have this test yet, or this source ran it for this model alone
Prism-1: Position- and Rival-Independent Scoring for System One Models	Prism-1 study · Newsgroups [correct]	75.3%	fewer than three models have this test yet, or this source ran it for this model alone
Prism-1: Position- and Rival-Independent Scoring for System One Models	Prism-1 study · BEV decision mix [correct]	76.4%	fewer than three models have this test yet, or this source ran it for this model alone
Prism-1: Position- and Rival-Independent Scoring for System One Models	Prism-1 study · Typed decisions [correct]	72.8%	fewer than three models have this test yet, or this source ran it for this model alone
Prism-1: Position- and Rival-Independent Scoring for System One Models	Prism-1 study · JevBench held-out [correct]	88.5%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · NEU-ESC · Topic classification [correct]	37.3%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · UIT-VSFC · Topic classification [correct]	74.4%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · UIT-VSMEC · Emotion recognition [correct]	50.0%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · ViHSD · Hate speech detection [correct]	59.3%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · UIT-ViCTSD · Hate speech detection [correct]	69.3%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · UIT-ViOCD · Hate speech detection [correct]	92.8%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · UIT-VSFC · Sentiment analysis [correct]	73.4%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · PhoATIS · Intent detection [correct]	79.3%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · ViMQ · Intent detection [correct]	80.2%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · ViNLI · Natural language inference [correct]	81.0%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · ViANLI · Natural language inference [correct]	53.5%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · ViMedNLI · Natural language inference [correct]	73.4%	fewer than three models have this test yet, or this source ran it for this model alone
Vietnamese decision model transfer study	Vietnamese EN labels · vnNLI · Natural language inference [correct]	85.9%	fewer than three models have this test yet, or this source ran it for this model alone
Structured state reasoning probe	Structured state reasoning · V3 probe [correct]	85.0%	fewer than three models have this test yet, or this source ran it for this model alone
PriorBench	PriorBench · French reference classification [correct]	95.9%	fewer than three models have this test yet, or this source ran it for this model alone
Morrenhale · Decision benchmark	Morrenhale · typed-decisions [correct]	73.2%	fewer than three models have this test yet, or this source ran it for this model alone
Morrenhale · Decision benchmark	Morrenhale · agnews [correct]	89.2%	fewer than three models have this test yet, or this source ran it for this model alone
Morrenhale · Decision benchmark	Morrenhale · emotion [correct]	58.9%	fewer than three models have this test yet, or this source ran it for this model alone
Morrenhale · Decision benchmark	Morrenhale · banking77 [correct]	80.0%	fewer than three models have this test yet, or this source ran it for this model alone
Morrenhale · Decision benchmark	Morrenhale · massive [correct]	67.7%	fewer than three models have this test yet, or this source ran it for this model alone
Anthus · Hard-Decisions	Hard-Decisions · ProofWriter CWA [correct]	89.3%	fewer than three models have this test yet, or this source ran it for this model alone
Anthus · Hard-Decisions	Hard-Decisions · ProofWriter OWA [correct]	83.8%	fewer than three models have this test yet, or this source ran it for this model alone
JevResearch	JevResearch · MMLU-Pro native choice [correct]	82.7%	fewer than three models have this test yet, or this source ran it for this model alone
JevResearch	JevResearch · GPQA Diamond native choice [correct]	76.5%	fewer than three models have this test yet, or this source ran it for this model alone
JevResearch	JevResearch · HLE text multiple-choice [correct]	21.9%	fewer than three models have this test yet, or this source ran it for this model alone

$ improve this page

Update model information Add a place to run it Add a test result Propose a benchmark Correct a benchmark score Edit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated 2026-11-03 02:37 UTC · sources and licences · json