JevBench: An Open Evaluation Framework for Typed Decision Models

Source abstract

A class of models does not generate text but returns typed decisions: a probability for a yes/no question, a distribution over a set of choices, or an expected level on an ordered rubric. We present JevBench, an open evaluation framework that runs models exposing this interface on identical task inputs across 12 slices covering medicine, examinations, banking intent, sentiment and agent safety. We evaluate four systems: the hosted Jev 1.13 and Liquid D1 services, djev-0.1 on a DiffusionGemma 26B-A4B backbone, and Laya, a 421M-parameter local model. The original evaluation contains 24,799 decisions; the D1 extension adds 8,014 successful text decisions, with two rate-limit failures. The original three-system evaluation exposes threshold and option-order effects. On prompt-injection detection Jev reaches an AUROC of 0.978 yet only 73.3% accuracy at 0.5; selecting a threshold on the scored data raises accuracy to 94.8%, an optimistic diagnostic rather than a held-out gain. Correctness changes under option reordering on 2.6–5.4% of matched items for Jev, against 12.0–33.6% for djev and Laya. On shared cases, D1 exceeds Jev by 4.4 percentage points on BANKING77 and 3.8 on Aegis prompts, but trails by 14.1 points on MMLU-Pro. These D1 comparisons are point estimates, without paired uncertainty estimates in this revision. We additionally evaluate djev’s image path: accuracy is 82.8% on image-bearing ScienceQA rows and 84.7% on text-only rows. The framework and original predictions are released; the D1 extension is recorded separately from the existing dataset release.