Auditing JevBench

Applying JevBench’s scoring functions to exact gold responses on its public hard tier produces a calibration score of 97.0. The same responses receive 100% accuracy and 100% probability fidelity. The difference comes from treating uncertainty about an event as uncertainty about answering the question correctly.

The score arithmetic for all 31 system rows reproduces from the supplied aggregates, and the repository’s 68 tests pass. I followed my benchmark audit checklist at commit f8ce713, using results revision v1.2.7. That snapshot ranks 26 systems over 534 decisions for each complete run.

Benchmark Heaven maintains JevBench independently and explicitly disclaims affiliation with Jev’s maker, TypeSafe. The benchmark evaluates models that return a typed decision from a piece of state and a bounded question, ideally with probabilities over the allowed answers. Rerunnable probes and receipts.

Two meanings of 65%

The public question hard-opus-b-probability-02 asks an incident-response team for root-cause probabilities based on a table of past outages. The new alert followed no recent deployment. Twenty historical incidents share that property: four came from bad pushes, thirteen from an upstream provider and three from hardware.

The answer key gives the corresponding probabilities as 20%, 65%, 15%. The most likely cause is the upstream provider. Task and gold.

The probability that the provider caused the incident is 65% under the supplied reference distribution. Confidence that thirteen divided by twenty is the correct calculation can be 100%.

The calibration code uses the largest returned probability as confidence in the selected label. Here that label matches the key, so the calculation compares 0.65 with a correctness value of 1. It counts the remaining 0.35 as underconfidence, although that uncertainty is part of the correct answer.

The composite calibration score averages two components. One measures agreement between confidence and label correctness using expected calibration error (ECE). The other measures fidelity to the gold probability distribution using total variation distance (TVD). On this item, the first rewards more certainty; the second rewards preserving the stated uncertainty.

For this item alone, the confidence component is max(0, 100 × (1 − 2 × ECE)); the fidelity component is 100 × (1 − TVD). The exact gold response scores 30 and 100 on the two components, averaging 65. The one-hot response scores 100 and 65, averaging 82.5.

This controlled probe demonstrates the formula’s ordering for one fixed pair of responses. The two components use different denominators across the full suite, so establishing an effect on the leaderboard would require regrading the models’ actual responses.

The same mismatch appears across the full public hard set. All ten probability items receive their exact gold distributions; the other 101 receive one-hot gold labels. Every label passes and every probability distribution matches its target, yet the calibration score is 97.0023. Probe output.

Composite scoring

Jev ranks first on the JevBench Score and third on its weighted-accuracy view, behind GPT-5.6 Luna and DeepSeek. The overall score combines accuracy, calibration, speed and cost through a geometric mean. Each transformed axis has 25% weight. Formula.

A model that returns labels without probabilities has no calibration score. The composite treats this as zero and then floors it to one. A hypothetical classifier scoring 100 on accuracy, speed and cost would therefore reach 31.6 out of 100 overall.

For the speed axis, nonproduction endpoints’ standard-plus-judge p50 and p95 latencies are doubled. The benchmark authors’ own servers receive another 0.15 seconds. The authors label this adjustment as an assumption about production load. Some self-hosted models are priced using estimated hosted tariffs. The resulting board compares models, deployment conditions and economic assumptions together.

The score changed after measurement, as the revision history records. The work-in-progress score had three axes and assigned half its capability weight to hard tasks. The final score has four axes and assigns 30% of its accuracy weight to hard tasks. Earlier views remain available.

Model configurations

The shipped results report accuracy and latency by tier, with cost averaged across the full run:

DeepSeek’s hard-tier result counts eight failed responses as wrong. It has 209 correct answers and three incorrect answers among the remaining responses. Jev gets 163 correct and Gemini 165. Compared with Jev, Gemini gains two correct decisions and DeepSeek gains forty-six.

Jev costs about one-fifteenth as much as the tested DeepSeek configuration, which the artifact labels thinking default. The adapter asks for a JSON probability for every option. Reported hard-tier output usage averages about 1,753 tokens per decision. The published comparison does not measure the cost and accuracy of DeepSeek with thinking disabled and a short constrained label, and I have not run that configuration.

Across all 534 decisions, Jev has 468 correct answers and DeepSeek 511. For that same cohort, DeepSeek costs $0.554 more per thousand decisions and returns 80.5 more correct answers per thousand. Dividing the cost difference by the accuracy difference gives $0.00688 per additional correct answer, about 0.69 cents. This calculation uses unweighted task counts and treats errors equally. It excludes latency, retries and downstream costs.

Published artifacts

The checkout contains 231 of the 534 task texts. These comprise 48 easy, 72 standard and 111 hard items. The other 303 are held out or imported, including all 146 judge items.

For Jev, Gemini and DeepSeek, the public per-task file contains a verdict code and rounded latency. It does not contain the selected label, probability vector, request or response. Fuller records are shipped for another entrant, but not these three.

The supplied data supports recounting verdict codes and reproducing the composite from aggregate calibration numbers. Without the responses, I cannot inspect the primary models’ incorrect answers, reconstruct their calibration metrics or regrade their distributions under a different interpretation.

The benchmark’s runner captures responses and their hashes outside the public repository. Publishing those public-item responses would not require releasing the withheld task texts.

Separate phishing comparison

The Haiku comparison comes from a different benchmark, on 2,000 phishing and legitimate emails with synthetic bodies. Haiku is absent from the audited JevBench snapshot.

That study reports 62.6% accuracy for Jev and 81.3% for Haiku 4.5 on the direct verdict. Both kinds of error rise with Jev: phishing recall falls from 76.4% to 43.2%, while the false-positive rate rises from 13.8% to 18.0%. Its published confusion matrices reproduce those percentages; its raw responses are not shipped, so this is an arithmetic check of reported results.

At its reported list prices, choosing Jev saves $0.424 per thousand emails while adding 187 errors on that balanced dataset. That is about 0.23 cents saved per additional error.

The same study reports higher accuracy after extracting five signals and fitting a classifier. Jev reaches 95.0% against Haiku’s 93.2% on an internal holdout, with reported paired p = 0.063. The dataset’s construction informed the signal questions, and the holdout analysis followed the initial exploration. I did not examine independent external validation of that pipeline.

Validation and limits

All 231 public keys pass the scorer when encoded as one-hot answers. This checks each key’s compatibility with the grader; semantic correctness across the full set remains unaudited. Malformed probability maps fail closed in the mutations I ran. The aggregate score arithmetic and all 26 ranks agree. The authors disclose their deployment assumptions, their nonaffiliation with TypeSafe and earlier cost corrections.

The hard questions were model-authored and cross-reviewed. The 95% human-performance target appears in the authoring specification; no measured human baseline is reported. I found no proven miskeyed label. The calibration witness establishes a mismatch between a requested quantity and its evaluation. Its effect on the winning rank remains unmeasured.

Discussion

I would start with DeepSeek V4.1 Flash. It would take more evidence to convince me to choose Jev. On this measured mix, DeepSeek costs about $0.55 more per thousand decisions and returns about 81 additional correct answers. The accuracy difference is substantial; the absolute price difference is small. For work whose errors require human review or correction, less than a cent per additional correct answer is a modest premium.

To choose Jev, I would want evidence from the intended workload that its accuracy is acceptable and that its lower cost or latency materially improves the application. Naming a possible use case does not supply that evidence. Neither does a large cost ratio when both absolute prices are low. Until that comparison exists, the measured accuracy advantage gives me a reason to prefer DeepSeek.

I would still test DeepSeek on the intended workload. DeepSeek’s observed result already includes thinking cost. A configuration with thinking disabled would need its own accuracy and cost measurements.

Separating distribution fidelity from confidence calibration would make JevBench’s probability scores easier to interpret. Public-item responses would allow independent checks of those scores. The cost comparison also needs measurements of DeepSeek with thinking disabled and a short constrained label. Show the measured accuracy, price and latency together, and let the application set the acceptable error rate. The overall rank fixes tradeoffs among accuracy, calibration, speed and cost. Each application still has to justify those tradeoffs.

Receipts: audit script, results and hashes, reproduction instructions. This audit uses the pinned public artifacts and makes no new model calls. The findings have not yet received a maintainer response.