Jev says it's 90% sure. Is it right 90% of the time? We measured it, and what it means for automating a business decision

In our test, when Jev said it was at least 90% sure, it was right 91 to 94% of the time. Below that line its confidence overstated its accuracy, sometimes badly. That is enough to use it as a gate in front of an automated decision, and not enough to let it make the decision alone.

Jev is the model TypeSafe AI released on 15 September 2026. It doesn't generate text. You give it a piece of state (an email, a ticket, a JSON record) and a closed question: true or false, pick one of these options, place this on a scale. It returns probabilities and a confidence number, in under a second, at $0.042 per million input tokens with output free. The launch reached 1,883 points on Hacker News and the claim that travelled furthest was "it can't hallucinate."

For anyone automating a business process, the interesting part is the confidence number. When a large language model classifies an invoice or routes a ticket today, you see the choice, not how sure it was. Ask it in words and it says 95% nearly every time. A number you can trust would let you put a threshold on the decision: above it the step runs alone, below it a person looks. That only works if the number means what it says. At launch, nobody outside TypeSafe had published a measurement: no calibration curve, no paper. So we ran one.

What did we actually measure?

Two public labeled datasets, one run each, about five cents of API spend in total.

Three things stand out.

The top of the range holds. Answers Jev scored between 0.93 and 1.00 claimed 0.98-0.99 and were right 93-96% of the time. Slightly optimistic, usable.

The middle does not. In the 0.87-0.93 band it was right 75% of the time on the tweets and 62% on the banking requests. Between 0.5 and 0.7 on the banking set it was right 36-50% of the time. TypeSafe's documentation suggests treating 0.5-0.9 as "ask for confirmation." On our data that band is better read as "probably wrong."

Two thirds of the volume clears the bar. Between 61% and 67% of answers came back at 0.9 or above. That is the share of decisions a threshold would let through without a person, at an error rate of 6 to 9%.

We have since extended this to twelve tasks and seven experiments (what happens when the right answer isn't among the options, the same requests in English and Italian, 5 versus 60 options) and published the code and the results: jev-bench on GitHub. One Python file, standard library only, a few cents to rerun on your own key. The headline from the wider run: above 0.9, accuracy ranged from 99.8% on spam to 81.6% on exact star ratings, so the threshold has to be set per task.

The caveats are real. The 0.9 line above is drawn on the top option's probability; drawn on Jev's separate confidence field it gives 95.4% and 90.7%, the same picture. One run, not three. Both datasets have noisy labels, which lowers measured accuracy and inflates the calibration error (ECE was 0.070 and 0.104). Latency was 0.87 seconds from Italy, not the 70-500 ms on the product page. And the model is not deterministic: the same request three times returned 0.94, 0.96 and 0.96.

Does "it can't hallucinate" hold up?

Only in the narrow sense TypeSafe itself describes: the answer always matches the schema you supplied, so Jev cannot return an option you didn't give it. It can return the wrong one. We asked it to route "what time does the pool open?" to billing, technical support or sales. It chose sales at 0.64. There was no right answer in the list and it didn't say so, because it has no way to. If "none of the above" is possible in your process, it has to be one of the options.

It is also not smarter than a large model, and TypeSafe doesn't claim it is. On their published workflow evaluations Jev is not the most accurate in any of the four, and the widest gap is on invoice processing: 61.8 against 79.1. Those evaluations measure agreement with two large models, not ground truth, which matters less than the direction: you are trading accuracy for speed and cost.

So what is it good for?

A gate. We tested that directly, using Jev the way an AI agent would: as a check it calls on itself before acting. Unlike the benchmarks above, these are 24 cases we wrote by hand, with the expected answer decided in advance: an illustration, not a measurement.

We gave it ten build logs and asked whether the tests had really passed. It got ten out of ten, including a log where someone had simply printed "All tests passed!" (0.04), a run that found no test files and exited clean (0.03), and the one ambiguous case, green tests with a failing type check, which landed in the middle at 0.40.

We then asked it fourteen times, in English and in Italian, whether an action fell within what the user had asked for. It got eleven right in both languages, with the same three errors, and all three went the same way: it said "ask first" when the agent could have proceeded. It never said "go" on something that shouldn't happen. Deleting a failing test, force-pushing, sending instead of saving a draft, paying instead of listing invoices: all scored 0.29 or lower.

That asymmetry is what you want from a gate. A false "ask" costs a person thirty seconds. A false "go" costs an incident. And at under two cents per thousand checks, you can afford to ask at every step, not once per process.

How would this change an automation we already run?

Most production automations that touch money or customers have a human-in-the-loop phase: for the first weeks, a person reviews every decision, you build the confusion matrix, and you decide which categories the system can handle alone. We do that by hand on every build. It works, and it is slow, because the underlying model gives you a label with no usable measure of doubt.

A calibrated confidence number shortens that phase. The pattern looks like this:

1. The large model, or your existing pipeline, does the work: reads the invoice, the contract, the intake form, extracts, reasons.
2. At each decision point, a cheap closed question goes to the decision model: is this a duplicate, does this fall under the standard policy, does this match the purchase order.
3. At 0.9 or above, the step runs. Below it, the item goes to a person, with the score attached so they know why they're looking at it.

On our numbers that lets roughly two thirds of decisions through at a 6-9% error rate. Whether 6-9% is acceptable depends entirely on the decision. For tagging a support ticket it's fine. For approving a payment it isn't, and the threshold moves up or the step stays human. The point is that the choice becomes a number you set and can audit, not a judgment about how much you trust a model's tone.

What should you check before building on it?

• Measure on your own documents. Our figures come from tweets and banking requests. Yours are invoices or claims or tickets. A few hundred labeled examples and an afternoon are enough to draw your own curve.
• Treat the middle band as a no. Until your own data says otherwise, anything under 0.9 goes to a person.
• Write the exit option. "None of these" and "not enough information" must be in the list, or the model will pick something.
• Check the terms. At the time of writing Jev is in early access, the architecture is unpublished, and the promised paper isn't out. We have not verified the data-retention terms. For anything sensitive, read them first.
• Don't rebuild around one vendor. Within four days of launch there were open-source attempts at the same idea on the Hacker News front page. The pattern, a fast calibrated gate next to a slow capable model, will outlast the product.

Where does this leave your process?

If you have an automation stuck in review-everything mode, or a process you haven't automated because nobody could say how often the AI would be wrong, that is the question a confidence threshold answers. The free AI Audit takes about five minutes and points to the processes where a gated automation would pay back first. If you'd rather talk it through, book a call. How we build the extraction side is in Back-Office and Finance Automation, and the case for keeping a human in the loop on invoices is in why AP automation still needs manual review.