THE RESEARCH RECORD
108 experiments on Jev
Jev is an AI model made by a company called TypeSafe. Unlike a chatbot, it does not write sentences, code, or explanations. You hand it some material, such as a support ticket, a paragraph, or a record, plus a fixed set of questions with fixed possible answers, and it tells you which answer it picked, in about a tenth of a second per question.
We handed it: Some text: a support ticket, a document, a record
We asked: Fixed questions with fixed answer options
It returned: The chosen answer, and a probability for every option
There are three kinds of question. A Choice question picks one option from a list you supply, up to 255 options. A Score question places the material on a numbered scale you define, from 2 up to 10 levels, and its answer can land between two levels rather than exactly on one. A Noul question answers a plain yes or no question and returns a single number: the probability that the answer is yes.
Alongside its pick, Jev returns a probability for every option it was given. For Choice and Score it also returns a confidence number from 0 to 1, how sure it is, kept separate from which option it chose. A Noul has no separate confidence, because its one number already carries that meaning: near 1 is a strong yes, near 0 a strong no, near 0.5 means it cannot tell.
Jev cannot generate text, write code, or explain its reasoning. It only reads text, not images, audio, or video. It cannot chain decisions together on its own, and it cannot use one question's answer as hidden context for another question asked in the same request.
108 experiments7 groupsLimits and error messagesE01–E16 · 16 experimentsE01E02E03E04E05E06E07E08E09E10E11E12E13E14E15E16The confidence number, testedE17–E28 · 12 experimentsE17E18E19E20E21E22E23E24E25E26E27E28Choosing where a request goesE29–E46 · 18 experimentsE29E30E31E32E33E34E35E36E37E38E39E40E41E42E43E44E45E46Gate and verifyE47–E62 · 16 experimentsE47E48E49E50E51E52E53E54E55E56E57E58E59E60E61E62Scoring, curating, and finding textE63–E78 · 16 experimentsE63E64E65E66E67E68E69E70E71E72E73E74E75E76E77E78Multi-step tasks and rechecking claimsE79–E92 · 14 experimentsE79E80E81E82E83E84E85E86E87E88E89E90E91E92Headlines, claims, and sourcesE93–E108 · 16 experimentsE93E94E95E96E97E98E99E100E101E102E103E104E105E106E107E108
What we found
·
Ask Jev to rate something on more than 10 levels and it does not quietly cap the request: it returns an HTTP 400 error, not the 422 the docs describe (E01).
·
The same holds for picking one of many options: past 255 choices the request fails with a 400 error, and nothing is silently trimmed to fit (E02).
·
Confidence is not just another name for the top answer's probability: across 360 answers, confidence came in lower than the top probability 25.8% of the time, and the gap widened as the list of options grew (E18).
·
A written permission policy that correctly blocked 95% of actions it should refuse also blocked 11 of 120 actions it should have allowed, a 9.2% false positive rate (E47).
·
Asking Jev the exact same question twice does not reliably give the exact same answer: across 20 repeats of one request, only 2 of 12 questions came back byte-identical every time (E12).
·
A widely repeated claim that Jev could not navigate a map at all, succeeding 0 times in 128 tries, did not hold up on a rerun, where it succeeded 75 times out of 128 (E87).
108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.
Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.