START HERE
What Jev is
Jev is an AI model made by a company called TypeSafe. Unlike a chatbot, it does not write sentences, code, or explanations. You hand it some material, such as a support ticket, a paragraph, or a record, plus a fixed set of questions with fixed possible answers, and it tells you which answer it picked, in about a tenth of a second per question.
There are three kinds of question. A Choice question picks one option from a list you supply, up to 255 options. A Score question places the material on a numbered scale you define, from 2 up to 10 levels, and its answer can land between two levels rather than exactly on one. A Noul question answers a plain yes or no question and returns a single number: the probability that the answer is yes.
Alongside its pick, Jev returns a probability for every option it was given. For Choice and Score it also returns a confidence number from 0 to 1, how sure it is, kept separate from which option it chose. A Noul has no separate confidence, because its one number already carries that meaning: near 1 is a strong yes, near 0 a strong no, near 0.5 means it cannot tell.
Jev cannot generate text, write code, or explain its reasoning. It only reads text, not images, audio, or video. It cannot chain decisions together on its own, and it cannot use one question's answer as hidden context for another question asked in the same request.
What we did
This site is the record of 108 experiments, run in 7 groups, against the real Jev API. Each experiment started as a claim that could lose: a plain statement of what we expected, plus the exact result that would prove it wrong, written down before any call was made.
A group is a set of related experiments on one broad question, such as whether Jev's confidence numbers mean anything, or whether it can pick the right item out of a long list. An experiment id like E01 just marks its place in the running order across all 108 experiments, from E01 to E108; it is not a ranking of importance.
Every experiment ends in a verdict: confirmed, falsified, mixed, inconclusive, or blocked. Every number on this site is read straight from that experiment's own results file, the file the run itself produced, never estimated or remembered afterward.
How to read a verdict
C
Confirmed
The claim held up: what we expected is what we measured.
F
Falsified
The claim did not hold up: what we measured was different from, or the opposite of, what we expected.
M
Mixed
Part of the claim held up and part did not: different pieces of the same hypothesis came out different ways.
I
Inconclusive
The experiment ran, but the result could not settle the claim either way.
B
Blocked
The experiment could not run: the real input it needed did not exist yet, and the programme does not allow invented data to fill the gap.
108
experiments
38,194
API responses logged
45.9M
input tokens
$1.93
at $0.042/Mtok
108 experiments7 groupsLimits and error messagesE01–E16 · 16 experimentsE01E02E03E04E05E06E07E08E09E10E11E12E13E14E15E16The confidence number, testedE17–E28 · 12 experimentsE17E18E19E20E21E22E23E24E25E26E27E28Choosing where a request goesE29–E46 · 18 experimentsE29E30E31E32E33E34E35E36E37E38E39E40E41E42E43E44E45E46Gate and verifyE47–E62 · 16 experimentsE47E48E49E50E51E52E53E54E55E56E57E58E59E60E61E62Scoring, curating, and finding textE63–E78 · 16 experimentsE63E64E65E66E67E68E69E70E71E72E73E74E75E76E77E78Multi-step tasks and rechecking claimsE79–E92 · 14 experimentsE79E80E81E82E83E84E85E86E87E88E89E90E91E92Headlines, claims, and sourcesE93–E108 · 16 experimentsE93E94E95E96E97E98E99E100E101E102E103E104E105E106E107E108
The seven groups
E01–E16
16 experiments
Limits and error messages
We sent Jev deliberately broken, oversized, and edge-case requests to find out where it actually breaks and what it tells you when it does.
Confirmed 12 of 16
Falsified 3 of 16
Mixed 1 of 16
E17–E28
12 experiments
The confidence number, tested
This group tested what Jev's confidence number is good for: telling right answers from wrong ones, warning when an answer might flip, and deciding when a gate should let an answer through on its own.
Confirmed 7 of 12
Falsified 4 of 12
Mixed 1 of 12
E29–E46
18 experiments
Choosing where a request goes
We tested whether Jev can pick the right destination for a request as reliably as the hand-built matching code it would replace.
Confirmed 5 of 18
Falsified 5 of 18
Mixed 1 of 18
Inconclusive 1 of 18
Blocked 6 of 18
E47–E62
16 experiments
Gate and verify
This set of tests checks whether Jev can be trusted to gate risky actions before they happen and verify claims and work after the fact.
Confirmed 7 of 16
Falsified 5 of 16
Mixed 4 of 16
E63–E78
16 experiments
Scoring, curating, and finding text
This group tested how to build a good rating question, whether a curation decision should discard its distribution, and how well Jev can point at one piece of text inside a longer document.
Confirmed 7 of 16
Falsified 3 of 16
Blocked 6 of 16
E79–E92
14 experiments
Multi-step tasks and rechecking claims
This group tested whether doing well on single steps predicts doing well over a whole task, and re-ran several widely cited results from other projects to see whether they hold up.
Confirmed 6 of 14
Falsified 8 of 14
E93–E108
16 experiments
Headlines, claims, and sources
Whether Jev can read a short headline or claim and correctly judge its stance, its packaging, its overstatement, and whether the evidence behind it actually holds up.
Confirmed 11 of 16
Falsified 1 of 16
Mixed 3 of 16
Blocked 1 of 16
Words used here
Jev
The AI model this whole site is about, built by a company called TypeSafe. It does not generate text; it answers fixed questions with typed answers and probabilities.
System One model
TypeSafe's name for the kind of model Jev is: built for fast, narrow judgments rather than open-ended writing or multi-step reasoning.
state
The material you hand Jev to judge, such as a support ticket, a paragraph, or a record. It can be plain text, a JSON object, or a list of text values.
question
One thing you ask Jev to decide about the state. Every question has a fixed type and a fixed set of possible answers.
Choice
A question type that picks one option from a list you supply, up to 255 options.
Score
A question type that places the state on a numbered scale you define, from 2 up to 10 levels.
Noul
A question type that answers a plain yes or no question. It returns one number: the probability that the answer is yes.
probability
How likely Jev thinks each possible answer is. For Choice and Score, every option's probability is returned, and they add up to 1.
confidence
A single number from 0 to 1 that says how sure Jev is, kept separate from which answer it picked. Choice and Score return it; Noul does not.
calibration
Whether Jev's stated probabilities match how often it is actually right, checked across many answers, not any single one.
records
The number of API responses an experiment logged and kept, including both successful calls and rejected ones.
input tokens
A count of how much text, the state plus the questions, was sent to Jev in a request. This is what a call is billed on.
experiment id
A label like E01 or E47 marking an experiment's place in the running order across all 108 experiments. It is not a ranking of importance.
group
A set of related experiments testing one broad area, such as whether confidence numbers mean anything, numbered w1 through w7.
verdict
The outcome of one experiment: whether what was measured matched what was expected going in.
falsified
The claim was tested and the measurement came out the way that would prove it wrong.
blocked
The experiment could not run because the real material it needed did not exist yet, and made-up material was not allowed to stand in for it.
hypothesis
The specific claim an experiment sets out to test, written down before the experiment runs.
wrong if
The condition, stated in advance, that would count as proof the hypothesis failed. Every experiment names one before it runs.
108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.
Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.