E17–E28 · GROUP 2 OF 7
The confidence number, tested
This group tested what Jev's confidence number is good for: telling right answers from wrong ones, warning when an answer might flip, and deciding when a gate should let an answer through on its own.
Every system built on Jev has to decide, at some point, whether to trust an answer without a person checking it, and confidence is the one number sold for exactly that job. Nobody had measured whether it actually predicts correctness, whether it says anything the model's plain top probability does not already say, or whether a cutoff picked on one batch of answers holds up on the next batch. Twelve tests ran against real logged answers to find out, several of them by replaying stored data rather than asking new questions, which is why some of them cost nothing at all.
What we ran it on
bank customer-service messages with known category labels, plus one attempted set of SEC company filings that could not be built in time
12
experiments
5,875
records
8,215,046
input tokens
7
Confirmed
4
Falsified
1
Mixed
The confidence number, testedE17–E28E17E18E19E20E21E22E23E24E25E26E27E28
What we learned
·
Confidence tracks the model's own top answer probability closely (Spearman correlation 0.9646), but the two numbers disagree on 93 of 360 answers (25.8%), so they are not the same statistic.
·
On a straightforward labelled test, the bottom tenth of answers by confidence were right only 42.5% of the time, while the top tenths were right up to 100% of the time, a 52.5-point spread.
·
Comparing five different ways to measure uncertainty side by side, the best one beat Jev's own confidence by only 0.31 percentage points at a fixed 80% acceptance rate, far short of the 5-point gap set as the bar for calling any of them the clear winner.
·
Answers with confidence above 0.9 never changed their answer on a repeat, 0 flips out of 67 items asked six times each, so high confidence is a genuine sign an answer will hold up.
·
The certainty numbers from three different question types disagreed badly on the same material, with one pair correlating at -0.517, so a cutoff tuned for one question type cannot be reused for another.
The experiments
E17
Does confidence separate the answers that are right from the answers that are wrong?
C
Confirmed
Does a high confidence score mean an answer is more likely correct?
We handed it
Two sets of text: 400 short customer-service messages from a public banking dataset with known category labels, and an attempt to build 240 SEC company filings for a second set.
We asked
One multiple-choice question per item, picking the matching category, sent as 401 separate requests. Answers were then grouped into ten bins by their confidence score to see how accuracy changed bin by bin.
It returned
On the 400 customer messages (332 correct, 83.0% overall), accuracy rose from 42.5% in the lowest confidence tenth to 100% in the seventh and eighth tenths, a 52.5-point spread, climbing in 7 of 9 steps. The SEC-filing set never ran: only 12 of the planned 240 filings could be built in the time available, so no result exists for it.
What that means
Good news for the customer-message case: confidence really does track correctness there, and by a wide margin. But the second dataset was never tested, so it is still unproven whether this holds on every kind of material.
E18
Is confidence a second axis, or a restatement of the winner's probability?
C
Confirmed
Is confidence a repeat of the model's top probability, or new information?
We handed it
The 360 logged answers from the previous test, each asked again with 3, 10, and 40 possible answers instead of a fixed number, to see if a bigger answer list changed the picture.
We asked
The same multiple-choice question at three different answer-list sizes, 361 requests total, then a statistical comparison between the confidence score and the top answer's probability on every answer.
It returned
Confidence and the top probability moved together closely (Spearman 0.9646, Pearson 0.9886), but confidence came in lower than the top probability on 93 of 360 answers (25.8%), far above the 2% gap that would have made them the same statistic. The gap grew with the size of the answer list: 16.7% of answers diverged at 3 options, rising to 40.0% at 40 options. Twenty pairs of answers had nearly identical top probabilities (within 0.02) but confidence scores at least 0.15 apart.
What that means
Good news for anyone reading confidence as extra information: it is not just a repackaged version of the top probability. The catch is that the gap grows as the answer list gets longer, so systems with large option lists will see this divergence the most.
E19
Which statistic should the gate read? A five-way bake-off at fixed coverage.
F
Falsified
Out of five ways to measure how sure an answer is, which works best?
We handed it
The same 640 logged answers from the earlier tests, reused at no extra cost with no new text sent to the model.
We asked
Four ways of measuring uncertainty on those stored answers, Jev's confidence, the top probability, the gap between the top two probabilities, and a spread-based measure, compared at a fixed 80% acceptance rate. A planned fifth measure, combining several sub-answers with a worst-case rule, could not be built in the time available.
It returned
At 80% coverage, accuracy on accepted answers was 90.31% using confidence, 90.31% using the top probability, and 90.62% using the two remaining measures, a gap of only 0.31 percentage points, far short of the 5-point difference needed to call any measure the winner.
What that means
Bad news for the hypothesis, but a useful finding: no single measure of uncertainty, including Jev's own confidence, clearly beats the others here. That means effort is better spent picking the right cutoff than picking which number to threshold on.
E20
Where do threshold bands start paying? Precision, coverage and the price of abstaining.
M
Mixed
Where should a confidence cutoff be set, and does it pay off?
We handed it
The 400 logged customer-message answers from the first test, split into two halves of 200: one to pick a cutoff, one to check it.
We asked
A sweep of confidence cutoffs from 0 to 1, applied first to the 200-item fitting half and then checked against the other 200-item held-out half. No new requests were sent; this reused stored answers.
It returned
The best cutoff on the fitting half (0.925) raised accuracy on accepted answers from 81.50% to 92.67%. On the held-out half the same cutoff still raised accuracy, from 84.50% to 93.28%, an 8.8-point gain, but it only accepted 67% of answers, a 33% coverage loss, missing the 25% target. The best cutoff under an even cost trade-off (0.850) matched a value already used elsewhere as a fixed rule.
What that means
Mixed news: raising the confidence bar genuinely improves accuracy, so cutoffs are worth using. But the exact cutoff that looked best on one batch of 200 answers did not hold its coverage promise on a second batch the same size, so a cutoff picked on too little data can overpromise on how many answers it will let through.
E21
Confidence or a companion Noul: which is the better abstain signal?
C
Confirmed
Is confidence a better warning sign than adding a yes/no check question?
We handed it
300 customer messages: 120 that matched one of the model's known categories, 100 from categories outside its list, and 80 written to look like a match without actually being one.
We asked
One request per message carrying three questions: the multiple-choice category pick, and a yes/no question (asked two different ways) checking whether the text fits any listed category. 300 requests total, compared at fixed acceptance rates of 70%, 80% and 90%.
It returned
Confidence separated good matches from bad ones best (accuracy-ranking score 0.777), followed by the probability of a catch-all "other" option (0.750). Both yes/no checks did no better than a coin flip or worse (0.382 and 0.492). Confidence caught the tricky look-alike cases best (0.840 on that group), while the catch-all option caught the plainly off-topic cases best (0.868).
What that means
Good news for anyone weighing whether to add an extra check question: confidence alone already does the job better here, so the extra question is not needed for this kind of material. The two signals do specialize differently, though, so a system facing both kinds of bad input may still want both.
E22
Does confidence survive paraphrase?
F
Falsified
If a question is reworded without changing its meaning, does confidence hold steady?
We handed it
120 items, each asked six ways: the original wording, four reworded versions of the instructions, and one version with the answer order reversed.
We asked
The same judgment asked with six different phrasings per item, 720 requests total, checking whether the chosen answer changed and by how much confidence moved.
It returned
The chosen answer changed on only 6 of 120 items (5.0%) across the four reworded versions. But confidence itself moved by 0.10 or more on 21 of 120 items (17.5%), below the 25% the test needed to count as a real problem. Reversing the answer order changed the answer even less than rewording did, 3 of 120 items (2.5%).
What that means
Good news in practice: a confidence cutoff tuned on one phrasing of a question kept working when the phrasing changed, on this material. The formal test came back "falsified" only because confidence moved less than the hypothesis expected, which is a reassuring result, not a real problem.
E23
Does low confidence predict that the answer will flip?
C
Confirmed
Does low confidence warn you that an answer might flip on a repeat?
We handed it
148 items from earlier tests, each asked six times in a row with a fresh random marker each time, so each repeat counts as an independent try.
We asked
The same question asked six times per item, 888 requests total, then answers grouped by the first answer's confidence to see how often a repeat changed the answer.
It returned
Items whose first answer had confidence below 0.5 changed on a repeat 6 of 11 times (54.5%). Items between 0.5 and 0.9 changed 6 of 70 times (8.6%). Items above 0.9 confidence never changed, 0 of 67. The drop was steady across all three bands.
What that means
Good news: low confidence is a real, useful warning that an answer is unstable, and it costs nothing extra to check since it comes with every answer already. Above 0.9 confidence, an answer can be trusted to repeat; below 0.5, expect it to change more than half the time.
E24
Near-uniform versus peaked: can the shape be manufactured, and does confidence follow it?
C
Confirmed
Can how a question is written push confidence up or down?
We handed it
40 short texts, each turned into a multiple-choice question six different ways (overlapping options, distinct options, one option quoting the text, and so on), plus 30 more texts asked as a rating question in four labelling styles.
We asked
The 40 texts run through six versions of the multiple-choice question (240 requests) and the 30 texts through four versions of the rating question (120 requests), 360 requests total, comparing average confidence across versions.
It returned
Average confidence ranged from 0.442 on the weakest multiple-choice version up to 0.912 on the strongest, but it did not climb in a straight line. It rose, dipped, and rose again. Making an answer option quote the text back lowered confidence compared to a plain version, in both places it was tried. On the rating question, confidence rose from 0.409 with plain numbered levels to 0.634 with worded levels to 0.814 with an unrelated example attached, but a matching example scored lower, 0.768, than an unrelated one.
What that means
Good news for question design: wording clearly changes confidence, so a question whose confidence never moves between versions is probably not doing useful work. The catch is that two tricks meant to raise confidence, quoting the text and adding a matching example, both backfired here, so the specific technique matters more than the intuition behind it.
E25
Is confidence comparable across roster sizes?
C
Confirmed
Does confidence mean the same thing with 3 answer options as with 40?
We handed it
150 items from a labelled customer-message corpus, each asked with progressively longer answer lists of 3, 10, and 40 options, the extra options added from the same category taxonomy.
We asked
The same judgment on the same 150 items asked at three answer-list sizes, 450 requests total, comparing typical confidence and actual accuracy at each size.
It returned
Typical confidence barely moved across list sizes (1.000, 1.000, 0.990 at 3, 10, and 40 options, a spread of 0.010, well under the 0.08 limit set for this test), while accuracy fell from 95.3% to 90.0% to 81.3% over the same range.
What that means
Good news for systems that mix questions with different numbers of answer options: one confidence cutoff can be reused across list sizes here without adjusting it. The flip side is that confidence will not warn you when a longer list is making the model less accurate; only a labelled check catches that.
E26
Can a calibration curve be built from stored answers at zero API cost, and does it survive a held-out replay?
F
Falsified
Can a confidence cutoff be picked from old answers instead of new tests?
We handed it
No new material. This reused the 400 logged customer-message answers, split by collection order into the earliest 240 and the latest 160.
We asked
A prediction curve fit on the earliest 240 answers, then checked against the latest 160 to see how well it predicted their actual accuracy. Zero new requests were sent.
It returned
The fitted curve's predictions missed the held-out group's actual accuracy by as much as 27.8 percentage points in one confidence band, far above the 7-point target the test set. Several confidence bands in the held-out group had very few answers to check against, five to twenty-two items, some sitting entirely at confidence 1.0 with no wrong answers to learn from.
What that means
Bad news, but a useful one: fitting a cutoff from old stored answers, at zero extra cost, did not carry over reliably here. The problem traced to too little data spread across the confidence range, not to the replay approach itself, which ran correctly and cost nothing.
E27
Do the three primitives agree about the same uncertainty?
F
Falsified
If you ask the same judgment three ways, do their certainty scores agree?
We handed it
100 pieces of text, each asked as three separate questions in a single request: a multiple-choice pick, a four-level rating, and a yes/no check for a clear category fit.
We asked
Three certainty-like numbers per item compared for agreement across all 100 items, 102 requests total.
It returned
The certainty scores barely agreed with each other: the multiple-choice and rating confidences correlated at only 0.180, the multiple-choice and yes/no scores at -0.164, and the rating and yes/no scores at -0.517, meaning that pair moved in opposite directions. Switching which question type a cutoff was applied to changed which answers got accepted on 35 of 100 items.
What that means
Bad news for any system that mixes question types: a confidence threshold tuned on one kind of question, say multiple-choice, cannot be assumed to work on a rating or yes/no question built from the same material. Certainty is not one property the model reports the same way across question shapes.
E28
Does a confidence gate change what a loop actually does?
C
Confirmed
Does a confidence-based pause button actually change what an automated loop does?
We handed it
A record of 300 consecutive decision steps from a text-based loop, each a multiple-choice pick logged with its full confidence and probability.
We asked
Three replayed strategies run offline over the same recorded steps, act on every answer regardless of confidence, pause and repeat the last action below a cutoff, or pause and take a fixed safe action below a cutoff, swept across every possible cutoff. 300 requests built the log; the replay itself cost nothing.
It returned
At a cutoff of 0.8, 19.3% of steps (58 of 300) changed compared to acting on every answer, clearing the 15% bar set for this test; at 0.9 the change grew to 27.3%. Pausing and repeating the last action raised the loop's own accuracy from 83.0% to 87.3% at the 0.9 cutoff, while pausing and taking a fixed safe action instead made things worse, dropping accuracy to 67.7%.
What that means
Good news: a confidence gate does not just look safe, it measurably changes what the loop does and can improve the outcome, at least when the fallback is repeating the last action rather than a generic safe default. The loop record was built fresh for this test, so this shows the method works, not that the exact numbers repeat elsewhere.
THE NOTES AS WRITTEN
108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.
Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.