E63–E78 · GROUP 5 OF 7
Scoring, curating, and finding text
This group tested how to build a good rating question, whether a curation decision should discard its distribution, and how well Jev can point at one piece of text inside a longer document.
Anyone building a real pipeline with Jev has to pick between a rating, a multiple-choice question, or a plain yes/no question, and then decide what to do with the numbers that come back. Getting that choice wrong either wastes requests on a design that adds nothing, or throws away a signal a curation or search pipeline needed. Several of these tests also try to run a real curation pipeline end to end and could not, which is itself a finding about what such a pipeline needs before it can be trusted.
What we ran it on
wine tasting notes with a known critic score, agent transcripts and reasoning traces, and long scripted documents with one planted defect at a known line
16
experiments
4,588
records
7,106,654
input tokens
7
Confirmed
3
Falsified
6
Blocked
Scoring, curating, and finding textE63–E78E63E64E65E66E67E68E69E70E71E72E73E74E75E76E77E78
What we learned
·
A five-level rating did not beat asking the same five descriptions as a plain multiple-choice question: the gap in rank correlation was 0.0056, and the report's own interval for that gap runs from -0.0090 to +0.0230, which includes zero.
·
Rating scales top out early: rank correlation against the critic score rose to 0.7968 at 5 levels and did not improve at 7 levels (0.7966) or 10 levels (0.7951).
·
The text a rating question echoes back to describe its own levels never changed a single word: 0 drifts across 640 checked answers.
·
Putting many rows into one request broke consistency fast: agreement with the same row asked alone fell to 0.718 at just two rows per request and down to 0.036 at 64 rows per request.
·
6 of the 16 experiments planned for this group never ran at all, because the real transcripts, turns, traces, or documents they needed were never built.
The experiments
E63
Which answer shape for one ordered judgement
F
Falsified
Does a 5-level rating beat multiple-choice for judging wine quality?
We handed it
200 wine tasting notes, averaging about 242.5 characters, with a known critic score from 80 to 98.
We asked
One request per note asking the wine to be judged four ways: a 5-level rating, a multiple-choice question with the same five descriptions, the same choices numbered 1 to 5, and a plain yes/no question, each compared to the critic score. 200 requests.
It returned
The rating's rank correlation was 0.7951, close to the plain multiple-choice question's 0.7895 (a difference of +0.0056 with an interval of -0.0090 to +0.0230, which includes zero). It did beat the numbered choices (0.7849) and the yes/no question (0.7333).
What that means
For this kind of judgment, how well the options are worded matters more than whether you use a rating or a multiple-choice question. That is a caveat against assuming a rating is always the right shape.
E64
Level count: 2, 3, 5, 7, 10 on one fixed judgement
C
Confirmed
How many rating levels give the best accuracy: 2, 3, 5, 7, or 10?
We handed it
200 of the same wine tasting notes, averaging about 242.5 characters, with the critic score as ground truth.
We asked
One request per note asking the same rating built at five different level counts (2, 3, 5, 7, 10), each compared to the critic score. 200 requests.
It returned
Rank correlation went 0.7714 (2 levels), 0.7604 (3), 0.7968 (5), 0.7966 (7), and 0.7951 (10). At 10 levels, the fraction of answers landing between two levels rather than on one rose to 0.170.
What that means
Five levels is already enough for a judgment like this; adding more levels does not buy more accuracy, even though the model does start interpolating between levels at 10. That is good news for anyone tempted to build an overly fine scale.
E65
Legend: echoed or generated, and whether the level construction changes the answer
C
Confirmed
Does the rating's returned legend text ever drift from what was submitted?
We handed it
160 wine tasting notes, averaging about 242.5 characters, each rated with the same five levels written four different ways.
We asked
One request per note comparing the returned level-description text word for word against what was submitted, across four ways of writing the levels, and comparing rank correlation between them. 160 requests.
It returned
Across 640 checked answers, the returned level descriptions matched the submitted wording exactly every time: 0 drifts. Adding matched examples to the descriptions did not raise accuracy: plain wording scored 0.8135 rank correlation against 0.7828 with matched examples and 0.7830 with doubled examples.
What that means
The text a rating question echoes back describing its levels is just a copy of what you sent, not new output, so code that treats it as the model's own wording is wrong. It is also good news that spending effort on example-based level descriptions does not pay off here.
E66
Is the Score distribution informational or decorative
F
Falsified
Does the shape of a rating's probabilities predict better than its mean alone?
We handed it
200 wine tasting notes rated on a 10-level scale, split into 100 notes to fit a model and 100 to test it.
We asked
One request per note for a 10-level rating, then in code six ways of reading the same answer (mean; mean and spread; the full 10-number breakdown; that breakdown with its columns randomly shuffled; mean and confidence; mean and nearest level) were compared against the critic score. 200 requests.
It returned
Using only the mean gave the best error, RMSE 1.9036. The full 10-number breakdown did worse, at 2.0091, and a shuffled version of that same breakdown scored almost identically, at 2.0036. The paired difference between the full breakdown and the mean was +0.1055, with an interval of -0.111 to +0.303 that includes zero.
What that means
Keeping only the mean and throwing away the rest of the breakdown is not a mistake for predicting an outcome like this one. That is good news for anyone who only reads the mean, and it means the extra numbers are unused rather than lost.
E67
Score stability across a perturbation ladder
C
Confirmed
How much does a rating change on its own, with no change to the input?
We handed it
40 wine tasting notes, each sent four ways: unchanged, with a fresh session id added, cosmetically rewrapped, and with one field renamed.
We asked
One request per version per note, five repeats of each, asking a rating question and measuring how much the returned probabilities moved between versions. 800 requests.
It returned
No version tested was calmer than sending the exact same request twice: repeating an identical request moved the answer by about as much as the cosmetic rewording did. A further set of rungs testing paraphrased wording and a change of tone was not run in this pass.
What that means
Because even an unchanged request wobbles about as much as small formatting changes do, a gap seen on a single run is not trustworthy without checking it against a repeat first. It is good news that adding a fresh session id changed nothing, so caching by session key is safe.
E68
Holistic versus decomposed: is the gain from the split or from the fitting
C
Confirmed
Does splitting a rating into parts help, or is it just fitted weights?
We handed it
200 wine tasting notes, each rated once as a single 10-level judgment and again as five separate ratings (fruit, structure, criticism, oak, enthusiasm).
We asked
One request per note holding the single rating and the five separate ratings together; in code the five were combined with equal weights and again with weights fitted on half the notes, then compared to the critic score on the other half. 200 requests (330 log records once retries are counted).
It returned
The fitted combination reached rank correlation 0.7232, beating the equal-weight combination's 0.3923 by +0.3309 (interval +0.1042 to +0.5233). But it did not beat the single overall rating's 0.7948: the difference was -0.0355, with an interval of -0.1399 to +0.0618.
What that means
The advantage that splitting a judgment into parts is supposed to have comes from fitting weights on labelled data, not from the split itself. Without labels to fit, one well-worded rating is just as good, which saves asking several extra questions.
E69
Score as policy or Score as sensor: mean, nearest level, and the gate
C
Confirmed
Does deciding by a rating's mean differ from deciding by its nearest level?
We handed it
No new material. This re-read the 200 wine-note ratings and yes/no answers already collected in E63.
We asked
In code, four decision rules were compared on the same 100 held-out notes: a fitted cutoff on the mean, a cutoff on the nearest level, that same cutoff plus a confidence gate, and a plain yes/no question. Zero new requests.
It returned
The mean-based rule and the nearest-level rule disagreed on only 1 of 100 rows. Adding a confidence gate changed nothing (precision and recall stayed at 0.675/0.750 either way). The plain yes/no question was weaker overall: precision 0.398, recall 0.972.
What that means
Whether you read a rating by its mean or round it to the nearest level barely matters here, so there was no real gap for a confidence gate to close. That is good news for anyone worried the two approaches would disagree.
E70
Keep/drop precision and recall at volume, and how much state the question needs
B
Blocked
Does a keep/drop decision improve when the model sees more surrounding text?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: the design needed 12 real agent transcripts of 40 rows each plus 160 hand-labelled rows, and that input set was never built.
What that means
This would have settled whether a curation model needs the surrounding conversation to judge one row well, or whether the row alone is enough. Until real transcripts and labels exist, that question stays open.
E71
The anti-sycophancy Noul: two bars, a paraphrase, and a regex that might do
B
Blocked
Can a simple code check replace a yes/no question for spotting sycophancy?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: the design needed 240 real assistant turns, 120 with a planted problem and 120 natural, with human labels, and that input set was never built.
What that means
This would have shown whether a cheap code check catches most flattery and fake disclaimers on its own, saving the cost of asking the model at all. Without the real turns, that saving is unproven.
E72
The circular-reasoning Noul: one vote on a long trace, and the per-step companion
B
Blocked
Does one yes/no question miss circular reasoning buried in a long trace?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: the design needed 200 reasoning traces with circular steps planted at known positions plus human verification, and that input set was never built.
What that means
This would have shown whether one question over a whole trace scores a single bad step the same as a trace that is entirely circular, and whether checking step by step catches what that one question misses.
E73
Batch-size sweep: does batching rows into one request change the answers
F
Falsified
Does putting many rows in one request change the answers you get?
We handed it
500 short rows of text, asked at batch sizes of 1, 2, 4, 8, 16, 32, and 64 rows per request, with the wording kept identical across sizes.
We asked
One request per batch, comparing every batched answer back to the same row asked alone. 990 requests in total.
It returned
Agreement with the single-row answer fell from 0.718 at batch size 2 to 0.258 at batch size 8 to 0.036 at batch size 64. Over the same range, measured throughput rose from 1.18 rows per second at batch size 1 to 51.80 rows per second at batch size 64.
What that means
Batching rows together to go faster is not free: it changes the answers you get, so the speed gain comes at a real accuracy cost. That is bad news for anyone planning to batch purely for throughput.
E74
Threshold or rank at a fixed budget
B
Blocked
At a fixed budget, does ranking beat a simple threshold for keeping rows?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: it is a pure re-read of E70's stored answers, and E70 itself never ran.
What that means
This would have told a builder whether sorting rows by score and keeping the top ones beats a plain yes/no cutoff at the same budget. It stays untested until the data it depends on exists.
E75
Span extraction: the roster bound, the escape hatch, and the verbatim guarantee
B
Blocked
When the right text span is missing from the options, does the model say so?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: the design needed 300 templated documents plus 100 hand-marked real documents with known spans, and that input set was never built.
What that means
This would have shown whether a 'none of these' option actually gets picked when the right answer is missing, or whether the model guesses a nearby wrong one instead, an important safety check before trusting span extraction.
E76
Roster scale and the 255 cap: a wall or a slope
C
Confirmed
Does accuracy fall as the list of choices grows toward the 255-option limit?
We handed it
40 scripted 250-line documents (a log, a diff, and a config file), each with one planted defect at a known line.
We asked
One request per document offering the defect location among lists of 10, 60, 250, and 260 options, plus a version asking a separate yes/no question per option; three repeats each, about 603 requests in total.
It returned
Accuracy held at 1.000 (120 of 120) at 10, 60, and 250 options. Asking for 260 options was rejected outright with an HTTP 400 error reading "Too many choices. Must have at most 255 choices." The separate-yes/no-per-option version scored 0.950 (114 of 120).
What that means
Below the hard 255-option cap, a long list of choices costs nothing here: it is a wall you hit all at once, not a slope you slide down. The defects used were easy to spot, though, so this does not rule out a real accuracy slope on harder cases.
E77
Narrow-then-read or speculative fan-out: the topology of localisation
C
Confirmed
Does asking everything at once match a two-step narrow-then-read approach?
We handed it
60 scripted 120-line documents split into 6 regions of 20 lines, each with one planted defect at a known line.
We asked
One-shot, two-step, and ask-everything-at-once approaches to finding the defect line were compared per document, three repeats each. 720 requests in total.
It returned
Every approach found the defect every time: 180 of 180 for the one-shot approach, the two-step approach, and the ask-everything-at-once approach alike. The report's own heading calls this inconclusive because no approach ever made a mistake, but the ask-everything-at-once approach used about half as many requests per document as the two-step approach (1 versus 2).
What that means
With defects this easy to spot, there was no room for the two-step approach to show an advantage; asking everything at once was at least as accurate and used fewer requests. That needs testing on harder, less obvious defects before it can be trusted generally.
E78
The out-of-roster case: does the escape hatch fire, and does confidence cover its misses
B
Blocked
When nothing on a list matches, does the escape-hatch option catch it?
We handed it
Nothing. This experiment never ran.
We asked
No requests were sent.
It returned
It did not run: the design needed 300 items across three severities plus about ninety minutes of human near-miss checking, and that input set was never built.
What that means
This would have shown whether a 'no match' option and a low-confidence score each catch different missed cases, or whether one already covers the other. That matters before relying on either alone as a safety net.
THE NOTES AS WRITTEN
108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.
Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.