E79–E92 · GROUP 6 OF 7
Multi-step tasks and rechecking claims
This group tested whether doing well on single steps predicts doing well over a whole task, and re-ran several widely cited results from other projects to see whether they hold up.
Anyone building a loop that asks Jev the same question over and over needs to know whether cheap, isolated single-step tests actually predict how the whole loop performs, since isolated tests are far cheaper to run. This group also re-ran several results repeated across other projects, such as a maze-navigation failure, a game where the model supposedly always picked the same action, and a claim about whether option order matters, because a widely repeated claim that does not hold up can send engineering effort in the wrong direction.
What we ran it on
grid mazes, a synthetic activity log, made-up appliance readings, and a handful of widely cited claims and tasks borrowed from other projects and re-run here
14
experiments
15,512
records
10,373,277
input tokens
6
Confirmed
8
Falsified
Multi-step tasks and rechecking claimsE79–E92E79E80E81E82E83E84E85E86E87E88E89E90E91E92
What we learned
·
The phrasing that scored best alone (39 of 60 correct) finished only 8 of 40 mazes in a real loop, while a phrasing that scored worse alone (34 of 60) finished the most mazes (14 of 40), so single-step scores did not predict loop success.
·
Sending the exact same request 10 times in a row never produced 10 identical answers, in any of six test cases (0 of 10 exact matches every time), so every other result in this group has to be read against that background wobble.
·
A widely cited claim that showing Jev the whole map, instead of a small window, destroys its maze-navigation ability (originally 0 mazes finished out of 128) did not repeat here: the whole-map version finished 75 of 128 mazes, clearly beating a random-move baseline's 0 of 128.
·
Jev's separately reported "confidence" number is genuinely not the same as its top answer's own probability: confidence came in lower in 28.3% of 120 test cases and never once came in higher.
·
Reordering the options in a question, without changing their wording, changed the model's answer in 23 of 24 test combinations, confirming that the order options are listed in is part of the question, not just how the question is displayed.
The experiments
E79
Does per-step competence compose into loop performance?
F
Falsified
Does doing well on single steps predict doing well on a whole task?
We handed it
Three slightly different phrasings of the same "which way do I move" question, each used to navigate 40 different 12x12 grid mazes shown as a small 5x5 window around the current position.
We asked
Each phrasing run as a real step-by-step loop across the 40 mazes (about 3,300 requests total), plus a separate one-shot check of 60 fixed maze positions per phrasing to measure accuracy without letting mistakes carry into the next step.
It returned
The phrasing that scored best on the isolated check, 39 of 60 correct, finished only 8 of 40 mazes as a real loop. The phrasing that scored worst alone, 34 of 60, finished the most mazes, 14 of 40. The middle phrasing scored 35 of 60 alone and also finished only 8 of 40 mazes. The ranking from the isolated check did not match the ranking from the real loop, by more than the study's own margin for error.
What that means
For anyone building a Jev-powered loop, testing single steps in isolation is not a reliable stand-in for testing the whole loop; the two rankings can disagree. This is bad news for anyone hoping to save money by only measuring accuracy on isolated questions instead of running the full task.
E80
Do loop errors compound like independent failures?
F
Falsified
Do mistakes in a loop pile up the way independent errors would?
We handed it
The same maze-navigation loop, but with mazes built so the shortest path is exactly 4, 8, 12, or 16 steps long, using 20, 20, 20, and 10 mazes at each length.
We asked
How often the loop actually reached the goal at each length, compared with a simulated model of a policy that makes the same average rate of per-step mistakes but where each mistake is independent (2,000 simulated runs per length).
It returned
The real loop finished 14 of 20, 12 of 20, 7 of 20, and 1 of 10 mazes at the four lengths. The independent-error model predicted it would finish far less often, only about 21.7%, 28.1%, 19.6%, and 17.2% of the time at those same lengths. The real loop beat that model everywhere, and the gap narrowed by the longest length.
What that means
This is good news read a specific way: a wrong move early in a loop does not doom it the way independent chance failures would, the loop recovers more than a simple error model predicts. That self-correcting margin does shrink as the task gets longer, so it is not a result to rely on for very long loops.
E81
Drift over 60 steps, with and without history in the state
C
Confirmed
Does judgement get worse the longer a step-by-step task runs?
We handed it
A fixed 60-step walk through a maze, always taking the correct move, shown two ways: a plain view of just the current window, and a version that also listed the last 5 moves and how many cells had been visited.
We asked
How often each version agreed with the correct move, split into six groups of 10 steps from the start of the walk to the end, with each version run through the sequence three times (360 scripted requests, plus about 140 more replayed as a real closed loop).
It returned
Agreement stayed flat across the walk in both versions: 17, 9, 12, 15, 15, 15 correct per group of 10 with the plain view, and 9, 9, 12, 15, 15, 15 with the extra history added, no drop bigger than the study's own 0.15 threshold. Adding history did lower the model's reported confidence, 0.449 versus 0.531, without lowering how often it got the right answer. The report's own heading calls this result "not falsified"; the results ledger records the verdict as Confirmed.
What that means
Good news for long loops: judgement does not fade the further into a 60-step sequence the model gets, at least at this length. Piling extra history into what the model sees can make it sound less sure of itself even when it is not actually getting worse, which is worth knowing before treating a confidence drop as a warning sign.
E82
Which number gates a loop: confidence or the winner's probability?
F
Falsified
Which number better flags a bad step, confidence or the top answer's odds?
We handed it
150 newly built maze positions split into three equal groups (only one safe move available, an open area with several good moves, and a spot near the goal), plus every stored answer from the two previous maze experiments, reused for free.
We asked
How well confidence, the top answer's own probability, the gap between the top two options, and a separate yes/no "can you still reach the goal" question each predicted whether the step taken was actually correct, measured as a separating power score (AUC) on both the new set and the reused answers.
It returned
On the deliberately hard new set, confidence separated correct from incorrect steps worse than the top answer's probability did, 0.568 versus 0.597, a gap bigger than the study's allowed 0.02. On the larger reused set of 3,080 past answers, confidence scored 0.520 against the top probability's 0.579 and the top-two gap's 0.535. The separate yes/no reachability question scored 0.494, no better than a coin flip.
What that means
Bad news for common gating advice: for catching a bad individual step, confidence was not the better number here, the top answer's own probability did slightly better. Neither is strong on its own, both stayed close to chance, so a step-level safety gate in a loop may need a different signal altogether.
E83
Self-consistency inside one request: does the same question agree with itself?
F
Falsified
Does asking the same question many times in one request get the same answer?
We handed it
Five short pieces of text, a support ticket, a 12-row table, a short code diff, a shell command, and a chat exchange, each used as the background for a request that asked one question 8 times, a yes/no question 4 times, and a rating question 3 times, all inside the same request.
We asked
Across 20 such requests, whether every copy of the same question, sitting in the same request, gave the exact same answer, tested both with and without 60 extra unrelated questions padded into the same request.
It returned
None of the 20 test cases reached the study's 90% exact-agreement bar. The multiple-choice labels still agreed 18 of 20 times, but the probabilities attached to identical copies of the same question moved by as much as 0.11 within one request, a shift about the same size as the background noise measured separately in E84, so the result reflects that general noise rather than the extra padding questions changing anything.
What that means
This is a caveat more than a clean failure: sending the same question twice in one request is not perfectly reproducible, but the wobble is no bigger than what shows up anyway between separate requests. Builders should not expect bit-identical repeats even within a single request, and should not blame extra questions sharing that request for it.
E84
Is a byte-identical request repeatable, and does the uid buster do anything?
F
Falsified
Does asking the exact same question twice get the exact same answer?
We handed it
Three short pieces of text, a support ticket, a table, and a shell command, each paired with one multiple-choice question and one yes/no question, run first in this group so its numbers could set the baseline for every other experiment here.
We asked
The identical request sent 10 times unchanged, and 10 more times with a random tag added each time, across all three pieces of text, to see whether an unchanged request always gets an unchanged answer and whether the random tag changes anything (60 requests, plus one extra check).
It returned
An identical request never came back with 10 identical answers, in any of the six test cases, 0 of 10 exact matches every time, with 59 different answer bodies out of 60 sends. The size of the variation was the same whether or not the random tag was added, probabilities shifted by 0.06 to 0.12 either way, and the multiple-choice label flipped in 1 of 60 sends.
What that means
Bad news for anyone assuming a fixed request gets a fixed answer, it does not, even without deliberately adding randomness. The random-tag trick some cookbooks use to force variety turned out to be unnecessary, since the same amount of variety showed up on its own. Every other result in this group has to be read against this background wobble.
E85
Two serial requests versus one large request, with an oracle-premise control
C
Confirmed
Is asking everything in one request as good as asking in two steps?
We handed it
60 made-up scenarios, each a 60-row activity log split into 12 blocks of 5 rows, with exactly one row in each scenario breaking a hidden rule that only the test code knew.
We asked
Which row broke the rule, asked four different ways on the same 60 scenarios: one big question over all 60 rows at once, two separate requests (first narrow to the right group of 5, then pick the row), one request that guessed the group and all 12 possible rows at the same time, and one request told the correct group in advance (302 requests total).
It returned
All four ways found the right row every time, 60 of 60, for the single big question, the two-step version, the guess-everything-at-once version, and the version told the answer's group in advance. Picking the right group of 5 was also correct 60 of 60 in both multi-step versions.
What that means
Good news with a real caveat: asking everything in one request worked just as well as splitting the question into two separate requests, at this task's difficulty. The report is upfront that the hidden rule here was easy to spot, so this shows the one-request shortcut can work, not what it costs on a harder task.
E86
Context length: accuracy, confidence, and position inside the shared budget
C
Confirmed
Does accuracy hold up when the input text gets very long?
We handed it
A made-up operations report containing a 40-row table with one row breaking a rule, padded out with filler text to four different total lengths, 1,000, 8,000, 40,000, and 120,000 characters, with the rule-breaking row placed near the start, middle, or end.
We asked
Which row broke the rule, whether the report contained a violation at all, and how clear the evidence was, repeated 5 times for each combination of length and position (60 requests, 180 answers).
It returned
Every one of the 12 length-and-position combinations found the right row all 5 times, 60 of 60 total, even at 120,000 characters, with no drop compared to the 1,000-character case (the study allowed up to a 15-point drop). Confidence stayed at 1.00 in every case, and the separate check for whether a violation existed at all scored 0.98 to 0.99. Token counts scaled as expected with length, from about 940 up to about 18,500, showing nothing was silently cut off.
What that means
Good news for anyone worried about stuffing a lot of text into one request: Jev found a single relevant row just as reliably at 120,000 characters as at 1,000, with no penalty for where in the text it sat. The report cautions this is a best case, an easy-to-spot violation with a small fixed set of candidate rows, not proof that harder reasoning over long documents holds up equally well.
E87
Replication: whole-map navigation against a bounded window (the 0/128)
F
Falsified
Does giving the model the whole map really break its navigation?
We handed it
128 mazes, each run two ways, once shown as the full 40x40 map and once shown as the small 5x5 window used in the rest of this group, with the same question and options both times.
We asked
How many of the 128 mazes each version finished, compared against a random-move baseline and a simple greedy code rule run on the same maps at no extra cost.
It returned
The full-map version finished 75 of 128 mazes and the small-window version finished 80 of 128; a random-move baseline finished 0 of 128 and a simple greedy code rule finished 97 of 128. The widely cited result being tested here was a full-map score of 0 out of 128; that did not happen, the full-map version clearly beat random rather than matching it.
What that means
This overturns a well-known negative claim from elsewhere: showing Jev the whole map, instead of just a small window, did not wreck its performance the way the original report claimed, at least at this smaller 40x40 scale. The report notes the maze size and precision differences here mean this narrows the original claim rather than erasing it entirely.
E88
Replication: the structured-criteria null (12/15 vs 12/15)
F
Falsified
Does structuring the question's criteria actually improve answers?
We handed it
15 made-up scenarios about a washing machine, its power draw, whether the door is open, minutes since the last cycle, and whether someone is in the room, built so the correct answer was known in advance, run 5 times each under four question designs.
We asked
Whether the load was finished but still in the machine, asked using plain readings alone, readings plus a structured yes/no criteria field, readings plus the rule written directly into the question text, and the comparison already worked out and stated in the input (300 requests total).
It returned
Adding the structured criteria field changed almost nothing, 14 or 15 of 15 correct in every repeat, matching plain readings alone. Writing the actual rule into the question text raised the score by 10 more correct answers across the repeats. Handing the model the answer pre-computed did not help at all, only 5 of 15 correct. The plain-readings version was heavily biased toward answering yes.
What that means
This is a genuinely useful negative result for question design: a structured criteria field does not do the work people hope it does, but writing the actual decision rule into the question text does. Oddly, doing the comparison in advance for the model did not help either, so simply handing over the answer is not a shortcut.
E89
Replication: visual policy collapse in a loop (38/38 Hold)
C
Confirmed
Does a text-based version of a game avoid locking onto one action?
We handed it
228 decisions across six practice runs of a paddle-and-ball game, shown as a text grid rather than an image, asking the model to choose Hold, Left, or Right, plus a second version asking which of five lanes the ball would land in.
We asked
Whether one action dominated almost every decision, the way a widely cited image-based version reportedly always chose "Hold", and how often the five-lane version guessed correctly.
It returned
The three-choice version did not collapse onto one action: Left was chosen 84 times, Hold 57 times, and Right 87 times out of 228 decisions, no action reaching even 40%, well under the study's 90% collapse threshold. The five-lane version got 125 of 228 decisions right, well above the 1-in-5 rate expected from guessing.
What that means
Good news with a real caveat: when the same kind of decision is shown as text instead of an image, the model does not lock onto a single answer the way the original image-based report described. The report is careful to say this clears the text version only, since the original collapse may be specific to how images are processed, not to the decision itself.
E90
Replication: the routing claim (and @bggoranoff's non-reproduction)
F
Falsified
Does picking the right tool for a task beat simple keyword matching?
We handed it
Three short task descriptions taken word-for-word from a demo video, repeated 20 times each, plus 30 new task descriptions built against a rule written in code, repeated 5 times each, all asking the model to pick which of six systems should handle the task.
We asked
Whether the model's picks agreed with a code-defined correct answer more often than a simple keyword-matching rule would, and whether its picks stayed the same across repeats (210 requests total).
It returned
On the three original tasks, the model gave the same pick at least 4 of 5 times in all 20 repeats of each, fully stable every time. On the 30 new tasks, the model agreed with the code-defined answer on 29 of 30, but a simple keyword-matching rule also got 29 of 30, missing the same one task; always picking one system got only 9 of 30 right. All 30 tasks were stable across repeats.
What that means
This is the middle outcome, neither a clean win nor a clean loss: the model's picks were stable and correct, but a simple keyword rule did just as well, so this particular task may be simple enough that it does not need a model at all. That is consistent with the demo looking good while an outside tester found it unimpressive on a harder benchmark, without either being wrong.
E91
Replication: is real Jev's option order exactly an input, or exactly not?
C
Confirmed
Does the order in which options are listed change the model's answer?
We handed it
Five kinds of questions, multiple-choice lists of 3, 6, 12, and 60 options plus a five-level rating question, each asked about six pieces of text under several conditions, unchanged twice, options rotated or reversed, a rating scale reversed, one option's wording changed, and options renamed to generic labels (211 requests total).
We asked
Whether reordering the options, without changing their wording, ever changed the model's answer, using the two sanity-check conditions to confirm the test setup could detect a real effect at all.
It returned
Reordering the options changed the answer in 23 of 24 test combinations, in one case moving a five-level rating from 0.34 to 3.50, and one case flipped which option won outright. Both sanity checks worked as expected, reversing the rating scale changed the score, and changing one option's wording changed the answer in all 30 cases, confirming the setup was sensitive enough to catch real effects.
What that means
This settles an open disagreement in favor of order mattering: for the real system tested here, the order options are listed in is part of the question, not just a presentation detail. Anyone generating multiple-choice questions from data whose order can shift should treat a reorder as changing the question itself.
E92
Replication: confidence is not max(probabilities)
C
Confirmed
Is the model's confidence just its top answer's probability in disguise?
We handed it
120 made-up support-ticket questions across six option-list sizes, 2, 3, 5, 12, 60, and 255 choices, built to range from obvious answers to genuine toss-ups to cases with a clear top pick and a strong runner-up.
We asked
Whether the model's separately reported confidence number ever came out lower, or higher, than the probability of its own top-picked answer, across all 120 answers.
It returned
Confidence came in below the top answer's own probability in 34 of 120 cases, 28.3%, well past the study's 10% bar, and never once came in above it; the two numbers were equal in 86 of 120 cases. The two numbers still tracked each other closely overall, a correlation of 0.874.
What that means
Good news for a specific piece of engineering advice: confidence really is a different number from the top answer's own probability, so gating a decision on confidence is not just reading probability under another name. Because the two track each other closely but are not identical, a threshold tuned on one may need recalibrating before it is applied to the other.
THE NOTES AS WRITTEN
108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.
Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.