F

151 short trigger phrases (single words or short terms) that the router's own code already uses to name each of 24 destinations, sent one phrase at a time with no other context.

A single destination pick over all 24 destinations, plus one yes/no check per destination, repeated 3 times per phrase, 454 requests in all.

Overall accuracy was 92.7% (420 of 453), below the 95% bar the hypothesis set. Phrases that shared a word with another destination still scored 88.9%, only 4.3 points below phrases that named their destination directly, so the router is not simply matching substrings.

Bad news for the router's own regression test: it fails the bar it set for itself. Most of the misses were phrases written before newer destinations like documents and finance records existed, so the label list may need updating rather than the model.

I

Sixteen real, already-labelled conversation turns, each tried with three versions of surrounding context: the question alone, the question plus the prior answer, and both plus a flag for whether the session already has data.

27 yes/no destination checks plus one main pick per turn, run for each of the 3 context versions with 3 repeats, 144 requests total.

Every version reached 100% recall (48 of 48) because the check accepted every correct destination for every turn. But only 3 to 4 of the 16 turns matched the exact right set of destinations with nothing extra added, and the extra context made no measurable difference, a 0.0 point change.

Caveat: the test set was too small and too easy (16 turns, no case needing more than one destination) to prove the extra context helps. The real finding is that the router tends to open extra destinations beyond the correct one, which a recall number alone hides.

M

40 real trigger sentences pulled from a live set of 58 published skill descriptions, tested against destination lists that grew from 2 up to all 58 destinations.

A single pick among the destinations, at list sizes 2, 4, 8, 16, 32, and 58, 3 repeats per size, 720 requests total.

Accuracy stayed at or near 100% through 32 options (100% at 2, 4, and 8; 97.5% at 16; 98.3% at 32), then dropped to 90.0% at 58 options. The drop at 58 traced to four sentences where a broader, newer destination absorbed the answer.

Good news up to a point: a list of 32 destinations is as easy to route as one of 2. The original design wanted to test up to 255 options and the real data only reached 58, so where a harder cliff sits is still open.

C

The same 16 labelled turns, each sent against the same 24-destination list written four ways: name only, name plus one summary line, name plus that line plus a mix-up warning, and the full four-part description.

One main destination pick per turn per version, 3 repeats, 192 requests total.

The full description beat name-only by 18.8 points (85.4% versus 66.7% correct). But adding the mix-up warning on top of the summary line made things worse: it lost 14.6 points compared to the summary line alone (79.2% versus 93.8%), which was the best-performing version of the four.

Good news that descriptions matter more than bare names. But the specific warning sentence meant to prevent confusion between similar destinations backfired here, so its wording needs testing before it is trusted.

F

The same 16 labelled turns as before, tested against the full four-part description with one part removed at a time: the summary line, the mix-up warning, or the examples.

One main destination pick per turn per version, 3 repeats, 192 requests total.

Removing the mix-up warning improved accuracy by 8.3 points (87.5% versus 79.2% with everything included), the opposite of what the hypothesis expected. The planned check of whether vague wording changes this result was never made because that extra labelling step was skipped.

Bad news for the mix-up warning sentence specifically: cutting it helped rather than hurt in this test. That points to a wording problem worth fixing rather than a case for adding more descriptive text.

B

Nothing was sent to the model. The test needs 30 hand-written examples of requests with no correct destination, and those examples were never written.

No requests were sent; the design compares two abstain methods but needs the missing examples first.

The experiment did not run. It needs a set of 30 written unanswerable-request examples that does not exist, about 90 minutes of writing work.

Open question. Until those 30 examples exist, nobody knows whether a dedicated exists-check or a plain none-of-these option catches missing-answer cases better.

B

Nothing was sent to the model. The test needs about 40 real turns that genuinely require two destinations at once, and the available labelled set has only one such turn.

No requests were sent; there were not enough qualifying turns to build the comparison.

The experiment did not run because the multi-destination turn set the design calls for does not exist.

Open question. Until a real set of multi-destination requests is built, nobody knows whether the current single-pick design silently drops a second, needed destination.

B

Nothing was sent to the model. The test needs sentences that sit right on the boundary between two easily confused destinations.

No requests were sent; a mechanical search for boundary sentences came up empty.

The experiment did not run. A mechanical check found that none of the 151 existing trigger phrases sit on the boundary between the four confusable destination pairs the design targeted.

Open question. This confusion-ladder test still needs a purpose-built set of boundary sentences before it can run.

F

Real turns tried against three list sizes: 6 named models, 24 full destination descriptions, and 58 live skill descriptions, each with the same options shuffled into different orders.

One main pick per turn per ordering (5 shuffles plus the correct answer pinned first and pinned last), 286 requests total.

At 6 options, the winning answer flipped between orderings 14.1% of the time, above the 5% bar. At 24 options it flipped 6.2% of the time, and at 58 options it flipped 35.0% of the time. Pinning the correct answer first versus last made no difference (0 of 13 cases disagreed), so this is a genuine order effect, not a position bias.

Bad news: the order options are listed in changes the answer far more than expected, and worse with more options. A stored answer, or a comparison between two versions of a list with different option order, cannot be trusted at face value.

C

16 real turns split into ones sitting right at a decision threshold and ones further from it, each asked 10 times with byte-identical wording.

One main pick and a set of yes/no checks per turn, repeated 10 times unchanged, 160 requests total.

Which destinations got flagged jittered near the threshold, 6 of 11 boundary turns flagged inconsistently at least 10% of the time, but the top answer itself never changed across any of the 160 repeats, 0 flips.

Mixed but mostly reassuring: the headline pick is stable even when you ask twice, so a single run can be trusted for the main decision. Secondary flags right at a threshold are noisier and need more than one run to read.

B

Nothing was sent to the model. The test needs a destination list that can have options added or removed freely with known correct answers, and the only such list is the fixed, real 27-destination router.

No requests were sent; building the needed list would mean inventing labels rather than using real ones.

The experiment did not run because the required adjustable, labelled destination list does not exist.

Open question. Whether a convincing but wrong option can steal an otherwise-correct answer is still unmeasured.

B

Nothing was sent to the model. The test needs turns labelled with a first-choice and a second-choice destination, and that labelling was never done.

No requests were sent; the required primary and secondary labels do not exist.

The experiment did not run because the required primary and secondary labelling of multi-destination turns does not exist.

Open question. Whether a follow-up request recovers a dropped second destination is still unmeasured.

C

The same 16 labelled turns and the same 24-destination list and wording, sent to three systems: Jev, a separately hosted frontier AI model, and a similarity search over the destination descriptions.

One destination pick per turn per system, 3 repeats, 48 requests to each of the three systems, 144 total.

Jev got 81.2% right (39 of 48), the frontier model got 83.3% (40 of 48), and the similarity search got 68.8% (33 of 48). The frontier model cost 82 times more per decision than Jev and took 3.4 times as long; the similarity search was cheaper and faster than Jev but 12 points less accurate.

Good news for Jev's design: it comes close to an expensive frontier model's accuracy at a fraction of the cost, and clearly beats a plain similarity search. The case for Jev here is cost and speed, not being the single most accurate option.

B

Nothing was sent to the model. The test needs large labelled destination lists of 64, 182, and 255 options that do not exist beyond the 58 real skills available.

No requests were sent; the needed large lists could not be built without inventing labels.

The experiment did not run because it depends on the same missing large labelled destination pool that the roster-size sweep also lacked.

Open question. Whether splitting a large destination list into two steps ever beats asking about it all at once is still unmeasured.

F

No new questions were sent. This experiment replayed the 144 stored answers from the earlier real-turn test, which already recorded four numbers per turn: confidence, the winning option's own probability, the gap between first and second place, and whether the answer was right.

An offline comparison of four ways to decide when to accept an answer, at three fixed reject rates (5%, 10%, 20%), zero new requests.

At matched reject rates, gating on confidence let through exactly the same wrong answers as gating on the winning option's own probability, for example 0.102 wrong-among-accepted for both at a 5% reject rate. Combining both numbers did a little better, by about 2 points, short of the 3-point bar the hypothesis set.

Caveat: in this test, the fancier confidence number bought nothing over the simplest one, and combining two numbers barely helped. With only 20 wrong answers in the data, this could look different at a larger scale, so it is not a final word.

C

No new questions were sent. This experiment replayed the same 144 stored answers and swept each of five built-in numeric thresholds across a wide range of possible values.

An offline sweep of five thresholds, including one from 0.10 to 0.70, over stored answers, zero new requests.

None of the five thresholds moved by more than 0.05 on solid grounds. Recall stayed at 100% across the entire 0.10 to 0.70 range tested for one threshold, meaning the data was too easy to find the real cutoff; only the rate of opening extra, unneeded destinations changed, dropping from 4.53 to 0.31 per turn.

Caveat, not a pass or fail: the current thresholds are not shown to be wrong, but the data available cannot prove they are right either. A bigger, harder labelled set is needed before anyone can say these five numbers are correctly tuned.

C

16 real turns, each sent against two ten-item lists of the same size and shape: one listing destinations, things that act, like specific AI models, and one listing descriptive labels, things that describe, like intent categories.

One main pick per turn per list type, 3 repeats, 96 requests total.

The model's confidence and certainty were sharply lower for destination lists than for label lists on every measure: top probability 0.593 versus 0.959, confidence 0.510 versus 0.944, and the gap between first and second place 0.400 versus 0.923, the same direction on all 16 turns.

Good news, with a design caveat: the two kinds of questions genuinely need different trust settings. A destination-picking question will look less certain than a label-picking one even when it is working correctly, so the same confidence cutoff should not be used for both.

F

8 real turns, each tried three ways: one request holding all 27 destination checks together, 27 separate single-question requests repeating the same turn, and the bundled request followed by a second request re-checking just the top 3 destinations.

27 yes/no checks and one pick per turn per method, 3 repeats, 696 requests total across the three methods.

Bundling used only 1.43 times fewer input tokens than asking separately, far short of the 8-fold saving the hypothesis expected. The bundled and separate methods agreed on the winning destination only 75% of the time (18 of 24), below the 95% bar, because bundling inflated some destinations' scores enough to flip the top answer on 2 of 8 turns. Adding the second follow-up request recovered one of those two wrong turns and raised accuracy by 12.5 points (21 of 24 versus 18 of 24 correct).

Bad news for the assumption that bundling many questions together is free: it changes answers, not just cost, and the savings were far smaller than advertised. Good news for a second follow-up request on the most confusable cases: it measurably fixes some of the damage bundling causes.