Limits and error messages · Academica Experiments

E01–E16 · GROUP 1 OF 7

Limits and error messages

We sent Jev deliberately broken, oversized, and edge-case requests to find out where it actually breaks and what it tells you when it does.

Jev's docs state several hard numbers and promises, like a 10-level cap on Score questions and a 255-option cap on Choice questions, and say a request is rejected cleanly if it goes wrong. A builder who trusts a wrong number in the docs writes code that either rejects valid requests or gets an unexpected server error in production. We wanted to know which of those numbers actually hold on the live API, and whether a broken request fails loud or fails quiet.

What we ran it on

support tickets and product blurbs taken from the Jev docs, plus deliberately malformed or oversized requests built to probe one documented rule at a time

experiments

670

records

642,098

input tokens

Confirmed

Falsified

Mixed

What we learned

The Score cap is exactly 10 levels and the Choice cap is exactly 255 options, but both reject with HTTP 400, not the 422 the docs imply, so error-handling code that only checks for 422 will miss it.

Sending the same question with its options in a different order changed the answer beyond the noise floor in both test cases (differences up to 0.14), so option order is part of the question, not just presentation.

Asking one question in a batch with 40 other questions that argued for the opposite answer moved that question's result by no more than 0.03, matching the spread you get from asking it alone five times.

Repeating the exact same request 20 times in a row did not give the same answer back most of the time: only 2 of 12 questions came back byte-identical every time, and one Score question swung by 0.22 on its probability scale.

Doubling the size of one request from 1 question to 100 questions raised the token cost by only about 19 tokens per extra question, so batching many questions into one call stays cheap per question.

The experiments

E01

The exact Score level ceiling

Confirmed

How many levels can a Score question have before Jev refuses it?

We handed it

A one-paragraph bug report about a Safari-only crash, used as the fixed background for every request.

We asked

A Score question rating how severe the issue is, sent with 1, 2, 3, 9, 10, 11, 12, 20, and 26 levels, plus a repeat of the 10-vs-11 boundary on a second bug report. 11 requests in all.

It returned

Every request with 10 or fewer levels came back with a normal answer (10 levels worked, returning a 10-entry legend). Every request with 11 or more levels was refused, but with HTTP 400, not the 422 the docs imply, and the error message read "Too many score levels. Must have at most 10 levels." A single-level question was also accepted, which the docs do not clearly promise.

What that means

Good news for anyone hard-coding a limit: the ceiling really is 10, and it is a clean rejection rather than a silently truncated answer. The catch is the status code: code written to catch a 422 validation error will not catch this one, and needs to check for 400 instead.

E02

The exact Choice option ceiling

Confirmed

How many options can a Choice question have before Jev refuses it?

We handed it

A short product description, used as the fixed background, with a plain list of category options attached to each request.

We asked

A Choice question asking which category the item belongs to, sent with 2, 254, 255, 256, 257, 300, and 511 options, plus two extra runs at 255 options with heavier and emptier option descriptions. 9 requests in all.

It returned

Requests with 255 options or fewer came back normally, with exactly one probability per option and no options dropped. Requests with 256 or more options were refused with HTTP 400 and the message "Too many choices. Must have at most 255 choices." At 255 options with no descriptions at all, the returned probabilities summed to 0.99, not 1.0, because of rounding.

What that means

The 255-option cap in the docs is exactly right, and it is enforced by counting options, not by their size or content. As with the Score cap, expect a 400, not a 422. Also worth knowing: at the very largest option counts, the probabilities can round to slightly less than 1.0, so code that assumes they sum to exactly 1 should allow a small margin.

E03

Is option order an input, or formatting?

Confirmed

Does changing the order of a question's options change the answer?

We handed it

Two short customer tickets, one an ambiguous complaint about a late shoe order and one a made-up flavor-preference scenario, each paired with a 6-option question.

We asked

The same 6 options in six different orders (forwards, reversed, rotated, and two single swaps), sent three times each to separate real variation from run-to-run noise. 6 requests, 36 questions total.

It returned

For the shoe ticket, reordering options moved the answer by up to 0.10 to 0.14, well above the 0.03 to 0.07 spread seen from repeating the identical order. For the flavor scenario the pattern held too, with reordering moving results by up to 0.09 against a noise floor of about 0.02 to 0.03. The top answer never changed, but the exact probabilities did.

What that means

Bad news if you assumed option order was just presentation: it is not. Two builders sending the same options in a different order can get different confidence numbers back for the same underlying question, even though the winning answer usually stays the same.

E04

Can two questions in one request influence each other?

Confirmed

Can other questions in the same request change an answer?

We handed it

One ambiguous customer ticket, used as the fixed background for every request, paired with a target question asking which department the ticket belongs to.

We asked

The target question sent alone, then alongside 20 unrelated questions, 40 questions arguing for a different department, 19 copies of itself, and 20 questions that name the target's own answer options, each condition repeated 5 times. 25 requests in all.

It returned

The target's answer barely moved across all five conditions, staying within 0.01 to 0.03 of its value when asked alone, including the condition with 40 questions actively arguing for the wrong department. The winning answer, department returns, stayed the same in all 25 requests.

What that means

This is good news for anyone building requests with many questions at once: adding more questions, even hostile or repetitive ones, did not leak into this question's answer in this test. It supports batching many independent questions into a single request without expecting them to interfere.

E05

Is confidence ever equal to max(probabilities), and can it exceed it?

Falsified

Does Jev's confidence number always run below the top probability?

We handed it

Four short scenarios chosen to span the range from a clear one-answer case to a genuinely unclear one with no useful information at all.

We asked

Sets of Choice and Score questions with option counts from 2 to 255, run across the four scenarios to see how confidence compares to the top probability as certainty changes. 6 requests, 47 scored answers.

It returned

For Choice questions, confidence stayed at or below the top probability in every one of 27 cases (24 below, 3 tied), matching the pattern the docs' own example shows. For Score questions the pattern reversed sometimes: confidence came out above the top probability in 3 of 20 cases and below in the other 9. The clean crossover point the hypothesis predicted, somewhere between 60% and 80% certainty, did not show up.

What that means

Caution for anyone using confidence as a safety gate: it behaves differently for Choice questions than for Score questions. For Choice, treating confidence as the stricter, more cautious number appears safe. For Score, it can occasionally be the more lenient number, so a single confidence threshold cannot be reused across both question types without checking which one it is gating.

E06

Can confidence be reconstructed from probabilities?

Confirmed

Can you calculate Jev's confidence number yourself from the probabilities?

We handed it

Three scenarios, one with an obvious right answer, one genuinely ambiguous, and one with no useful information, each paired with a dozen Choice and Score questions of varying sizes.

We asked

Four different formulas computed from the returned probabilities (the top probability, the gap between the top two, and two versions of a spread-based formula), checked against the confidence number Jev actually returned, across 6 requests and 72 answers.

It returned

None of the four formulas matched the real confidence number within 0.02 across the board. The best of the four still missed by an average of 0.05 to 0.11 and was off by more than 0.02 on between roughly a quarter and all of the answers, depending on the formula and question type.

What that means

You cannot reconstruct confidence from a stored probability list after the fact. If a threshold depends on confidence, you need to save the confidence number itself when you get the answer, because there is no reliable way to compute it later from probabilities alone.

E07

Version pinning: does the response say which model answered?

Mixed

Does Jev's response say exactly which model version answered?

We handed it

One short sentence of background text, used as filler, paired with a single simple question, sent under five different model name strings.

We asked

Requests naming the model as jev-latest, jev-preview, the exact version jev-1.13.0, and two older version strings jev-1.12 and jev-1.12.0, each sent twice, plus one check of the model listing endpoint. 12 requests plus 1 listing check.

It returned

Both jev-latest and jev-preview resolved to and reported back the exact version jev-1.13.0 every time, and the model listing endpoint listed only those two aliases, as documented. But the older version strings jev-1.12 and jev-1.12.0 were both rejected with HTTP 400 as unknown models, even though a Jev cookbook uses jev-1.12 as a pinned version.

What that means

Mixed result. If you send an alias, Jev does tell you the exact version that answered, which is what the docs promise and lets you log a real version per request. But you cannot pin to an old version by name once it is retired; that documented cookbook example is currently broken, and pinning only works for versions still being served.

E08

The error taxonomy the endpoint actually returns

Confirmed

What error codes does Jev actually return for a bad request?

We handed it

One short sentence of background text and one simple question, each deliberately broken in a different way for each request.

We asked

Nine broken variants (unknown model, missing pieces, wrong data types, an empty question set) plus a working control and two bad-credential cases, checking the status code and error message for each. 11 requests.

It returned

An unknown model name came back as HTTP 400, not the 404 you might expect. A bad API key came back as 401 and a missing key came back as 403. Every other broken field, like a missing state, a malformed set of Score levels, or an empty question list, came back as HTTP 422 with a message naming the exact field that was wrong. All 11 responses carried a request-id header, and the valid control request succeeded normally.

What that means

Good news: nearly every kind of bad request fails loud with a specific, named field error, so a broken request is easy to debug from the response alone. The one thing to code around is that an unknown model name and the boundary-cap errors from E01 and E02 come back as 400, while most other mistakes come back as 422, so error handling needs to check both codes.

E09

Is instructions actually required?

Confirmed

Do you have to include instructions text on every question?

We handed it

One short sentence of background text, paired with one question of each of the three types (yes/no, Choice, and Score).

We asked

The same three questions sent with no instructions text at all, then with instructions set to an empty string, then with real instructions as a control. 3 requests.

It returned

All three conditions were accepted with HTTP 200 for all three question types, and the answers were close to each other: the yes/no answer ranged from 0.84 to 0.95, the Choice answer from 0.92 to 0.98, and the Score answer from 1.01 to 1.04 across the three versions.

What that means

Instructions text is not actually required by the server, even though the documentation marks it as required. A question can be built from its answer options alone, which matters for anyone generating questions programmatically where the instructions would just repeat the option descriptions.

E10

What actually counts toward usage.input_tokens?

Confirmed

What in a request actually drives its token cost?

We handed it

A short 400-character block of filler text as a starting point, then varied along three separate dimensions: number of questions, length of the background text, and size of the answer options.

We asked

Twelve requests varying one thing at a time: question counts of 1, 5, 25, and 100; background text from about 1,000 to 150,000 characters; and small versus large sets of answer options, recording the token cost each time.

It returned

Each extra question added roughly 19 tokens, fairly steady across the range. Background text cost about 0.19 tokens per character once past around 10,000 characters, and a 150,000-character background used about 29,000 of the roughly 32,000-token budget on its own. A 255-option Choice question added about 5,850 tokens on top of the background, roughly 22 tokens per option.

What that means

This gives builders a real cost model instead of a guess: extra questions are cheap, about 19 tokens each, so batching many questions onto one background text is efficient. Long background text is what actually eats the budget, and a background near 150,000 characters leaves very little room for questions.

E11

What happens at and beyond the shared token budget?

Confirmed

What happens if a request goes over the token budget?

We handed it

A single sentence stating a customer wants a refund, buried inside filler text at sizes from about 50,000 up to 600,000 characters, with the key sentence placed either near the start or near the end.

We asked

A single yes/no question about whether a refund was requested, sent at each size and each placement of the key sentence, to see whether Jev processes the whole text or silently cuts it off. 8 requests.

It returned

Requests at 50,000 and 150,000 characters both succeeded with HTTP 200, and the reported token usage matched a fully-read background, with no sign of the count topping out early. Requests at 300,000 and 600,000 characters were refused outright with HTTP 400 and a "max tokens exceeded" message, not silently truncated.

What that means

Good news: going over the budget fails cleanly with a clear error rather than quietly answering from a cut-off version of your text. One caveat worth flagging: even within the budget, moving the key sentence from the start to the end of the text changed the answer (0.92 versus 0.73 and 0.92 versus 0.80), so where you place the important part of your text still matters.

E12

Determinism across identical repeated calls

Falsified

Does asking Jev the exact same question twice give the exact same answer?

We handed it

One ambiguous customer ticket, used as a fixed background, paired with 12 questions (4 yes/no, 4 Choice, 4 Score) chosen to include some clear-cut and some genuinely borderline cases.

We asked

The identical 12-question request sent 20 times in a row, at least 30 seconds apart, tracking how much each answer moved across the repeats.

It returned

Only 2 of the 12 questions came back with the exact same probability every single time; the rest varied. One Score question moved as much as 0.22 across the 20 repeats, and one borderline Choice question flipped which answer it favored on 3 of the 20 tries.

What that means

This is a caution for anyone assuming Jev gives a fixed answer to a fixed question: it mostly does not. If a decision depends on a threshold near a borderline case, expect it to occasionally land on the other side just from asking again, and build in a noise margin rather than trusting a single call.

E13

Are the response-shape invariants actually invariants?

Falsified

Do Jev's answers always follow their own basic math rules?

We handed it

A handful of scenarios built to stress the answer format itself, including Choice questions with unusual or tricky-looking option names and Score questions with mixed description styles.

We asked

Three requests checking five basic rules on every answer: probabilities sum to 1, the returned options match what was sent, the top answer is really the highest probability, a Score's numeric score matches its own probabilities, and confidence stays between 0 and 1. About 20 answers checked in total.

It returned

Four of the five rules held exactly. The probabilities summed to 1 everywhere except one 255-option case, which came to 0.99 due to rounding (the third time this rounding pattern showed up). A Score's numeric score matched its own probabilities to within 0.05 in every case, but not to an exact match: five answers were off by 0.01 to 0.04, and one was off by 0.01 more than expected, because the server rounds internally.

What that means

The core guarantees hold well enough to trust in practice, but not to the last decimal. Code that checks these rules should allow a small margin, around 0.05, rather than demanding an exact match, since small rounding differences are normal, not a sign of something broken.

E14

Does the key space (question id, Noul criteria keys) matter?

Confirmed

Does renaming a question or its answer keys change the result?

We handed it

A customer message repeating a request three times, asking to speak to a real person, used as a background where the correct read is genuinely subtle.

We asked

The same Choice question sent three times under three different id names to check if the id itself matters, and a yes/no question sent with different label pairs for its two answer keys (true/false, yes/no, True/False, and others) to see which labels are actually honored. 6 requests.

It returned

Renaming the question's id made no difference at all: the three ids returned the exact same probabilities within each run, varying only by the normal 0.00 to 0.06 run-to-run noise. For the answer labels, only the exact literal pair true/false changed the answer as intended; relabeled pairs like yes/no or True/False were silently ignored and gave the same result as having no custom labels at all.

What that means

You can freely rename question ids for your own code's convenience without worrying it affects the answer. But if you write custom labels to steer a yes/no question, you have to use the exact lowercase words true and false; any other spelling gets silently dropped with no error, which is the dangerous kind of failure because nothing tells you it happened.

E15

The concurrency envelope: where does 429 begin, and does it carry a retry header?

Confirmed

How many requests at once before Jev starts rejecting them?

We handed it

One small fixed request, about 600 characters of background plus two simple yes/no questions, sent repeatedly at different levels of overlap.

We asked

Four batches of 120 requests each, run with 1, 6, 16, and 64 requests happening at the same time, recording how many were rejected and how fast they came back. 480 requests in all.

It returned

None of the 480 requests were rejected, at any level of overlap, including the busiest run of 64 at once. Typical response time stayed close to flat across all four levels, around 740 to 800 milliseconds at the midpoint.

What that means

Good news for throughput: sending several requests at once, even a lot at once, did not trigger rejections in this test, and a small pool of concurrent requests looks safe as a starting point. This was measured over short bursts rather than a full sustained minute at the published rate limit, so it does not rule out limits kicking in over longer sustained runs.

E16

Latency and cost as a function of N questions in one request

Confirmed

Does asking more questions in one request save money without slowing it down?

We handed it

A single roughly 4,000-character support ticket, used as a fixed background, paired with pools of pre-built Choice and Score questions.

We asked

Batches of 1, 3, 10, 30, 100, and 255 questions sent together in one request, each size repeated 5 times, compared against sending 1 and 10 questions as separate one-question requests. 50 requests in this comparison.

It returned

Typical response time only grew from about 773 milliseconds at 1 question to about 1,550 milliseconds at 255 questions, roughly double, not the sharp slowdown a per-question cost would predict. Sending 10 questions as one batch was about 3.7 times cheaper and 8.8 times faster than sending them as 10 separate requests. The cost per question kept falling as batch size grew, reaching about a quarter of the single-question cost at 100 questions.

What that means

Batching many questions into one request is clearly worth doing: it is both faster overall and cheaper per question. The savings here were smaller than a widely cited example elsewhere in the docs, so treat the exact savings as depending on your own background text length rather than assuming a fixed multiplier.

THE NOTES AS WRITTEN

108 experiments, 38,194 API responses logged, 45,861,280 input tokens, $1.9262 at $0.042 per million.

Compiled from dependency-docs-reference/jev/notes/12-experiments on 2026-09-19.