C

14 real news headlines from BBC News and Hacker News (out of a planned 400), each naming a person or organization.

Two versions of a multiple-choice stance question, one with three options (for, against, neutral) and one with a fourth option added (unrelated), plus two yes/no checks, all in one request per headline; 14 requests total.

The two versions picked different answers on 5 of the 14 headlines, and 'unrelated' captured an average share of 0.30 of the probability on those 5 rows. On those same rows, a yes/no check for whether the headline was really about the entity also came back low (0.38 or below), agreeing with the four-option answer. The full 400-row test still needs 150 headlines labelled by two people before this can be scored against human judgment.

The fourth option is not just decoration: it changes real answers, so anyone using a for/against/neutral scale should add it. This is still an early read, though; the real accuracy check against human labels has not happened yet.

B

Nothing was sent to the model. This experiment needed two people to label the same 150 headlines by hand first, and that labelling never happened.

The plan was to compare two human readers' agreement with each other, then compare the model's agreement with each of them, on the same headlines and stance question used in E93's design.

It did not run. Only one labeller was available, not the two the design required, so there was no way to measure agreement between humans at all.

Without knowing how much two careful humans normally disagree with each other on this task, no accuracy number for the model can be trusted; a mismatch with one human might just be ordinary human disagreement. This stays blocked until a second labeller is found.

C

The same 14 real news headlines used in the option-splitting check.

The same for/against/neutral stance question asked twice in one request per headline, once with the options listed forward and once listed in reverse; 14 requests total.

The answer never flipped, on 0 of 14 headlines, and the probability given to 'for' moved by an average of only 0.016, with the largest single change at 0.09.

Option order does not appear to bias this question. That is good news: the order these options are listed in does not need to be fixed or shuffled to get a trustworthy answer.

F

500 real headline pairs (1,000 individual headlines) from the Upworthy Research Archive, a public collection of headline A/B tests run on real articles between 2012 and 2015.

A three-level multiple choice describing headline packaging (straight, framed, curiosity-gap) plus two yes/no follow-ups, asked once per headline; 1,000 requests covering the 500 winning and 500 losing headlines.

The model favored the curiosity-gap style on the headline that actually won its click test only 42.4% of the time, worse than a coin flip and below the 50% bar the test needed to pass. It called both the winning and losing headline curiosity-gap on 398 and 386 of 500 headlines. Checking the archive itself explained why: in 275 of 500 pairs the two headlines shared the same photo and the same article, so both were written in the same style to begin with, leaving no real style difference for the model to find.

This is bad news for using the model to predict which headline will win a real audience test: it cannot see a difference where the archive's own headlines do not have one. Anyone trying this should first check whether their pairs actually differ in packaging, not just in wording.

C

300 real headlines from 16 public news feeds, matched against two public category lists: IPTC's news-topic tree and the National Library of Medicine's medical-subject tree.

A two-step category choice, first the top branch, then the sub-branch under it, run once keeping only the top answer and once keeping the top three branches, for each headline against each of the two trees; 2,400 requests total.

On the news-topic tree, the wider search agreed with the cheaper single-path search on 96.7% of headlines (290 of 300). On the medical tree, a deliberate mismatch for news headlines, the two searches agreed less often, at 90.3% (271 of 300), because the first-step choice was far less clear-cut there (its top two options were within 1.5x of each other on 67 of 300 headlines).

For headlines checked against a topic list built for them, the cheap single-path search is basically as good as the expensive wider one, so there is no need to pay for the wider search. The medical-tree number is really a finding about using the wrong list for the job, not about the search method.

C

14 of the 400 headlines used in this group, each with one named person or organization.

Two versions of the same stance question in one request per headline, one naming the entity directly in the question text and one pointing to it through a data field, with a controlled pair that changed only that pointer; 28 requests total.

On the controlled pair, where only the pointer changed, the two versions picked the same answer on all 14 headlines, and the version that pointed to the entity in the data was slightly more confident on average, not less (a difference of 0.066).

This is good news for building at scale: the target can be handed to the model through structured data instead of being written into the question text every time, without losing accuracy. That matters once one headline needs to be checked against several different people or organizations.

C

10 real BBC headlines paired with their article summaries; this pilot used short RSS summaries rather than full articles.

A yes/no overstatement check, the same question worded the opposite way as a check on itself, and a graded overstatement score, all in one request per headline; 10 requests total.

The overstatement check came back above 0.6 on 4 of the 10 headlines (as high as 0.81), clearing the bar the test needed to pass. The two oppositely worded yes/no checks never both said yes on the same headline, so the question was not just flipping a coin. There are not yet hand-made labels to say how many of the flagged headlines are actually right.

The question produces a real, non-random signal even at this small scale, which is encouraging, but it still needs a comparison against a human's own judgment before anyone should trust the rate it reports.

C

400 real articles from 20 public news outlets, each with a full article body (median length 6,567 characters, ranging from 909 to 54,905).

The same overstatement check asked three times in one request per article, once seeing only the first 200 characters, once 800, and once the full article; 400 requests total.

The score moved by more than 0.2 points between the shortest and longest views on 70.2% of the 400 articles, far past the one-in-four rate the test needed to fail on. The average score dropped from 0.640 at 200 characters, to 0.489 at 800, to 0.358 on the full article, more text making the headline look less overstated on 363 of 400 articles.

This is a real caveat: the overstatement rate depends heavily on how much article text is shown, and a 200-character summary reads as far more sensational than the full piece. Any reported overstatement rate needs to say how much text was used.

C

14 real news headlines, scored on three separate scales in one request each.

Three graded scores, tone, urgency, and how much the headline frames the story as a conflict, asked together in one request per headline; 14 requests total.

None of the three scales moved together: the correlations between them were -0.07, 0.19, and -0.02, all far below the 0.9 level that would mean two scales were measuring the same thing. All three scales also had enough spread in their answers (1.89, 1.83, and 2.09, well above the required minimum of 0.05) to rule out a flat, unresponsive scale.

Good news: these three questions are worth asking separately on this sample, and none of them should be dropped as redundant.

M

10 pairs, each a sentence from a real news article checked against a real medical paper abstract; 6 pairs where the claim paraphrased a different paper, and 4 self-source pairs where the claim came from the very paper being checked.

A three-way multiple choice (supports, contradicts, says nothing) plus a yes/no check on whether the source addresses the claim, in one request per pair; 10 requests total.

On the 6 cross-paper paraphrase pairs, the model said 'says nothing' every time, at 0.95 to 1.00 confidence, rather than supports. On the 4 self-source pairs it correctly said 'supports' every time, at 0.94 to 1.00 confidence, and the yes/no check agreed with the multiple choice on all 10 pairs.

This is a mixed result: the check works when a claim is lifted straight from its source, but a paraphrased claim without matching wording gets read as unrelated rather than supported, a real limit for checking a news claim against a different document than the one it paraphrases.

M

310 real clinical trial records, each paired with its registered outcome and the outcome the published abstract actually reported, split by a fixed rule into 34 records that genuinely conflict, 28 that partly conflict, and 248 that agree.

A three-way relation choice, the same choice with a fourth 'mixed' option added, and a yes/no partial-support check, in one request per record; 124 requests across a sample of the three groups.

'Mixed' won on only 3 of the 34 genuine-conflict records (an average share of 0.111) but won on 18 of the 28 partly-conflicting records (an average share of 0.468), doing real work on the two-sided records the data actually supplied, just not the fully-conflicting ones the test expected. It was also picked as a hedge on 9 of 62 agreement records (14.5%), where the two sides did not actually conflict.

This is a genuine mixed result: 'mixed' is doing useful work on records that are truly two-sided, but it also gets picked sometimes when it should not, so it cannot be trusted on its own and needs a second check.

C

12 real clinical trial records (6 registered as randomized, 6 as observational) and their published abstracts, with the field naming the actual study design removed before the model saw them; a title-only version was tried first, then a fuller version adding the study summary, phase, and enrolment.

A six-way choice of study design types, a yes/no check on whether the design can support a causal claim, and two checks for made-up or off-target answers, in one request per record across the two state versions; 24 requests total.

With summary, phase, and enrolment included, the design-type choice matched the record's own randomized-or-not label on 5 of 6 observational records but only 2 of 6 randomized ones. Checking those 6 afterward found that none of the 6 states given to the model contained any word signalling randomization, though the registry marks all 6 as randomized, so the misses came from missing information, not a misreading. Separately, the yes/no causal-strength check was clearly lower on randomized-labelled records (average 0.58) than observational ones (average 0.12), a gap of 0.46.

The causal-strength check works well and separates the two kinds of evidence clearly. The design-naming check needs the actual design description included in its input; leaving it out and expecting the model to infer 'randomized' from nothing is a data problem, and an easy one to fix.

C

85 matched pairs of real medical claims, each checked against a real randomized-trial abstract and a real observational-study abstract on the same exposure and outcome, drawn from PubMed.

A yes/no check on whether the evidence can support the claim, asked once per source per pair; 170 requests total.

The check averaged 0.488 on the randomized-trial side against 0.138 on the observational side, a gap of 0.349 that clears the 0.3 bar the test set. The two sets of scores overlapped by only about a quarter (an overlap of 0.235), well under the 'more than half' failure line, and the observational average of 0.138 was nowhere near the 0.7 mark that would mean the check just says yes to everything.

This is good news: the model reliably rates randomized-trial evidence as stronger support for a causal claim than observational evidence for the same claim, exactly the distinction a claim-checking tool needs it to make.

M

80 real clinical trial records for the supersession check, and 147 real PubMed records (120 retracted, 27 matched clean articles) for the retraction check.

A yes/no 'has this been superseded' check plus a multiple choice naming which later trial superseded it, checked against a fixed mechanical rule; and a separate yes/no 'was this retracted' check with the word 'retracted' stripped from the title first; one request per record, 239 requests total.

The supersession check agreed with the mechanical rule on 65.0% of records (52 of 80), above the 50% failure line but below the 70% bar the test needed to pass. The multiple choice naming the specific superseding trial almost never fired, picking 'none' on 79 of 80 records. The retraction check missed 90.8% of actual retractions (109 of 120) and never falsely flagged a clean article (0 of 27); closer inspection showed it only caught a retraction when the word 'retract' literally still appeared in the text it read, catching all 11 of those and none of the other 109.

The supersession check is a partial signal, not yet reliable enough to use alone, and the trial-naming option needs rework since it barely ever answers. The retraction check is closer to a warning than a tool: it is not reading the fact of retraction at all, only spotting the word, so it should not be relied on once that word is missing, which is most of the time.

C

6 real claims taken from PubMed abstracts, each checked against 10 candidate abstracts twice, once with the true source included and once with it swapped for an eleventh real abstract.

A multiple choice naming the best-matching source, with a 'none of these' option, plus a yes/no check on whether the true source exists among the candidates, one request per condition; 12 requests total.

When the true source was removed, the yes/no check correctly stayed at or below 0.35 on all 6 rows (average 0.13), well under the 0.5 failure line, and the multiple choice correctly answered 'none' on all 6 of those rows too. When the true source was present, the multiple choice named it correctly on 4 of 6, and on the 2 misses the yes/no check had already scored low (0.10 and 0.17), flagging trouble even though the multiple choice picked wrong.

This is good news for building a claim-checking pipeline: pairing the yes/no existence check with the multiple choice is the safer design, because the yes/no check catches cases the multiple choice gets wrong, and reading the multiple choice alone would sometimes mislead.

C

130 real studies, drawn from 258 candidates, from three published Cochrane systematic reviews, each carrying the review's own inclusion or exclusion decision and stated research question.

A yes/no relevance check, a multiple choice naming the review's own exclusion category, and a graded evidence-strength score, in one request per study; 130 requests total.

The relevance check averaged 0.538 on studies the review actually included versus 0.261 on studies it excluded, a gap of 0.277 that clears the 0.20 bar, with the two groups overlapping by only about a fifth (a separation measure of 0.800, where 1.0 is perfect). This was not even across the three reviews, though: one review, on Alzheimer's drugs, showed a strong gap of 0.365; a second, on hearing aids, a smaller gap of 0.164; and the third, on e-cigarettes in kids, showed no separation at all, a gap of -0.016, no better than chance.

The check works well on average against a genuinely independent, expert-made label, which is rare for this kind of test. But it hides real variation: it worked clearly on two of the three reviews and failed completely on the third, so anyone using this check should test it on their own topic rather than trust the overall average.