All three misses came from one question, the hardest synthesis question in the set: which safety signals all three therapies share, and which sponsor makes which. A heart-rate increase is reported for all three, but the FDA lists it under Warnings and Precautions for two and under Adverse Reactions for the third. An answer that compares only the warnings lists concludes, from a true premise, that the signal is not shared. Asked from memory, two of three frontier models got the placement wrong, in opposite directions.
This release guides the answering model to search for a specific finding in each product's own documents before stating that something is absent. Result: 3 of 6 correct, citing the Zepbound label. Previous release: 1 of 6.
We publish the miss because it is what makes the rest of the page worth reading.
The knowledge base and retrieval stay the same. Each use case picks its model.
| Answering model | Correct | Right every repetition | Invented | Correct source document | Cost per answer |
|---|---|---|---|---|---|
| Claude Opus 5, two runs | 93 of 96 | 15 of 16 | 0 | 100% | $0.61 |
| GPT-5.4 mini | 40 of 48 | 11 of 16 | 0 | 98% | $0.12 |
| Claude Haiku 4.5 | 38 of 48 | 11 of 16 | 0 | 93% | $0.09 |
The frontier run was set up for accuracy, not economy: extended reasoning and as many searches as each question needed. Low-cost models on the same documents answer 8 of 10 correctly from $0.09 per answer, still with nothing invented.
Ten public documents on Saxenda, Wegovy and Zepbound: FDA clinical and multidisciplinary reviews of 230 to 390 pages, prescribing information, approval packages for added indications, and the STEP 1 trial in the New England Journal of Medicine. Scanned pages, dense tables, the same measure reported in several places for different trials and time points, labels revised across versions. A correct answer depends on the right passage, not just the right document.
Ingest: each document becomes a structured, page-referenced knowledge base once, when added.
Answer: every question takes the same path a user's question does. The model searches the collection through Hippocampus as often as it needs, using hierarchical retrieval (patent pending), and cites document, section and page for every claim. Retrieval is restricted to the ten documents.
Repetition: each question asked three times in independent conversations with no shared memory. The frontier model ran the full set twice.
Grading: a separate grading model compares each answer with a reference answer, three votes, majority verdict. Declining the control counts as correct.
Three repetitions separate stable behaviour from chance. They do not give a confidence interval. The one-answer spread between the two runs is the honest measure of repeatability.
The documents are public. We will load the same ten into a separate Hippocampus workspace so you can ask your own questions, with the model of your choice, and grade with your own reviewer. The question set and reference answers belong to the customer they were written with; with their agreement, we take qualified parties through every answer, retrieved passage, citation and grader rationale under NDA.
*Claude Opus 5 as answering model. 16 expert questions, each asked six times across two runs (96 answers), over 10 public FDA documents (2,198 pages). Retrieval accuracy: a correct source document reached for every answerable question. Grading by a separate model, three votes, majority verdict. Thalius evaluation, 16 to 17 September 2026.