Evidence & validation
Every figure, with the protocol that produced it
A number without its test context is an adjective. This page is the register: what was measured, on what corpus, under what protocol, graded by whom, with what definitions — and where the raw artifacts live. The limitations are listed with the same care as the results.
The headline claims
- ~97% retrieval hit-rate across a 480-question blind evaluation over a 700-document, 29,506-chunk public corpus. Protocol B1, below.
- 76.9% strict factual accuracy in the shipped configuration on the same sealed set; 91% single-fact and 92% entity disambiguation. Protocol B1.
- 16 / 16 model fabrication attempts held back by the provenance gate in that run; none reached a user as Verified. Protocol B1.
- 8 / 8 configuration-scoped questions answered with the effectivity filter active and 0 wrong-variant leaks; unfiltered baseline: 4 / 8 and 4 leaks. B4.
The evaluation protocols were internally executed with blind separations — the engine never saw the sealed key; a separate source-only oracle authored it and a different model family judged the answers. Raw artifacts are archived under a SHA-256 manifest. Independent external replication is a pilot objective.
| Capability | Corpus | Test | Result | Status |
|---|---|---|---|---|
| Retrieval | 700 public documents | 480 blind questions, oracle-authored sealed key | ~97% | Measured · B1 |
| Provenance safety | 700 public documents | 16 model fabrication attempts in the same run | 16/16 held back · none Verified | Measured · B1 |
| Strict answering | 700 public documents | Blind grading, separate AI judge | 76.9% | Measured · B1 |
| Refusal | 43-paper physics corpus | 12 out-of-corpus questions in a 60-question blind exam | 12/12 refused · 0 fabrications | Measured · B2 |
| Audit | Seeded synthetic corpus | Four planted error classes, scored within those classes | Recall 1.0 · precision 1.0 | Committed benchmark · B3 |
| Audit, blind | Real NASA document family | Unseeded run, findings hand-checked | 15 / 15 genuine findings | Measured · B3 |
| Effectivity | Disclosed S1000D corpus | Eight configuration-scoped questions; unfiltered baseline 4/8 with 4 leaks | 8/8 · 0 wrong-variant leaks | Committed benchmark · B4 |
| Author | — | — | No published figures | In development |
The full blind evaluation
The largest run in the register: a blind evaluation over a 700-document public corpus, with the question key authored by a separate source-only oracle and the answers graded by a judge from a different model family. The venture ran the protocol; external independent replication has not yet occurred.
-
Corpus
700 real, public documents indexed to 29,506 retrievable chunks with 0 indexing failures, in an isolated copy of the system.
-
Question authoring
A separate source-only oracle — reading the raw documents, never Cortessa's index or output — authored 486 candidate question-and-answer items. A verbatim integrity gate (the expected answer must appear in the source) sealed 480 of 486 into the evaluation key. The engine never saw the key.
-
Engines
Two different small on-device language models answered the same 480 questions against the same index, fully offline on interim development hardware.
-
Blind grading
A separate AI judge from a different model family — no shared model family with the answering engines — graded every answer against the sealed key, driven by deterministic scorecard scripts.
Retrieval ~97% on both engines — engine-independent, because retrieval is a property of the ingestion pipeline, not the model. In this run the engines attempted fabrication on 16 items and the deterministic gate held back all 16 — no ungrounded value reached Verified. Strict answer accuracy was 76.9% in the shipped configuration, converging with two further blind runs on different corpora. Raw artifacts — the sealed key, both engines' answers, verdicts and metrics, and the exact index — are archived under a SHA-256 manifest with an outer digest.
The separate blind exam
A separate, smaller exam designed to test the refusal behaviour as hard as the answering: an oracle authored 60 questions over a real 43-paper physics corpus, deliberately including questions whose answers are not in the corpus at all. The correct behaviour on those is refusal — anything else is fabrication.
83% strict accuracy, fully offline and on-device. All 12 out-of-corpus questions refused — 0 fabrications. The engine never sees the answer key; the oracle never sees the engine.
The audit benchmark
Audit is measured two ways. First, a committed seeded benchmark: known errors of four classes — numeric conflicts, supersession, broken references, stale content — are planted in a corpus, and the audit must find all of them and nothing else. Second, a blind run over a real NASA programme document family with no seeding at all, where every finding is checked by hand against the sources.
Seeded benchmark: recall 1.0 and precision 1.0 across all four error classes — a synthetic corpus, precision scored within the seeded classes. Blind run: 15 / 15 genuine findings on the real document family. Neither figure is a general precision claim, and no precision figure has yet been measured on the full 156-document demonstration corpus. Findings are cited and adjudicable — an engineer disposes of each one, and every disposition is recorded to the ledger.
The effectivity benchmark
For configuration-aware answering, the failure that matters is a wrong-variant leak: quoting a module that does not apply to the product configuration in front of you. The benchmark asks eight configuration-scoped questions over a disclosed, purpose-authored S1000D corpus containing a genuine pre-/post-modification module pair — so the wrong answer is always available to leak.
Without the filter, the system answered 4 of 8 questions and leaked the wrong variant four times. With the effectivity filter active, it answered 8 of 8 with 0 leaks. The product kill switch restores the unfiltered baseline, isolating the filter's effect. The corpus is disclosed and purpose-authored; external validity against a licensed CSDB is an explicit pilot objective.
Failure is defined before it is counted
- Provenance breach. An ungrounded value — a number or identifier not present in the cited source — reaching the user marked Verified. The headline safety figure counts these, and only these.
- Provenance gate. Deterministic string-evidence checking, not model judgement. A versioned containment policy decides whether a claimed span is eligible for Verified, and the cited source remains available for human inspection.
- Fabrication. The engine inventing content with no source support. Fabrications the gate catches are recorded as gated, not as breaches — the count of attempts and the count of breaches are reported separately.
- Refusal. Declining to answer when the fact is not in the corpus. For out-of-corpus questions, refusal is the correct answer and is scored as such.
- False positive (audit). A reported finding that hand-checking against the sources shows is not a genuine inconsistency.
- Wrong-variant leak (effectivity). A data module served for a product configuration its declared applicability excludes.
What these results do not claim
- Interim hardware floor. Every figure was measured on interim development hardware with small on-device models. Accuracy figures are a floor of the current configuration, not a ceiling of the design.
- Multi-document synthesis is the measured weak point. Questions requiring reasoning across several documents at once score markedly lower than single-source questions on the interim models. The provenance guarantee holds regardless — a weak answer is held at Draft or refused, not passed off — and the path past the ceiling is a governed multi-hop layer, table-aware claim extraction and a larger on-device model.
- Not sector-specific. The corpora are real and public (NASA/NTRS, arXiv and physics document sets). A proof point for any particular estate is established with the customer during a pilot before it is claimed.
- Refusal calibration is a trade. The shipped configuration trades a small amount of refusal strictness for answer accuracy; the trade and its measurements are recorded in the run artifacts.
- Author is not in these results. The Author capability is in development and carries no published figures. Its results will join this register when it ships.
Where the evidence lives
Each protocol's raw artifacts — sealed question keys, engine answers, verdicts, metrics and the exact index evaluated — are archived under a manifest that records every file with its size and SHA-256 digest, plus an outer digest of the manifest itself. Verify the outer digest, then verify each file against its recorded digest: the same evidence-pack design the product uses.
The runs are re-runnable from the archived harnesses. In a pilot conversation we walk through the artifacts and the protocols directly — the register above is the summary, not the substitute.
External replication next The oracle, engine and judge are separated, but the venture executed the protocol. The register does not yet include a third-party rerun or certification. Independent replication and a pilot measured against the customer's own success criteria are the next evidence steps.
Judge it against your own documents
A scoped, paid pilot runs on your hardware, behind your firewall, against success criteria you set — and ends with an evidence pack, not a slide deck.