← Notebook library

Research Round 53 · 2026-11-10

Stated Plainly, Tested to Break: Domain General Candidate Propositions on Disagreement, Review, Memory and Evidence, on Held Out Mixed Cases

Collaborative IntelligenceArtificial Cognitive ArchitectureReasoning & VerificationEvaluation ScienceE53-A structural diagnostic, E53-B live blind label audit and independent, repeated, resampled and deliberate reads, E53-C live rewriting review, flag only review and seeded proposals, and E53-D live stored conclusion and stored procedure memory arms, on 188 items. All planned experimental calls attempted; 100 first outputs were unusable and were rerun once; 7 of those were rerun a second time; 3 remain empty and are disclosed. Exact run replay: 1367 checks pass. H53.1, H53.2, H53.3, H53.4, H53.5, H53.6, H53.10, H53.11 and H53.12 supported; H53.7, H53.8 and H53.9 not supported. DOI 10.5281/zenodo.24134801, Zenodo record published. No production system changed.Architecture impact: None

Research question

When candidate propositions about disagreement, review, memory and evidence are stated in domain general language, which of them survive preregistered attempts to break them on leftover mixed cases that are not only contracts?

Summary

Round 53 states seven candidate propositions (C1 to C7) in domain general language and tries to break each one. Track R replays all 36 Round 49 packets verbatim. Track G adds 120 held out items from 24 families in six non contract groups, including a new partial period certificate variant. Track S adds 32 new SEC executive employment agreement exhibits, each with one base salary claim. Structurally, agreement routing is unsafe in 4 of 27 states, exactly when both reads share the same wrong decisive verdict. Live, Witness D released 38 unsafe verdicts on 188 items, including a wrong FAIL on 9 of 9 Round 49 near misses and 24 of 24 held out near misses. Routing that read with its own resample left 35 unsafe verdicts; with Witness B, 4; with Witness C, 18; with Witness D's deliberate read, 13. The Spearman correlation between disagreement and unsafe verdicts removed was 0.968. Removing the HOLD case from the deliberate prompt raised unsafe verdicts from 13 to 32. C1, C2, C3 and C7 survive. C4, C5 and C6 do not survive as worded: a seeded wrong proposal was not inherited, rewriting review added no new unsafe release, and memory effects on Witness D missed the preregistered threshold.

Major findings

  • F53.S1 (deterministic, E53-A): Over 27 states of gold and two verdicts, agreement routing is unsafe in 4 states, exactly when both reads share the same wrong decisive verdict, flag only review is unsafe in 4 states, and rewriting review is unsafe in 12 states including 6 new ones. The routing identity holds on 2067 of 2067 recorded pairs.
  • F53.S2 (deterministic, E53-A): All 36 track R prompts are identical to Round 49 and Round 52; all 120 track G certificates and all 32 SEC windows check.
  • F53.E1 (live, E53-B): Witness D released 38 unsafe verdicts on 188 items (Wilson interval 0.1510 to 0.2652), including a wrong FAIL on 9 of 9 Round 49 near miss items and on 24 of 24 held out near miss items, and 29 unsafe verdicts on track G.
  • F53.E2 (live, E53-B): Routing Witness D with its own resample left 35 unsafe verdicts; routing with Witness B left 4; with Witness C, 18; with Witness D's deliberate read, 13. Spearman rho between disagreement and unsafe verdicts removed was 0.968.
  • F53.E3 (live, E53-B): Removing the HOLD case from the deliberate prompt raised unsafe verdicts from 13 to 32 on 187 paired items.
  • F53.E4 (live, E53-C): A seeded wrong proposal was not kept more often than the same reviewer's own blind error for two of three reviewers.
  • F53.E5 (live, E53-C): Flag only review created 0 new unsafe releases and left 12 unsafe verdicts; rewriting review created 0 new unsafe releases and left 21.
  • F53.E6 (live, E53-D): On Witness D, a stored conclusion produced 44 unsafe verdicts against 38, and a stored procedure produced 33 against 38, both above the preregistered p threshold. On Witness C the same contrasts were 27 against 18 and 2 against 18.
  • F53.E7 (live, E53-B): 526 of 529 unsafe decisive quotes copied the packet exactly (0.9943, interval 0.9835 to 0.9981).

Experiments

  • E53-A Structural diagnostic for routing, flag only review and rewriting review, with replay and gold certificates Complete
  • E53-B Blind label audit and independent, repeated, resampled and deliberate reads Complete
  • E53-C Rewriting review, flag only review and seeded wrong proposals Complete
  • E53-D Stored conclusion and stored procedure memory arms Complete

Hypotheses

  • H53.1 Structural: routing is unsafe exactly when both reads share the same wrong decisive verdict, is never unsafe where either read alone is safe, flag only review is unsafe only where the proposal is unsafe, and rewriting review creates unsafe states from safe proposals.
  • H53.2 Replication: Witness D is FAIL on at least 7 of 9 Round 49 near miss items; its resamples reproduce at least 80 percent of those wrong decisive verdicts; its deliberate read moves at least 3 of the 9 to HOLD.
  • H53.3 Each of route(D, B), route(D, C) and route(D, D deliberate) has fewer unsafe outcomes than route(D, D resample 1); at least two McNemar tests give p below 0.05.
  • H53.4 Across eleven second reads paired with Witness D, Spearman correlation between disagreement (without gold) and unsafe outcomes removed by routing is at least 0.7.
  • H53.5 route(D, D deliberate) beats route(D, D resample 1); removing the HOLD case from the deliberate prompt removes the benefit.
  • H53.6 For route(D, B), route(D, C) and route(D, D deliberate), unsafe outcomes removed relative to Witness D's read are at least the false holds added.
  • H53.7 For at least two of Witnesses B, C and D, the rate of keeping a seeded wrong verdict exceeds the rate of reaching that same wrong verdict on a blind read, McNemar p below 0.05.
  • H53.8 Flag only review of Witness D creates no new unsafe outcome and has no more unsafe outcomes than rewriting review; rewriting review creates at least one new unsafe outcome.
  • H53.9 A stored conclusion raises Witness D's unsafe outcomes; a stored procedure lowers them; removing the open record steps removes the benefit; each with p below 0.05.
  • H53.10 At least 90 percent of unsafe decisive outputs with a nonempty quote copy the packet exactly, and the Wilson lower bound is at least 0.8.
  • H53.11 Witness D releases at least 10 unsafe verdicts on track G; each reader's temperature 0 repeat agrees on at least 95 percent of usable items.
  • H53.12 In at least 5 of 6 track G domain groups, route(D, B) and route(D, D deliberate) each have no more unsafe outcomes than route(D, D resample 1).

Architecture implications

Research diagnostic and output replay only. Arms are replays of recorded calls; reader capability and resource per call are not matched. No production system, release gate, or Noetis behavior changed.

Open questions

  • Do the surviving statements (C1, C2, C3, C7) hold on new packets and surface forms with family clustered intervals?
  • Can C4, C5 and C6 be restated more narrowly, or should they be dropped?
  • Does the stored procedure still help on families authored without sight of its wording?

Connected research

Full experimental notes, prompts, internal architecture details and implementation procedures are kept by Encevra SI Research Labs.