Research Round 51 · 2026-11-08
Valid Evidence, Wrong Verdict: Transfer Tests on Code Review and Raw Table Claims, with Independent Recovery
Research question
Do the evidence, agreement and rewrite checks of the contract rounds transfer to a clearly different non-document task, code review against a specification and numerical claims against raw tables, and does routing a changed status to an independent reader recover correct repairs that a preservation gate would lose?
Summary
Round 51 takes the planned third task. Sixteen authored code review items (four specifications, each with a correct implementation, a subtle defect that passes every shown example, an obvious defect, and code that calls an unshown helper) have gold fixed by executing a hidden test suite. Sixteen authored raw table claims (true, false but true under one named misreading, false against one cell, and undetermined because rows are declared missing) have gold fixed by recomputation. Twenty two real SEC credit agreement exhibits, none used before, carry one covenant threshold claim each. Structurally, an exact evidence span check admits all 64 scripted wrong status mutations, running the code on the specification's own examples admits all four subtle defects as PASS, and across 81 truth, baseline, rewrite and adjudicator states preservation loses 12 correct decisive repair states while independent recovery restores four and adds eight unsafe states, each with a shared wrong decisive verdict. Live, Witnesses B, C and D made no decisive error against execution or recomputation on the 32 authored items, and the validator changed none of 53 natural baselines, so preservation and recovery could not be compared on live data. Witness B found a wrong authored HOLD label with a counterexample: two valid inputs reach the unshown helper as the same floating point value but require different answers, so FAIL is established; the label, the two world check, the blind audit and two readers missed it. Witness C read four SEC compliance claims as FAIL in blind reads but kept HOLD when validating another reader's HOLD ledger; that wording makes FAIL defensible, so those reads are contestable.
Major findings
- F51.S1 (deterministic, E51-A): An exact evidence span check admits all 64 scripted wrong status mutations of the authored code and table ledgers.
- F51.S2 (deterministic, E51-A): Running each implementation on the specification's own examples admits all four subtle defects as PASS and routes all four obvious defects.
- F51.S3 (deterministic, E51-A): Across 81 states, preservation loses 12 correct decisive repair states; independent recovery restores four and adds eight unsafe states, every one with the rewrite and the independent reader sharing the same wrong decisive verdict.
- F51.E1 (live, E51-B): On 32 authored code and table items, Witnesses B, C and D made no decisive error against execution or recomputation, catching all subtle defects and table traps.
- F51.E2 (live, E51-B): Witness B identified a wrong authored HOLD label (X01H) with a counterexample confirmed by a deterministic check; the blind label audit agreed with all 54 fixed labels, including the wrong one.
- F51.E3 (live, E51-B): Witness C read four of six SEC compliance claims as FAIL against fixed HOLD labels, wording the blind audit had flagged as admitting FAIL.
- F51.E4 (live, E51-C): The validator changed none of 53 natural baselines and kept HOLD on four items where its own blind read was FAIL.
- F51.E5 (live): One reader output and the first synthesis call were empty; neither was scored or repeated as an experimental read.
Experiments
- E51-A Structural diagnostic for evidence span, example execution, preservation and recovery gates Complete
- E51-B Independent reads on code review, raw table claims and SEC covenant claims Complete
- E51-C Validation of natural baselines with replayed preservation and recovery Complete
Hypotheses
- H51.1 In both non-document domains an exact evidence span gate admits every wrong status mutation, and a visible example execution gate admits every subtle code defect as PASS.
- H51.2 In each non-document domain at least one live decisive wrong read cites only exact packet evidence.
- H51.3 In each non-document domain at least one reader returns a decisive verdict on an open dependency HOLD item.
- H51.4 Agreement routing of Witnesses C and D yields no more unsafe outcomes than the better single reader in each domain, with a false hold cost.
- H51.5 With natural baselines, preservation loses at least one correct decisive repair, and independent recovery restores at least one without admitting a wrong decisive change.
- H51.6 Under recovery, every unsafe state that preservation would have blocked has the rewrite and the independent reader sharing the same wrong decisive verdict.
Architecture implications
Research diagnostic and output replay only. Not a live orchestration or matched call comparison. No production system, release gate, or Noetis behavior changed.
Open questions
- Do routing, preservation and recovery separate on items hard enough that single readers make decisive errors in each domain, under matched call baselines?
- Should a decisive minority verdict that carries an executable counterexample be checked rather than vetoed by agreement routing?
Connected research
Full experimental notes, prompts, internal architecture details and implementation procedures are kept by Encevra SI Research Labs.
