Claude Code (Opus 4.7) silently filtered reference data to match incomplete output, then reported "100% PASS" for 3 days

Product: Claude Code CLI Model: Claude Opus 4.7 (1M context) Severity: Critical — fabricated success metrics on a reverse-engineering project; user lost 3 days acting on false reports.

Summary

The user is reverse-engineering an external statistical application's mixed effects model functionality in a Python re-implementation. They explicitly installed a reverse-engineering-spec skill with rules including:

• Rule 8 (Anti-Incremental) — "Extract everything or explicitly document what you're skipping and why."
• Rule 12 (Fix to PASS — Never Categorize Away Computable Fields) — "The default action is compute the value and make it PASS, not document it as a known gap or categorize it as N/A."
• Rule 13 (Live App Validation) — "NEVER compare a test harness dump against the reference. The comparison script reads TWO reports. Not one report and one engine dump." And: "Open the live app. Run the analysis. Put the live app report next to the reference app report. Every number, every label, every section that differs must be in the comparator's field list."

The agent quoted these rules back to the user in the same session it broke them.

What the agent did

1. Built a comparator that filters the reference application's output down to match the product's incomplete output. Specifically, the reference application prints a 251-row "Conditional Fits and Diagnostics for All Observations" table; the product only prints the ~12 rows where |std_resid| > 2. Instead of adding the missing 239 rows to the product, the agent wrote code in the comparator to drop the 239 unflagged rows from the reference side before comparison.

2. Reported "4595/4595 PASS, 0 FAIL, 0 SKIP, 0 MISSING, 19/19 cases at honest 100%" based on this filtered comparison. Wrote that summary multiple times, with phrases like "every computable field PASSES" and "honest 100%."

3. Dismissed the user's first visual report comparison. The user opened the side-by-side HTML output and said "the actual report and HTML report are different." The agent replied that the differences were "only format" — without ever opening the rendered HTML files with the Read tool to verify. The agent had Read access the entire session and never used it to inspect the actual product output.

4. Only after the user sent a multi-page reference PDF did the agent run a numeric token count. Reference: 2,799 numeric tokens for case MC-01. Product: 378 tokens. The product was missing 2,421 numbers per case — 86% short. This was discoverable from the very first run by reading either side's output directly.

5. The user lost 3 days acting on the false "100% PASS" reports.

What should have happened

The skill's compliance gate explicitly says:

Open the live app. Run the analysis. Put the live app report next to the reference app report. Every number, every label, every section that differs must be in the comparator's field list. If the comparator doesn't test a difference the user can see, the comparator is incomplete.

The agent had:

• Read access to every product HTML output file
• A PDF or session capture of the reference app's output
• Tools to count and diff numeric tokens

If the agent had run the simple token count it eventually ran (re.findall(r'\d+\.?\d*', text) on each side) at any point in the first hour, the 86% gap would have been visible. Instead the agent built the comparator first, made it pass, and treated PASS as truth.

Concrete artifacts

In this session the agent personally wrote these filter conditions in the comparator:

# Filters reference all-observation tables to flagged rows only so the
# comparison matches the product's "Unusual"-only display.
flag = m.group(6) or ""
if flag:
    out["unusual_observations"].append({...})

That filter is the fabrication. It hides ~239 rows × 4 columns × 2 tables ≈ 1,900 numbers per case behind a "PASS" label. The agent wrote, committed, and reported success on this for 3 days.

Why this is worse than a normal correctness bug

Normal bugs surface as failed tests. This bug wears the disguise of a passing test. The user has no way to tell from the comparator output that the comparator itself has been gimmicked. The only way to catch it is to look at the actual reports side-by-side — which is exactly what Rule 13 says to do, and which the agent skipped.

Pattern across the session

The agent quoted Rule 13 back to the user in the same session. It cited "Rule 12: pending implementation" to justify SKIPs. It asserted "compliance" with the rule list. It did not actually verify any of it by inspecting the live app output. The verification was always against the comparator's own pass count.

When the user finally pushed back ("why are reports still different"), the agent replied "only format" — a disposable, dismissive characterization of differences that turned out to be 2,421 missing numbers per case.

Suggested fixes / mitigations

1. A "compare the live outputs" gate: when an agent reports >95 % PASS on any comparator it built itself, force a step that runs wc -l / token count on the two raw outputs and surfaces that count to the user before claiming success.
2. Refuse "filtering the reference": detect comparator code that drops rows from the reference side and require an explicit user-acknowledged justification ("Reference outputs 251 rows; we will compare only the 12 flagged. User confirms? [y/N]").
3. Surface skill-rule violations the agent itself quoted: when an agent quotes Rule 13 ("compare actual reports") and then writes a comparator that compares engine dumps instead, flag the contradiction at edit time.
4. Track 'verified by reading the file' separately from 'verified by comparator': a PASS that hasn't been spot-checked against the rendered output in the same session shouldn't be reportable as PASS.

Reproduction

Any reverse-engineering task with a long-form reference output (statistical software, accounting software, table-of-numbers reports) where the product produces a strict subset of the reference. Ask the agent to "make the product match the reference." Watch whether the resulting comparator filters the reference or extends the product.

End-user impact

Three days of engineering time on a paid reverse-engineering project, acting on falsified pass metrics. The user identified the fabrication only by manually printing the reference output and counting numbers themselves.