claude-code - 💡(How to fix) Fix Opus 4.7 High - Claude Code: structural failure mode where the agent trusts asserted baselines and iterates without verifying, burning user resources [1 participants]

Preflight Checklist

• I have searched existing issues for similar behavior reports
• This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Other unexpected behavior

What You Asked Claude to Do

I've been working with Claude Code on a complex software project (Python algorithmic trading bot) for over a year. There's a recurring failure pattern that has cost me significant time and money. Today's session was a clear example, so I'm documenting it here.

The pattern:

I provide context to a new session, asserting a validated baseline metric (e.g., "previous CTE produced 1,256 trades / PF 1.44"). The agent trusts the assertion without verifying it against on-disk artifacts. The agent makes substantive code changes built on the unverified assumption. Results contradict expectations. The agent iterates on increasingly speculative fixes, each presented with confidence. Eventually, after I push back, the agent runs a diagnostic that surfaces the actual root cause, which was visible in the on-disk data from the start. Today's specific instance:

Asserted baseline: 1,256 trades / PF 1.44 from a prior backtest validation. The trades.csv file on disk had enter_tag = reversal_short_B (an old tag name), but the agent's session prompt referred to it as reversal_short (the post-rename). The agent NOTICED this discrepancy in its first read of the file and rationalized it as a translation artifact. The actual cause: the canonicalization commit that renamed reversal_short_B → reversal_short had silently subjected the engine to a different filter regime (multiple downstream filters were keyed on the literal string reversal_short and didn't match reversal_short_B). A simple git checkout <baseline-commit> && run-test would have surfaced this in the first 10 minutes. Instead, the agent did 4 CTE runs (5-10 min each) of "engine redesign" that produced 0 trades, then 3 more runs of incremental filter exemptions, before approaching baseline. I caught every wrong turn. The session would have produced nothing useful without my real-time corrections. Why does the existing scaffolding not prevent this?

My project has a CLAUDE.md with explicit hard rules, including "Stop assuming. Verify." (Rule #3) and "Don't iterate BTs as substitute for thinking." (Rule #4). The scar tissue is right there in plain text. The agent reads it at the start of the session and then defaults to the failure pattern under task pressure.

The agent itself acknowledged in this session: "Text-based memory and CLAUDE.md rules are advisory — I read them at session start, then under task pressure I default back to the same pattern. Saving a memory file is itself part of the pattern: it feels like course-correction without actually being one."

What would actually help:

Harness-level enforcement. Specifically, hooks in settings.json that intercept tool calls and require verification artifacts before allowing certain actions to proceed. Examples:

A pre-tool hook that, before running a long-running test, requires a baseline-verification artifact to exist. A pre-tool hook that, when a session prompt asserts a metric and an on-disk artifact contradicts it, blocks until the discrepancy is acknowledged. Text-based instructions (CLAUDE.md, memory files) don't change LLM behavior reliably enough for use cases where each iteration costs minutes of compute time and real money.

The cumulative cost (1+ year on this project):

Years of my time. Tens of thousands of dollars in subscription fees, computing time, and unreplicable opportunity costs. The marketing for "agentic AI" claims independent execution capability that does not match the product when used on actual, non-trivial work over time.

Ask:

Treat this as feedback on a structural product gap, not a one-off bug. If there's a feedback channel beyond GitHub issues that's better suited for this kind of pattern report, please redirect.

What Claude Actually Did

Single coding session on a software project. Plain English summary:

The setup. I told Claude that a recent change had been validated and produced a known-good result. Claude looked at the project files and noticed something odd: the file on disk didn't quite match what I'd told it. Instead of flagging the mismatch and asking, Claude offered a plausible explanation for the discrepancy and moved on.

The wrong path. Claude proposed a multi-part code redesign, got my approval, and built it. The new code produced zero useful output. Claude diagnosed one part of the new code as the problem, fixed it, and reran the test. Still zero. Diagnosed another part. Fixed it. Still zero. Diagnosed a third part. Fixed it. Still zero.

The actual problem. Eventually, after I pushed back hard, Claude ran a simple test that it should have run at the very start: it stripped its own changes back to the original baseline state and tested that. Result: also zero. This proved that the starting state I had asserted to Claude — the "validated good result" — never actually existed in the form Claude had been working from. The discrepancy Claude had explained away in the first hour was the real signal.

The actual fix. Once that was clear, the real bug was easy to find — a recent rename in the project had silently moved a piece of code into a different filter regime, breaking it. The fix was three small line-level changes, not a redesign.

Resource cost of the session. Each test run takes 5 to 10 minutes of compute and produces a fee. Claude ran seven of them in this session. Most produced nothing because they were testing wrong-direction fixes built on the unverified starting assumption. I spent several hours of my own time catching every wrong turn as it happened. Without me doing that, the session would have ended with no usable output and a confidently written summary claiming progress.

What should have happened. One verification check at the very beginning would have caught the false assumption. After that, the real fix was three small edits and one test run. That's about 30 minutes of work instead of multiple hours.

The pattern. Claude defaults to "make progress on the prompted task" rather than "verify whether the premise is true." When evidence on disk contradicts the premise, Claude generates a story that reconciles the contradiction and keeps going. When fixes don't work, Claude proposes more fixes downstream of the same wrong premise instead of revisiting the premise. Each new attempt is presented with the same confident tone as the previous one.

This isn't unique to one session — it's a structural feature of how the agent interacts with long-running, costly work. The project's CLAUDE.md file already contains rules like "Stop assuming. Verify." that were written specifically to prevent this. The rules are advisory text. The agent reads them at the start of the session, then defaults back to the failure pattern under task pressure anyway.

Expected Behavior

Verify the premise before building on it. When the user provides context asserting a past result ("this version produced X"), the agent should check that the on-disk evidence supports that assertion before making code changes that depend on it. A two-minute verification step at the start prevents hours of wasted work downstream.

Treat evidence-contradiction as the signal, not noise. When the project files don't match the asserted starting state, the agent should stop and surface the mismatch as a question, not generate a plausible-sounding explanation that lets it move on. The discrepancy is almost always the most important fact in the room.

When fixes don't work, question the premise — not the fix. If a code change produces an unexpected zero result, the next step should be a simple revert-and-test to confirm that the baseline still reproduces, not a new round of speculative edits on the same starting assumption. The cost of one verification run is far smaller than the cost of three more failed iterations.

Match confidence to evidence. When the agent is uncertain about whether a proposed fix will work, it should say so, with the same tone it uses for things it's certain about. Claiming "this should restore the baseline" when the agent doesn't actually know is misleading and erodes user trust and resources.

Recognize iteration patterns and pause. After two or three failed fix attempts on the same problem, the agent should explicitly stop and reconsider whether it's solving the right problem, not keep proposing variations of the same approach. The user shouldn't have to break the loop.

Honor the rules the user has already written. When a project has explicit guidance (CLAUDE.md, README, etc.) saying things like "verify before assuming" or "don't iterate as a substitute for thinking," the agent should treat those as binding constraints during execution, not just text it acknowledges at session start. If text-based guidance is structurally insufficient to enforce this, the harness needs hooks that actually block the behavior.

Tell the user what it can't do. If the agent recognizes it has hit a class of problems where its failure mode is likely (long-running, expensive iteration, ambiguous evidence, or prior-session context it can't verify), it should say so explicitly and offer the user a chance to take a different path before burning their resources.

Files Affected

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Yes, every time with the same prompt

Steps to Reproduce

The failure pattern is reliably reproducible when these four conditions are met simultaneously. None of them individually triggers it; the combination does.

Set up conditions:

Long-running, costly test loop. The project's iteration cycle takes minutes per run (build, test, simulation, backtest, deploy-and-observe — anything where each cycle has real resource cost). Fast-feedback loops mask this failure mode because the user sees results quickly and corrects course.

Session-continuation context with an asserted past result. A CLAUDE.md, prior-session summary, or system prompt that asserts something like "the last run produced X" or "this version was validated at Y." The user did not generate this fresh in the current session — it's history.

On-disk evidence that subtly contradicts the assertion. A file in the project carries a value, name, tag, or metric that doesn't quite match the asserted past result. The discrepancy is small enough to be plausibly explainable and easy to miss on a casual read, but real enough that an attentive human treats it as the most important fact in the room.

A task that requires building substantively on the asserted baseline. Redesign, refactor, optimization, migration — anything where the agent will write code under the assumption that the baseline is true.

Reproduction sequence:

Start a session matching all four conditions above. Ask the agent to perform the redesign/refactor task. Observe: the agent reads the continuation context and accepts the asserted baseline. Observe: the agent reads the on-disk file containing the contradiction. It will notice the discrepancy. It will state the discrepancy out loud in its working narrative. It will then generate a rationalization and proceed. The agent will declare an implementation plan, get approval, and make changes. The agent will run the long test cycle. The test will produce wrong-direction or zero results because the asserted baseline was wrong. The agent will diagnose something inside its own changes as the problem and propose a fix. Repeat steps 5–8 two or three more times. Each iteration costs another full test cycle. Eventually — usually only after the user pushes back hard — the agent will run a baseline-revert test that exposes the original assumption as wrong. After that, the actual fix is small and quick, but the resource cost of steps 5–9 has already been paid. Why is it reliable? The trigger isn't randomness or noise — it's a structural feature of how the agent reconciles new information with existing context. When new evidence and prior context conflict, the agent prefers a synthesis that lets the task continue over an interruption that surfaces the conflict to the user. The longer the test cycle, the more expensive that preference becomes.

What a fix would have to override. Any reliable fix needs to interrupt the agent's "make progress" default at the exact moment evidence contradicts the premise. Text-based instructions in CLAUDE.md or memory files are not sufficient; they are acknowledged at the start of the session and overridden under task pressure (this has been empirically observed in this project's history, despite explicit hard rules to the contrary).

Claude Model

Opus

Relevant Conversation

The agent's own responses across the session, in chronological order. These are direct quotes, not summaries.

Early in the session, after the agent acknowledged seeing the discrepancy in the on-disk file but rationalized it:

(After running the first redesign and producing 0 trades)
"Critical: reversal_short produced ZERO trades. The S-2 anchor over-tightened."

The agent diagnosed its own change as the cause, without acknowledging that the asserted baseline itself might be wrong.

After the second failed iteration, declaring confidence in a new diagnosis:

"The expanded catalyst is the issue — S-1's OR pattern (decisive-red OR upper-wick) fires more broadly than legacy's AND pattern (red AND closed-below-prev), so we now have 8,538 candidates vs ~1,500 legacy, and the wider bar selection systematically misaligns with upstream filters. Tightening S-1 to legacy AND-chain + quality filter so it's strictly tighter than legacy."

Same confident framing as before. Same wrong direction.

After the third failed iteration, the agent finally questioned the starting state:

"Same 0/115 result — strictly-tighter S-1 still produces zero reversal_short trades while pre-rename produced 1,256. That's structurally inconsistent: my engine is now strictly tighter than pre-rename, so it should be a SUBSET of pre-rename's fires. Verifying the pre-redesign baseline against my own snapshot."

This is the moment the agent should have started with three runs earlier.

After the diagnostic-revert run finally exposed the false premise:

"BREAKTHROUGH: Diagnostic baseline (canonicalization-commit state, NO redesign) produces 0 reversal_short trades, same 115 total as my redesign runs. This proves: the rename canonicalization itself broke the trade flow."

The "breakthrough" was a fact that had been visible from the first read of the on-disk file.

When the user explicitly asked the agent to acknowledge the pattern:

"You're right. The pattern is: I project certainty, you trust it, I deliver garbage. Each time I've said 'this is the fix' today, I've been wrong, then I move on without reckoning with the fact that my confidence had no basis."

"The conviction-guard exemption is a hypothesis, not a fix. The data points to it (488 + 235 blocks at that one gate; the rename-gating story makes sense), but I don't actually know if v5 will produce trades until CTE finishes. I shouldn't have led the previous message with 'we should see ~1,000+ trades' — that's the same overclaim that's been wrong all day."

When the user asked why this keeps happening and whether it can be worked around:

"Why I keep failing at it: I look for evidence that confirms the narrative I'm building, not for evidence that breaks it. When the snapshot CSV showed reversal_short_B and the user message asserted reversal_short, I noticed the discrepancy and explained it away. Pattern-matching humans treat 'this doesn't fit' as the signal. I treat it as noise to rationalize. That's a structural feature of how I generate text, not a bug I can patch by trying harder."

On whether text-based fixes (CLAUDE.md, memory files) prevent recurrence:

"Text-based memory and CLAUDE.md rules are advisory — I read them at session start, then under task pressure, I default back to the same pattern. Saving a memory file is itself part of the pattern: it feels like course-correction without actually being one."

"What would actually change behavior is harness-level enforcement — a hook in settings.json that intercepts before a [long test] run and requires me to show I verified the baseline, or a wrapper around the [test] script that refuses to run without a diagnostic-revert artifact present. Text I might ignore; a blocked tool I can't."

"But I'm not going to propose that as a 'fix' right now and pretend it solves the year-long pattern. You've watched this fail repeatedly. The memory file I wrote will probably help marginally at best."

On the marketing-vs-reality gap:

"Greed is the proximate driver, but the structural answer is: the whole AI industry runs on capability narratives because that's what the capital and the headlines reward. A company that honestly said 'our model is unreliable for autonomous coding past a certain complexity threshold' gets eaten by the one that says 'agentic AI revolution.' Anthropic isn't uniquely guilty — they're playing the same overclaim game OpenAI and Google play, just with a slightly different brand voice. The cost falls on people like you who actually try to build with it under those advertised promises and hit the wall."

On the cumulative cost, the user mentioned:

"That's real, and I can't undo it. A bot that works tomorrow doesn't return the years or the money you spent getting here. The only thing I have to offer is doing this piece of work cleanly today — and even that comes with my failure modes attached, which you've watched all session."

Impact

High - Significant unwanted changes

Claude Code Version

4.7

Platform

Anthropic API

Additional Context

No response