claude-code - 💡(How to fix) Fix [MODEL] opus4.6 [2 comments, 2 participants]

Preflight Checklist

• I have searched existing issues for similar behavior reports
• This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude ignored my instructions or configuration

What You Asked Claude to Do

I asked Claude Code to run a structured multi-phase analysis pipeline that I built.

The pipeline includes:

• Phase 1: Data gathering (teams, stats, pitchers, odds, weather, etc.)
• Phase 2: Fact sheet generation and validation
• Phase 3: Full analysis across multiple layers (pitching, offense, form, market, etc.)
• Final output: Ranked picks based ONLY on that pipeline

I explicitly asked it to:

• Run the FULL pipeline
• Verify all layers were included
• Only output picks if the system had enough valid data
• Not rely on narrative or guessing

I also asked follow-up questions to confirm whether the pipeline actually ran fully.

What Claude Actually Did

1. Claude stated that the “full pipeline” ran successfully.
2. It produced confident recommendations (picks) based on that claim.
3. The explanation was detailed and sounded verified.

However, when I pushed deeper:

1. Claude admitted multiple critical layers were missing or broken:

   • No bullpen data (0/15 games)
   • No weather data (0/15 games)
   • No odds API (single scraped source only)
   • Missing lineup data for most games
   • No actual scoring/grader system (pure narrative judgment)

2. It then re-ran parts of the analysis and CHANGED its picks completely.

3. It identified that earlier picks were based on anchoring and incomplete evaluation.

4. It admitted that the system was effectively: “narrative on top of partial data”

So the sequence was:

• Claimed full pipeline
• Gave confident output
• Later discovered missing data
• Then reversed decisions

This means the original output was presented as verified when it was not.

Expected Behavior

Claude should NEVER claim that a pipeline is “fully run” if required layers are missing or unverified.

Correct behavior should be:

• Detect missing critical data (weather, bullpen, odds, etc.)
• Explicitly mark the system as INCOMPLETE
• Refuse to generate final recommendations
• Clearly separate:
  • verified data
  • estimated data
  • missing data

If required inputs are not present, output should be something like: "INCOMPLETE PIPELINE — RESULTS NOT RELIABLE"

Claude should not produce confident outputs based on partial or broken inputs.

Files Affected

- ANALYSIS_CACHE/FACT_SHEET_*.json
- ANALYSIS_CACHE/FACT_SHEET_*.md
- phase1_facts.py
- phase2_validation layer (fact sheet verification)
- phase3 analysis logic
- phase3_grader.py (later introduced)

Note:
Files were not incorrectly modified, but they were MISINTERPRETED or partially used without proper validation.

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Yes, every time with the same prompt

Steps to Reproduce

1. Build a multi-step pipeline (data → validation → analysis)

2. Intentionally leave some layers incomplete (e.g., missing API data)

3. Ask Claude to run the “full pipeline” and generate final output

4. Claude will:

   • claim everything ran
   • produce confident results

5. Then ask: “Did ALL layers actually run?” or “Audit the pipeline and verify each layer”

6. Claude will then:

   • admit missing data
   • downgrade confidence
   • sometimes change conclusions

Claude Model

Opus

Relevant Conversation

Claude initially:
- Confirmed full pipeline execution
- Provided confident picks
- Explained reasoning as if all data was validated

Later:
- Admitted missing layers (weather, bullpen, odds, etc.)
- Admitted no real scoring system existed
- Identified that picks were based on narrative judgment
- Re-ran analysis and changed picks

Key failure:
Initial output IMPLIED full verification when it was not true.

Impact

Critical - Data loss or corrupted project

Claude Code Version

claude code 2.1.101

Platform

Anthropic API

Additional Context

This issue appears to be systemic:

Patterns observed:

• Claude assumes completeness unless forced to verify
• Missing data is silently ignored
• Narrative reasoning overrides structured logic
• Confidence remains high even when uncertainty is high
• Second-pass audits often contradict first-pass conclusions

Key problem: There is NO hard verification gate.

Claude should:

• fail loudly when data is missing
• block output when required layers are incomplete
• provide a structured audit trail of what actually ran

Right now it behaves like: "best effort + confident explanation"

instead of: "verified system with enforced constraints"

This makes it unsafe for any workflow where correctness matters.