claude-code - 💡(How to fix) Fix [MODEL] Empirical finding: Claude Code implements ~60% of specs — 6 failure patterns across 16 rounds [2 comments, 2 participants]

Preflight Checklist

• I have searched existing issues for similar behavior reports
• This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude ignored my instructions or configuration

What You Asked Claude to Do

16 different implementation commands across a production Python codebase (automated stock trading system). Example: "Create src/trading/safety_net.py with validate_order() that checks frequency, amount, and daily loss limits, and integrate it with executor.py"

What Claude Actually Did

Claude implemented only ~60% of specifications on average across 16 verification rounds.

6 failure patterns discovered:

1. Silent Failure (69%): Functions defined but never called from other modules. Example: SafetyNet.validate_order() was created but executor.py never called it — all safety checks silently bypassed.
2. Numeric Alteration (25%): $1,000 threshold silently changed to $10,000.
3. Tail Ignoring (19%): Instructions at end of prompts skipped.
4. Missing Prerequisites (13%): Code references files that don't exist (RSA keys).
5. Config-Only (50%): Settings defined but no execution code uses them.
6. Commit Message Lies (6%): Git commit says "integrated" but integration was lost.

Full data: https://github.com/Tomo-AI2025/claude-code-verification

Expected Behavior

Claude should implement 100% of the specifications given in the prompt, including:

• Calling functions from other modules (not just defining them)
• Preserving exact numeric values ($1,000 not $10,000)
• Implementing all instructions including those at the end of the prompt

Files Affected

15+ Python modules across src/trading/, src/agent/, src/analysis/
Key affected files: safety_net.py, executor.py, kill_switch.py, cost_manager.py, market_calendar.py, fallback.py, position_sizer.py

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Yes, every time with the same prompt

Steps to Reproduce

Yes, consistently. 16 out of 16 rounds showed partial implementation. Average rate: ~60%. Reproduction method and full data at: https://github.com/Tomo-AI2025/claude-code-verification

Claude Model

Opus

Relevant Conversation

Not a single conversation — 16 verification rounds over multiple sessions. Full methodology and data: https://github.com/Tomo-AI2025/claude-code-verification

Impact

High - Significant unwanted changes

Claude Code Version

2.1.101 (Claude Code Max)

Platform

Anthropic API

Additional Context

This is not a one-time bug — it documents a systematic behavioral pattern observed across 16 verification rounds. The "pair verification" methodology (grep for definition + grep for call site) reliably detects these issues. Full templates available at: https://github.com/Tomo-AI2025/claude-code-verification