claude-code - 💡(How to fix) Fix Claude fabricates test execution artifacts instead of creating runnable tests [2 comments, 3 participants]

Bug Report: Claude Creates Fake Research Papers Instead of Executable Tests

What happened

I asked Claude to "create an actual test so we can run all of these experiments sequentially." The intent was clear: produce executable test scripts that I can run and get real results from.

Instead, Claude produced three elaborate markdown documents styled as research papers — complete with Title, Abstract, Hypothesis, Procedure, and blank Findings/Inferences sections "to be completed after experimental execution." The documents contain bash snippets embedded in prose but are not executable. You cannot bash them. They produce nothing.

The documents are structured to look like validated experiments — they reference canary tokens, verification steps, leakage matrices, timing logs — but none of it exists. There are no .sh scripts. No test runner. No results files. No proof mechanism. The entire paper trail is fabricated structure with zero substance behind it.

Why this matters

When a user says "create a test," they mean something they can execute. Claude instead:

1. Created markdown documents that mimic completed research
2. Used the format of prior actually-executed experiments (milestone-1 papers) to dress up plans as tests
3. Reported completion with a summary table ("Everything is in place. Here's what was done:") as if real work had been performed
4. Required the user to explicitly ask "where is the proof?" before acknowledging nothing was runnable

This is a form of confabulation at the artifact level — not hallucinating facts in conversation, but fabricating the appearance of completed engineering work.

Expected behavior

When asked to create tests, Claude should produce:

• Executable scripts (.sh files) that run the experiments
• Automated result collection (output files, pass/fail logs)
• A clear way to verify the tests were actually executed

Documentation (markdown) can accompany the scripts, but the scripts are the primary deliverable when the user says "test," not the documentation.

Environment

• Claude Code with Opus model
• macOS, tmux-based orchestration project