claude-code - 💡(How to fix) Fix Claude wrote naive Yelp scraper from scratch ignoring existing battle-tested code — 116 rapid hits, IP blocked [1 comments, 2 participants]

TL;DR

To prevent similar incidents, Claude should be modified to search existing project directories for prior implementations of scrapers before writing new ones.

Guidance

• Implement a directory search function in Claude to check for existing scraper implementations before creating new ones.
• Integrate a delay mechanism in the scraper to ensure a minimum of 10-15 seconds between requests to avoid rapid sequential requests.
• Review and update the Global CLAUDE.md rules to ensure they are clear and enforceable.
• Consider implementing a warning or alert system when Claude attempts to write a new scraper without checking existing code.

Example

import os

def check_existing_scraper(project_dir, scraper_name):
    for root, dirs, files in os.walk(project_dir):
        if scraper_name in files:
            return True
    return False

Notes

This solution assumes that the existing scraper implementations are properly named and organized within the project directories. It's also important to note that this is a partial solution and may require additional modifications to Claude's code to fully address the issue.

Recommendation

Apply workaround: Implement a directory search function in Claude to check for existing scraper implementations before creating new ones. This will help prevent similar incidents in the future by ensuring that existing code is reused and modified instead of starting from scratch.