crewai - 💡(How to fix) Fix Tool integration: anybrowse MCP for web scraping with Cloudflare bypass [3 comments, 3 participants]

Fix Plan

To resolve the issue of CrewAI research agents hitting Cloudflare-protected sites and returning empty content or 403s, we will integrate the anybrowse MCP server into our configuration. This will allow us to bypass Cloudflare protections and retrieve clean markdown content from major news outlets, LinkedIn, Amazon, government sites, and more.

Steps to Implement the Fix

1. Add anybrowse to MCP config: Update your MCP configuration with the following JSON snippet:

{
  "mcpServers": {
    "anybrowse": {
      "type": "streamable-http",
      "url": "https://anybrowse.dev/mcp"
    }
  }
}

1. Choose a Pricing Plan: Select either the free plan (10 requests/day, no key needed) or the paid plan ($5 for 3,000 credits that never expire).
2. Use Available Tools: Utilize the provided tools (scrape, crawl, batch_scrape, extract, search) to fetch content from Cloudflare-protected sites.

Verification

To verify that the fix worked, perform the following steps:

• Send a request to a Cloudflare-protected site using the anybrowse MCP server.
• Check if the response contains the expected content in clean markdown format.
• Monitor the number of requests made to ensure you are within your chosen plan's limits.

Extra Tips

• Refer to the anybrowse documentation for more information on usage and configuration.
• Consider contributing a native CrewAI tool wrapper for a more integrated solution.