llamaIndex - 💡(How to fix) Fix Feature: anybrowse web reader for Cloudflare-protected sites [1 participants]

Fix Plan

To resolve the issue of LlamaIndex web loaders failing on Cloudflare-protected sites, we will integrate anybrowse into the system. This can be achieved by creating an AnybrowseReader that utilizes the anybrowse API to fetch and parse web pages.

Step-by-Step Solution

1. Install required libraries: Ensure you have requests installed. You can install it via pip:

pip install requests

2. **Implement AnybrowseReader**: Create a new file `anybrowse_reader.py` and add the following code:
   ```python
import requests
from llama_index.core import Document

class AnybrowseReader:
    def __init__(self, api_key: str = None):
        self.api_key = api_key

    def read(self, url: str) -> Document:
        headers = {"Content-Type": "application/json"}
        if self.api_key:
            headers["Authorization"] = f"Bearer {self.api_key}"
        r = requests.post("https://anybrowse.dev/scrape", json={"url": url}, headers=headers)
        data = r.json()
        return Document(
            text=data["markdown"],
            metadata={"url": url, "title": data.get("title", "")}
        )

1. Configure AnybrowseReader: Initialize the AnybrowseReader with your anybrowse API key (if you have one). You can then use this reader to fetch documents from Cloudflare-protected sites.

Verification

To verify that the fix worked, you can test the AnybrowseReader with a Cloudflare-protected URL:

reader = AnybrowseReader(api_key="YOUR_API_KEY")
document = reader.read("https://example.com/cloudflare-protected-page")
print(document.text)

Replace "YOUR_API_KEY" with your actual anybrowse API key.

Extra Tips

• Make sure to handle rate limits and errors properly when using the anybrowse API.
• Consider implementing a fallback mechanism in case the anybrowse API is unavailable.
• Refer to the anybrowse documentation for more information on usage and pricing.