langchain - 💡(How to fix) Fix Tool integration: anybrowse for Cloudflare-bypass web scraping in LangChain agents [3 comments, 3 participants]

Fix Plan

To resolve the issue of LangChain web loaders failing on Cloudflare-protected sites, we can integrate the anybrowse tool as a BaseTool or BaseLoader. Here are the concrete steps:

• Option 1: Using anybrowse as a BaseTool
  1. Install the requests library if not already installed: pip install requests
  2. Use the provided scrape_url function as a tool in LangChain
  3. Example usage:

from langchain import LLMChain, PromptTemplate

template = PromptTemplate( input_variables=["url"], template="Scrape {url} and return the markdown.", )

chain = LLMChain( llm=None, # Use a suitable LLM prompt=template, tool=tool(scrape_url), )

output = chain({"url": "https://example.com"}) print(output)

* **Option 2: Using `anybrowse` as a `BaseLoader`**
  1. Install the `requests` library if not already installed: `pip install requests`
  2. Use the provided `AnybrowseLoader` class as a document loader in LangChain
  3. Example usage:
  ```python
loader = AnybrowseLoader("https://example.com")
docs = loader.load()
for doc in docs:
    print(doc.page_content)

Verification

To verify that the fix worked, test the scrape_url function or the AnybrowseLoader class with a Cloudflare-protected URL and check if the returned markdown is correct.

Extra Tips

• Make sure to handle errors and exceptions properly when using the anybrowse tool or loader.
• Consider implementing a retry mechanism to handle temporary failures.
• Be aware of the usage limits and costs associated with the anybrowse API.