hermes - 💡(How to fix) Fix Feature request: Oxylabs AI Studio as a web provider (search + extract + crawl)

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
RAW_BUFFERClick to expand / collapse

What

Add Oxylabs AI Studio as a built-in WebSearchProvider backing web_search, web_extract, and web_crawl. Single vendor, single API key, all three capabilities.

Why

Today no bundled provider covers all three slots. Capability matrix of current built-ins:

CapabilityFirecrawlTavilyExaParallelSearXNGDDGSBrave-FreexAI
search
extract
crawl

Oxylabs AI Studio adds a row that has ✓ in all three columns. The vendor's distinguishing features:

  • JS rendering as a per-call flag (render_javascript) on extract and crawl.
  • Geo-targeting (geo_location like "Germany") on extract and crawl.
  • Natural-language instructions for crawl (user_prompt), with parity to Tavily's instructions and strict superset of Firecrawl which silently drops them.
  • Output format negotiation per call (markdown / json / csv / toon / screenshot for scrape).
  • Native async for scrape and crawl (sync for search).

Capability mapping (proposed)

Oxylabs appSDK classHermes slotNotes
AI-SearchAiSearchweb_search.search() + .instant_search(); map to {title, url, description, position}
AI-ScraperAiScraperweb_extractPer-URL .scrape_async() with NL schema + output_format
AI-CrawlerAiCrawlerweb_crawlSeed-URL traversal with user_prompt and return_sources_limit
Browser AgentBrowserAgent(no ABC slot today)v2 — needs ABC extension or sibling plugin
AI-MapAiMap(no ABC slot today)v2 — URL discovery without extraction

Initial scope is the three ABC slots. Browser Agent and AI-Map are v2 follow-ups.

Implementation status

A plugins/web/oxylabs/ directory is ready (provider implementation, plugin.yaml, lazy SDK import per Exa pattern, response-shape normalization, integration tag for vendor-side attribution). A 36-test pytest suite covers ABC contract, capability flags, kwarg forwarding, response normalization, and live API calls when OXYLABS_AI_STUDIO_API_KEY is set.

All three capabilities verified end-to-end against a public site, with Oxylabs credit spend confirmed for search, extract, and crawl independently.

The plugin depends on two pre-existing core gaps being closed. Both are tracked in separate issues filed alongside this one:

After those land, the Oxylabs PR is small: the plugin directory plus a one-line tools/lazy_deps.py entry plus three additive whitelist extensions in tools/web_tools.py (or, if the maintainers prefer, the registry-based refactor proposed in #31873 instead of additive whitelist extensions).

Open questions

  1. Acceptable as a new built-in provider, or would maintainers prefer it as an external community plugin?
  2. Preferred shape for the tools/web_tools.py whitelist extensions — additive (matching how xai, brave-free, ddgs were added) or registry-based refactor per #31873?
  3. Inclusion of web_crawl in the hermes-acp (editor-integration) toolset — yes by default, or opt-in?

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING