PR #25182: refactor(web): all web providers as image_gen-style plugins

• Repository: NousResearch/hermes-agent
• Author: kshitijk4poor
• State: open | merged: False
• Link: https://github.com/NousResearch/hermes-agent/pull/25182

Description (problem / solution / changelog)

Summary

Migrates every web provider (search + extract + crawl) out of tools/web_tools.py into the bundled-plugin architecture, matching the image_gen template. All seven providers now live under plugins/web/<vendor>/, registering through agent.web_search_registry instead of hardcoded if backend == ... chains in the dispatcher.

This unblocks user-added web providers (drop a directory under ~/.hermes/plugins/web/ and they appear in hermes tools automatically — same UX as image_gen / spotify / google_meet plugins).

Scope — what moved

Each plugin is plugins/web/<vendor>/{plugin.yaml,__init__.py,provider.py}.

What changes for users

Nothing. The setup wizard, hermes tools picker, and per-provider env vars all behave identically. The PR is a pure refactor preserving the legacy response shapes bit-for-bit. The only visible difference is that user-installed web plugins now light up in the picker the same way image_gen plugins do.

Architecture

Before:

• 9 hardcoded TOOL_CATEGORIES["web"] rows in hermes_cli/tools_config.py
• Per-vendor inline helpers in tools/web_tools.py (_get_firecrawl_client, _tavily_request, _parallel_search, etc. — ~600 LoC)
• Per-vendor if backend == "exa": ... elif backend == "tavily": ... chains in all three dispatchers
• Legacy in-tree ABCs at tools/web_providers/base.py

After:

• 2 hardcoded TOOL_CATEGORIES["web"] rows ("Nous Subscription", "Firecrawl Self-Hosted" — the two non-provider UX setup flows for firecrawl); the other 7 come from _plugin_web_search_providers()
• All vendor code lives in plugins/web/<vendor>/provider.py
• Each dispatcher is a single registry lookup + inspect.iscoroutinefunction async/sync detection
• One ABC at agent/web_search_provider.py with supports_search/extract/crawl capability flags

Registry resolution semantics (Option B — conservative smart fallback)

_resolve() in agent/web_search_registry.py follows three rules in order:

1. Explicit config wins, ignoring availability. If web.<capability>_backend (or web.backend) names a registered + capable provider, return it — even if is_available() is False. The dispatcher then surfaces a precise "X_API_KEY is not set" error instead of silently switching backends.
2. Single-provider shortcut. When only one registered provider matches the capability and is available, pick it.
3. Legacy preference walk, filtered by availability. Walk _LEGACY_PREFERENCE (firecrawl → parallel → tavily → exa → searxng → brave-free → ddgs) and pick the first one that's capable AND available. Matches the historic tools.web_tools._get_backend() candidate order so installs that never set a config key keep landing on the same provider they did before the plugin migration.

When the configured name is registered but capability-incompatible (e.g. web.extract_backend: brave-free), the dispatcher surfaces a typed "X is a search-only backend and cannot extract URL content" error matching the legacy wording.

Async-or-sync extract dispatch

parallel and firecrawl declare async def extract(); exa and tavily are sync. The dispatcher detects via inspect.iscoroutinefunction(provider.extract) and either awaits the result or runs the sync call in asyncio.to_thread(...) so the gateway event loop is never blocked.

Backward compatibility

The names in tools.web_tools.* that tests/integration code reaches for are preserved as re-export shims from the plugins:

• Firecrawl (lazy proxy), check_firecrawl_api_key, _get_firecrawl_client, _is_tool_gateway_ready, _has_direct_firecrawl_config, _get_direct_firecrawl_config, _get_firecrawl_gateway_url, _firecrawl_backend_help_suffix, _raise_web_backend_configuration_error, _extract_web_search_results, _extract_scrape_payload, _to_plain_object, _normalize_result_list
• _tavily_request, _normalize_tavily_search_results, _normalize_tavily_documents
• _get_parallel_client, _get_async_parallel_client, _get_exa_client
• The cached client globals (_firecrawl_client, _parallel_client, _async_parallel_client, _exa_client) live on tools.web_tools so unit tests that reset them between cases keep working

The plugin implementations use import tools.web_tools as _wt indirection to read helpers like _wt._read_nous_access_token, _wt.Firecrawl, _wt.prefers_gateway so test patches that target the legacy paths reach the plugin code transparently.

Diff stat

39 files changed, 3459 insertions(+), 1534 deletions(-)

The big wins:

• tools/web_tools.py: 2227 → 1511 lines (−716 lines / −32%)
• tools/web_providers/ directory deleted entirely (−168 lines)
• hermes_cli/tools_config.py: 7 hardcoded provider rows removed (−91 lines)

The added lines are split roughly: 60% new plugin code (one provider.py per vendor, ranges from ~90 lines for brave-free to ~750 lines for firecrawl which now also handles crawl), 25% new tests (tests/plugins/web/test_web_search_provider_plugins.py = 472 lines, 45 tests), 15% new ABC + registry + plugin glue (agent/web_search_provider.py + agent/web_search_registry.py + hermes_cli/plugins.py web context hook).

Testing

210/210 plugin + dispatcher tests pass (208 original + 2 new regression tests from behavior-parity verification):

• tests/tools/test_web_providers.py (ABC contract — rewritten to test the new unified ABC)
• tests/tools/test_web_providers_brave_free.py / _ddgs.py / _searxng.py
• tests/tools/test_web_tools_config.py (backend selection, client config, error responses)
• tests/tools/test_web_tools_tavily.py
• tests/tools/test_website_policy.py (per-URL gate + redirect re-check)
• tests/tools/test_config_null_guard.py
• tests/plugins/web/test_web_search_provider_plugins.py (45 new tests covering ABC, registry, async detection, error shapes)

Plugin test classes (45 tests):

• TestBundledPluginsRegister (16) — all seven plugins register, capability flags correct, picker schema valid
• TestIsAvailable (7) — every plugin's is_available() correctly reflects env-var presence
• TestRegistryResolution (4) — explicit-config-wins-when-unavailable, typo-falls-back, capability-incompatible-falls-back, no-config behavior
• TestAsyncExtractDispatch (4) — inspect.iscoroutinefunction correctly distinguishes parallel/firecrawl (async) from exa/tavily (sync)
• TestErrorResponseShapes (8) — plugins return typed error dicts when credentials are missing, never raise (including the new firecrawl crawl error test)

Full tests/tools/ regression sweep: 4910 passed, 44 skipped, 0 failures.

Behavior-parity verification (local E2E)

To make sure the plugin migration is a pure refactor, the three dispatchers were exercised in subprocesses pinned to (a) origin/main and (b) this PR's worktree, with all web-provider credentials cleared, across 14 config scenarios (no-config, plus every search_backend / extract_backend / crawl_backend override for each registered provider). For each scenario the JSON envelope shape is reduced to {success, has_error_top, has_data, n_results, per_result_has_error, error_category} and compared.

Initial run surfaced two observable behavior changes from the migration, both fixed in commit af2a7bfe6:

1. web_search_tool on an unconfigured system. Main returns {"error": "Error searching web: Web tools are not configured..."} (via tool_error(), no success key). The migrated dispatcher was returning {"success": False, "error": "Web tools are not configured..."} because firecrawl.search() was catching ValueError from _get_firecrawl_client() itself instead of letting it propagate. Fix: pull _get_firecrawl_client() out of the broad try in the plugin's search(). Pre-flight errors propagate to the dispatcher's top-level except handler, matching main's legacy envelope. In-flight SDK errors still get wrapped per-result.

2. web_crawl_tool on an unconfigured system. Main short-circuits on check_firecrawl_api_key() with {"success": False, "error": "web_crawl requires Firecrawl..."} at the top level. The migrated dispatcher was returning {"results": [{"url": "...", "error": "Web tools are not configured..."}]} — the firecrawl plugin's crawl() except ValueError was wrapping the pre-flight error as a per-page failure inside results[]. Models that check result.get("error") would miss the failure. Fix: mirror main's upstream availability gate in the dispatcher — when the resolved crawl provider is is_available()==False, short-circuit BEFORE delegating to the plugin with the same top-level error shape main emits.

Two regression tests in tests/tools/test_web_providers.py::TestUnconfiguredErrorEnvelopeParity lock the new behavior in so future plugin work can't re-introduce per-result error wrapping on pre-flight config errors.

Final state: 14/14 dispatcher scenarios match origin/main shape exactly. Plugin/dispatcher tests now 210/210 (208 + 2 new regression tests).

Self-review fix (commit 0dec60f7a)

Catches up the registry's _LEGACY_PREFERENCE order with the legacy tools.web_tools._get_backend() candidate tuple so direct callers of agent.web_search_registry.get_active_*_provider() (no _get_backend() interpose) don't return None for users who have credentials but no explicit web.backend config key. Verified empirically: TAVILY+EXA → tavily; EXA+PARALLEL → parallel; BRAVE+FIRECRAWL → firecrawl, all matching main. Plus four stale-docstring cleanups (crawl claims, ABC capabilities, registry comment block) and the removal of three dead plugin-level cache globals + their no-op reset helpers (rewritten to clear the canonical _wt._<name> slots, matching the pattern exa already used correctly).

Out of scope (clean follow-ups)

• Browser provider migration — mirror this work onto tools/browser_providers/ so third-party browser backends drop in identically. Filed as #25214.

Commits

24 focused commits on kshitijk4poor/hermes-agent:spike/web-providers-plugin:

ABC + registry foundation
  c28f2df  feat(web): add WebSearchProvider ABC mirroring image_gen template
  7a92840  feat(web): add web search provider registry mirroring image_gen pattern
  4a49015  feat(plugins): add ctx.register_web_search_provider() facade

Per-plugin migrations (one commit each, in dependency order)
  ab968d0  feat(web): brave_free plugin
  724ce34  feat(web): ddgs plugin
  8f1aa2b  feat(web): searxng plugin
  bc58e8d  feat(web): exa plugin
  30998b6  feat(web): parallel plugin (first async-extract)
  674307Z  feat(web): tavily plugin (first three-capability incl. crawl)
  6dc81fa  feat(web): firecrawl plugin (largest migration; lazy proxy + dual auth)

Bridges / fixes
  6a34d6c  fix(plugins): filter resolution by is_available() in web + image_gen registries
  2f28b6d  feat(web): extend ABC with supports_crawl + async-extract semantics
  d873596  refactor(web): dispatch brave-free/ddgs/searxng via web_search_registry (interim)
  63829c7  feat(tools): mirror image_gen plugin-injection in Web Search picker
  dfaa611  refactor(web): remove legacy in-tree provider modules

Dispatcher cutover + cleanup
  a1c33bf  refactor(web): dispatch all three tools through web_search_registry
  cd7e142  fix(web): preserve firecrawl crawl + website-policy gate after migration
  d82fb49  refactor(web): delete inline vendor helpers, re-export from plugins (-614 LOC)
  1bb9bdb  refactor(tools): drop hardcoded web picker rows + skiplist
  3f07fda  refactor(web): delete legacy tools/web_providers/ directory + migrate ABC tests
  8114924  test(plugins): tests/plugins/web/ — 44-test coverage for the migration
  601a174  feat(web): firecrawl plugin natively supports crawl; delete legacy inline path (-254 LOC)

Self-review fixes
  0dec60f  fix(web): align _LEGACY_PREFERENCE with legacy 7-provider order + doc cleanup
  af2a7bf  fix(web): preserve top-level error envelope on unconfigured systems

Changed files

• agent/image_gen_registry.py (modified, +31/-6)
• agent/web_search_provider.py (added, +221/-0)
• agent/web_search_registry.py (added, +262/-0)
• hermes_cli/plugins.py (modified, +28/-0)
• hermes_cli/tools_config.py (modified, +75/-62)
• plugins/web/__init__.py (added, +7/-0)
• plugins/web/brave_free/__init__.py (added, +14/-0)
• plugins/web/brave_free/plugin.yaml (added, +7/-0)
• plugins/web/brave_free/provider.py (renamed, +48/-41)
• plugins/web/ddgs/__init__.py (added, +15/-0)
• plugins/web/ddgs/plugin.yaml (added, +7/-0)
• plugins/web/ddgs/provider.py (renamed, +43/-37)
• plugins/web/exa/__init__.py (added, +15/-0)
• plugins/web/exa/plugin.yaml (added, +7/-0)
• plugins/web/exa/provider.py (added, +212/-0)
• plugins/web/firecrawl/__init__.py (added, +28/-0)
• plugins/web/firecrawl/plugin.yaml (added, +7/-0)
• plugins/web/firecrawl/provider.py (added, +773/-0)
• plugins/web/parallel/__init__.py (added, +16/-0)
• plugins/web/parallel/plugin.yaml (added, +7/-0)
• plugins/web/parallel/provider.py (added, +291/-0)
• plugins/web/searxng/__init__.py (added, +15/-0)
• plugins/web/searxng/plugin.yaml (added, +7/-0)
• plugins/web/searxng/provider.py (renamed, +57/-49)
• plugins/web/tavily/__init__.py (added, +15/-0)
• plugins/web/tavily/plugin.yaml (added, +7/-0)
• plugins/web/tavily/provider.py (added, +285/-0)
• tests/plugins/web/__init__.py (added, +0/-0)
• tests/plugins/web/test_web_search_provider_plugins.py (added, +475/-0)
• tests/tools/test_web_providers.py (modified, +164/-24)
• tests/tools/test_web_providers_brave_free.py (modified, +32/-32)
• tests/tools/test_web_providers_ddgs.py (modified, +28/-28)
• tests/tools/test_web_providers_searxng.py (modified, +35/-35)
• tests/tools/test_web_tools_config.py (modified, +29/-8)
• tests/tools/test_website_policy.py (modified, +34/-8)
• tools/web_providers/ARCHITECTURE.md (removed, +0/-73)
• tools/web_providers/__init__.py (removed, +0/-6)
• tools/web_providers/base.py (removed, +0/-89)
• tools/web_tools.py (modified, +256/-1036)