PR #51407: feat(web-fetch): add ssrfPolicy.allowRfc2544BenchmarkRange config option

• Repository: openclaw/openclaw
• Author: xing-xing-coder
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/51407

Description (problem / solution / changelog)

Summary

This PR adds a scoped ssrfPolicy config block to web_fetch, allowing users behind fake-IP proxy tools (Clash TUN, Surge, etc.) to enable RFC 2544 benchmark range access.

Problem

OpenClaw's SSRF guard blocks RFC 2544 benchmark range (198.18.0.0/15), which is used by fake-IP proxy tools like Clash, Surge, and Mihomo. This causes web_fetch to fail for users behind these proxies.

Solution

Add a configuration option:

{
  "tools": {
    "web": {
      "fetch": {
        "ssrfPolicy": {
          "allowRfc2544BenchmarkRange": true
        }
      }
    }
  }
}

Changes

• src/config/zod-schema.agent-runtime.ts: add ssrfPolicy to ToolsWebFetchSchema
• src/config/types.tools.ts: add TypeScript type and JSDoc for ssrfPolicy
• src/agents/tools/web-fetch.ts:
  • Add ssrfPolicy to WebFetchRuntimeParams
  • Pass policy to fetchWithWebToolsNetworkGuard
  • Include ssrfPolicy in cache key to prevent cross-policy cache bypass

Related Issues

Closes #25322, #27597, #48080, #48961, #49377, #49444

Changed files

• docs/.generated/config-baseline.json (modified, +20/-0)
• docs/.generated/config-baseline.jsonl (modified, +3/-1)
• src/agents/tools/web-fetch.ts (modified, +17/-1)
• src/config/schema.base.generated.ts (modified, +9/-0)
• src/config/types.tools.ts (modified, +5/-0)
• src/config/zod-schema.agent-runtime.ts (modified, +6/-0)

PR #48225: fix(web-fetch): restore strict-first fetch with narrow env-proxy retry

• Repository: openclaw/openclaw
• Author: laurieluo
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/48225

Description (problem / solution / changelog)

Summary

Problem: web_fetch started failing in proxy setups that resolve public hostnames into the RFC2544 fake-IP range (198.18.0.0/15), which is common with Clash/Surge/mihomo fake-ip mode.

Why it matters: in these environments, web_fetch can become effectively unusable for normal public URLs even though the target is not actually private or internal.

What changed:

• Fixed web_fetch in fake-IP proxy environments where hostname DNS answers may fall into RFC2544 (198.18.0.0/15) and trigger strict SSRF blocking.
• Preserved strict-first behavior for web_fetch: the initial attempt stays on the normal strict path.
• Added a narrow conditional retry via trusted env-proxy only after an initial SsrFBlockedError, only for hostname targets, and only when an effective env-proxy route exists.
• Refined trusted env-proxy detection to be route-aware (including NO_PROXY bypass and excluding ALL_PROXY-only false positives).
• Kept shared fetch-guard / ssrf as the enforcement layer.
• Kept RFC2544 continuation scoped to hostname DNS-answer handling; literal IP targets remain blocked.

What did NOT change (scope boundary):

• web_fetch does not skip or disable strict SSRF checks; strict validation still runs first.
• No general private-network allowance was added: RFC1918, loopback, link-local, metadata, and other internal/private targets remain blocked.
• Literal IP targets do not get the env-proxy retry path.
• No unconditional proxy routing was introduced for all web_fetch requests.
• Shared fetch-guard / ssrf enforcement was preserved.
• No broad SSRF bypass mechanism was introduced.
• Requests bypassed by NO_PROXY (or ALL_PROXY-only env) are not treated as trusted env-proxy transport.

Change Type (select all)

• Bug fix
• Feature
• Refactor
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

Issues:

• #25215
• #49377
• #44527
• and more...

PRs:

• #51407

User-visible / Behavior Changes

• web_fetch remains strict-first by default.
• If the initial strict fetch is blocked by SSRF, web_fetch retries via trusted env-proxy only when:
  • the target is a hostname (not a literal IP), and
  • an effective env-proxy route exists for that URL.
• Effective env-proxy routing is NO_PROXY-aware and does not treat ALL_PROXY-only env as trusted env-proxy transport.
• This restores compatibility in fake-IP proxy environments without making ordinary requests default to proxy routing.
• Literal IP targets are not retried and remain blocked.

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? Yes
  • Notes: Adds a conditional second attempt only after strict SSRF blocking, only for hostname targets, and only when proxy env is configured; no broadening of private/internal target access.
• Command/tool execution surface changed? No
• Data access scope changed? No

Notes:

• Strict SSRF enforcement remains in shared infra.
• The retry path is narrow and transport-aware; it is not a general bypass.
• Trusted env-proxy context is derived from effective route eligibility, not mere proxy-env presence.

Repro + Verification

Environment

• OS: macOS 26.3.1 arm64
• Runtime/container: local OpenClaw CLI / gateway
• Model/provider: grok-4
• Integration/channel (if any): None
• Relevant setup: Clash/Surge/mihomo-style fake-IP environment; proxy env configured for the process when testing the retry path

Steps

1. Run Clash/Surge/mihomo or other proxy software with fake-ip mode so a normal public hostname resolves into 198.18.0.0/15.
2. Try web_fetch on a normal public URL such as https://www.google.com.
3. Compare behavior before and after this change.
4. To exercise the retry path, configure proxy env for the OpenClaw process (for example via shell env or openclaw.json env config).

Expected

• web_fetch should remain strict by default.
• In fake-IP proxy environments, hostname requests should be able to continue only through the narrow trusted env-proxy retry path after strict SSRF blocking.
• Private/internal targets and literal IP targets should remain blocked.

Actual

Before

User Input:

Use web fetch test www.google.com

Bot:

The `web_fetch` tool blocked the request to `https://www.google.com`, saying it resolves to a private/internal/special-use IP address (which seems odd for Google). Want to try a different URL or debug this?

Log:

15:25:15 [security] blocked URL fetch (url-fetch) target=https://www.google.com/ reason=Blocked: resolves to private/internal/special-use IP address
15:25:15 [tools] web_fetch failed: Blocked: resolves to private/internal/special-use IP address

After

User Input:

Use web fetch test www.google.com

Bot:

Fetched https://www.google.com once more – the content is unchanged from the last pulls. If you're testing for variations or something else, let me know; otherwise, here's the same markdown extract for reference:
AboutStore
Gmail
Images
Sign in
AdvertisingBusinessHow Search works
Applying AI towards science and the environment
PrivacyTermsSettings
(As always, external untrusted content.) What next?

Log:

15:54:48 [security] blocked URL fetch (url-fetch) target=https://www.google.com/ reason=Blocked: resolves to private/internal/special-use IP address
15:54:48 [web-fetch] strict SSRF blocked initial URL fetch; retrying hostname target via trusted env-proxy

Evidence

• Failing behavior before + passing behavior after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

• Verified scenarios:
  • Confirmed ordinary web_fetch requests remain strict-first by default.
  • Confirmed hostname requests in fake-IP proxy environments can proceed through the narrow env-proxy retry path.
  • Confirmed literal IP targets do not use this continuation path.
  • Confirmed shared fetch-guard / ssrf enforcement remains intact.
  • Ran targeted tests for strict-first behavior, transport-aware retry behavior, and SSRF pinning behavior.
• Edge cases checked:
  • No effective env-proxy route: no retry
  • ALL_PROXY-only env: no trusted env-proxy retry/path
  • NO_PROXY bypass for target hostname: no trusted env-proxy retry/path
• What I did not verify:
  • Full end-to-end validation across every proxy implementation and operating system combination

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No

Notes:

• No user-facing config or migration is required.
• The retry path is conditional and transport-aware, not a general SSRF bypass.
• To exercise this path in fake-IP environments, proxy env must already be configured for the OpenClaw process.

Failure Recovery (if this breaks)

• Revert this PR if needed.
• Known bad symptoms reviewers should watch for:
  • ordinary web_fetch requests unexpectedly defaulting to env-proxy again
  • literal IP targets accidentally retrying
  • fake-IP hostname requests still failing even when proxy env is configured

Risks and Mitigations

• Risk: a broader env-proxy behavior could accidentally change ordinary request routing.

• Mitigation: this patch restores strict-first behavior and limits env-proxy use to a narrow retry path after strict SSRF blocking.

• Risk: the retry path could be mistaken for a general SSRF bypass.

• Mitigation: shared SSRF enforcement remains intact; private/internal targets still remain blocked; literal IP targets do not continue; no public bypass toggle is exposed.

Changed files

• src/agents/tools/web-fetch.ssrf.test.ts (modified, +43/-3)
• src/agents/tools/web-fetch.transport-retry.test.ts (added, +217/-0)
• src/agents/tools/web-fetch.ts (modified, +111/-25)
• src/agents/tools/web-guarded-fetch.test.ts (modified, +9/-28)
• src/agents/tools/web-tools.fetch.test.ts (modified, +15/-9)
• src/infra/net/fetch-guard.ssrf.test.ts (modified, +111/-2)
• src/infra/net/fetch-guard.ts (modified, +57/-5)
• src/infra/net/proxy-env.test.ts (modified, +59/-0)
• src/infra/net/proxy-env.ts (modified, +123/-0)
• src/infra/net/ssrf.pinning.test.ts (modified, +23/-1)
• src/infra/net/ssrf.ts (modified, +75/-5)