Summary

When an agent receives a Twitter/X URL (e.g. https://x.com/user/status/123), it consistently reaches for the web_fetch tool first, which always fails on x.com (returns empty page or login wall). This happens despite:

1. Documented SOPs with working alternatives
2. Explicit rules in workspace files (AGENTS.md, TOOLS.md)
3. Logged regressions (5+ incidents across sessions)
4. A dedicated web-search SOP with a fxtwitter routing table

The agent has the tools and knowledge to fetch tweets correctly. The problem is behavioral: the web_fetch tool description ("Fetch and extract readable content from a URL") is the strongest pattern match when a URL appears, and nothing at the tool-description level warns against using it on blocked domains.

What works (but the agent does not reach for)

• fxtwitter API: web_fetch("https://api.fxtwitter.com/<user>/status/<id>") — works instantly, returns full tweet JSON
• xAI x_search: Direct Twitter API via Grok Responses API — works for threads and fresh content
• SearXNG: Local search engine proxy — works for discovery
• xurl CLI: Authenticated X API tool (when configured)

What we tried to fix it (all failed to change behavior)

1. Detailed SOP (ops/playbooks/tools/sops/web-search.md)

• Full routing table: "Specific tweet by ID → https://api.fxtwitter.com/<user>/status/<id>"
• Decision matrix for every URL type
• Explicit "DO NOT give up" rule with fallback order
• Result: Agent does not read the SOP before acting on URLs

2. Rules in AGENTS.md (loaded every session in system prompt)

• Added: "When a URL arrives, check TOOLS.md and the relevant SOP before calling web_fetch"
• Linked to regression numbers for weight
• Result: Agent skips the check step entirely — goes straight from URL → web_fetch

3. Gotcha in regressions.md (searchable, referenced in boot files)

• Added to Gotchas section: "x.com / twitter.com URLs: NEVER use web_fetch"
• With explicit routing: fxtwitter → x_search → SearXNG
• Result: Agent does not search regressions before tool use

4. Regression logging (5 regressions: #144, #145, #146, #147 + earlier incidents)

• Each regression documents the failure pattern and fix
• Pattern tags: tool-exists-didnt-use, capability-gap, wrong-fallback
• Result: Regressions document the problem but do not prevent recurrence

5. URL routing table at top of TOOLS.md (latest attempt)

• Moved the routing table to the very top of TOOLS.md (system prompt injected)
• Table format: domain → "web_fetch works?" → "use instead"
• x.com row: ❌ NEVER → fxtwitter API
• Result: TBD — just implemented, but same pattern as previous attempts (adding text the agent needs to voluntarily read before acting)

Root cause analysis

The agent identified the root cause itself:

"The system prompt lists web_fetch as: Fetch and extract readable content from a URL. That is the most direct pattern match in my tool list when I see a URL. URL appears → I scan available tools → web_fetch says give me a URL, get content → I call it. No intermediate thinking."

The tool description is the strongest signal. All other mitigations require a deliberate pause-and-check step that the agent consistently skips under normal operation.

Proposed solution: Domain-aware tool routing at the platform level

The fix should happen at the OpenClaw/tool layer, not the agent instruction layer:

Option A: Domain blocklist on web_fetch

Add a config option like tools.web.fetch.domainBlocklist that rejects calls to known-bad domains and returns an error message pointing to the correct tool:

"web_fetch blocked for x.com — use fxtwitter API: https://api.fxtwitter.com/<user>/status/<id>"

Option B: Domain-to-tool routing config

A tools.web.fetch.domainRoutes config that automatically rewrites URLs before fetching:

{
  "tools": {
    "web": {
      "fetch": {
        "domainRoutes": {
          "x.com": "https://api.fxtwitter.com/{path}",
          "twitter.com": "https://api.fxtwitter.com/{path}"
        }
      }
    }
  }
}

Option C: Tool description override

Allow tools.web.fetch.description to be user-configurable so operators can append domain-specific warnings that appear at the same salience level as the default description.

Any of these would solve the problem at the layer where the decision actually happens, rather than relying on the agent to check external documentation before every tool call.

Environment

• OpenClaw 2026.3.23
• Models tested: Claude Opus, Claude Sonnet, GPT-5.3-Codex (all exhibit the same behavior)
• 5+ documented regressions across multiple sessions and models