PR #67717: fix(agents): detect Cloudflare challenge pages and provide actionable error message

• Repository: openclaw/openclaw
• Author: SARAMALI15792
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/67717

Description (problem / solution / changelog)

Summary

• Problem: Cloudflare returns 403 with JS Challenge page when openai-codex provider accessed via proxy from mainland China. Current error message says "Authentication failed" which is misleading — the issue is TLS fingerprint detection, not auth.
• Why it matters: Users waste time re-authenticating when the real fix is a reverse proxy with browser-like TLS fingerprint.
• What changed: Added cloudflare_challenge failure kind to detect Cloudflare challenge pages (403 with cf-browser-verification, challenges.cloudflare.com, etc.) and return actionable error message explaining TLS fingerprinting + reverse proxy workaround.
• What did NOT change: Generic 403 HTML errors still classified as auth_html_403. No changes to transport layer, provider config, or TLS implementation.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #67670
• Related #67642 (Cloudflare HTML error classification for 5xx)
• This PR fixes a bug or regression

Root Cause (if applicable)

• Root cause: classifyProviderRuntimeFailureKind classifies all 403 HTML responses as auth_html_403 without checking for Cloudflare-specific challenge patterns. Cloudflare challenge pages contain distinctive markers (cf-browser-verification, challenges.cloudflare.com) that distinguish them from genuine auth failures.
• Missing detection / guardrail: No pattern matching for Cloudflare challenge page signatures before falling through to generic 403 HTML classification.
• Contributing context: Issue #67642 fixed Cloudflare 5xx HTML classification but didn't address 403-specific challenge pages.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/agents/pi-embedded-helpers/provider-error-patterns.test.ts
• Scenario the test should lock in: 403 HTML with Cloudflare challenge patterns (cf-browser-verification, challenges.cloudflare.com, _cf_chl_, just a moment, ray id:) returns cloudflare_challenge failure kind, not auth_html_403.
• Why this is the smallest reliable guardrail: Unit test directly verifies classification logic with minimal HTML fixture containing only Cloudflare challenge markers.
• Existing test that already covers this: None (new functionality).
• If no new test is added, why not: N/A (3 new tests added).

User-visible / Behavior Changes

Before: Users seeing Cloudflare 403 challenges get misleading error:

"Authentication failed with an HTML 403 response from the provider. Re-authenticate and verify your provider account access."

After: Users get actionable error explaining root cause and workaround:

"Cloudflare blocked the request with a browser challenge (HTTP 403). Node.js TLS fingerprint was detected as non-browser traffic. Set up a local reverse proxy using a browser-compatible HTTP client (e.g. Python cloudscraper) and point the provider's baseUrl to it."

Diagram (if applicable)

Before:
403 + Cloudflare challenge HTML → classifyProviderRuntimeFailureKind
  → isHtmlErrorResponse(403) → "auth_html_403"
  → "Authentication failed" (misleading)

After:
403 + Cloudflare challenge HTML → classifyProviderRuntimeFailureKind
  → isCloudflareChallengePage(403) → "cloudflare_challenge"
  → "Cloudflare blocked... TLS fingerprint... reverse proxy" (actionable)

403 + generic HTML → classifyProviderRuntimeFailureKind
  → isHtmlErrorResponse(403) → "auth_html_403" (unchanged)

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No

Repro + Verification

Environment

• OS: macOS 26.3 (arm64), Windows 11 Pro
• Runtime/container: Node.js v22.22.2
• Model/provider: openai-codex (chatgpt.com/backend-api)
• Integration/channel: N/A
• Relevant config: Proxy routing traffic through VPN/Quantumult X from mainland China

Steps

1. Configure openai-codex provider with valid OAuth token
2. Route traffic through proxy from mainland China (or any region where Cloudflare triggers JS challenges for Node.js TLS fingerprints)
3. Attempt LLM request via openai-codex provider
4. Observe 403 response with Cloudflare challenge HTML

Expected

Error message explains TLS fingerprint detection and suggests reverse proxy workaround.

Actual

Before fix: "Authentication failed with an HTML 403 response from the provider." After fix: "Cloudflare blocked the request with a browser challenge (HTTP 403). Node.js TLS fingerprint was detected as non-browser traffic. Set up a local reverse proxy using a browser-compatible HTTP client (e.g. Python cloudscraper) and point the provider's baseUrl to it."

Evidence

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Test results:

• 3 new tests added, all passing (39/39 total in test suite)
• Test: "classifies 403 Cloudflare challenge as cloudflare_challenge" ✅
• Test: "classifies non-Cloudflare 403 HTML as auth_html_403 (regression guard)" ✅
• Test: "formats Cloudflare challenge error with TLS fingerprint guidance" ✅

Human Verification (required)

Verified scenarios:

• Cloudflare challenge HTML (with cf-browser-verification, challenges.cloudflare.com, _cf_chl_, just a moment, ray id:) → cloudflare_challenge classification
• Generic 403 HTML (no Cloudflare patterns) → auth_html_403 classification (regression guard)
• Error message contains "Cloudflare blocked", "TLS fingerprint", "reverse proxy", "cloudscraper"

Edge cases checked:

• 403 HTML without Cloudflare patterns still returns auth_html_403
• Non-403 status codes with Cloudflare patterns do not trigger cloudflare_challenge (status check first)
• HTML responses without cf-browser-verification or other markers fall through to generic classification

What I did NOT verify:

• Actual Cloudflare challenge page in production (requires mainland China proxy setup)
• Integration with live openai-codex provider endpoint

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No

Risks and Mitigations

• Risk: False positives — generic HTML pages containing "cloudflare" text might get misclassified as challenges.

  • Mitigation: Regex requires multiple CF-specific patterns (cf-browser-verification, challenges.cloudflare.com, etc.), not just keyword "cloudflare". Pattern is case-insensitive and matches partial strings safely.

• Risk: Cloudflare changes challenge page structure, patterns no longer match.

  • Mitigation: Regex covers multiple stable markers (div ID, script domain, form field prefix, heading text, Ray ID footer). Unlikely all change simultaneously. If needed, patterns can be updated in future PR.

Changed files

• .gitignore (modified, +1/-0)
• src/agents/pi-embedded-helpers/errors.ts (modified, +25/-0)
• src/agents/pi-embedded-helpers/provider-error-patterns.test.ts (modified, +58/-0)