openclaw - ✅(Solved) Fix [Bug]: firecrawl_search fails with local, self-hosted Firecrawl service using http [2 pull requests, 1 comments, 2 participants]

jhthompson12 · 2026-04-09T18:16:48Z

[openclaw] 2026.4.8 firecrawl search fails with Firecrawl baseUrl must use https error with "baseUrl": "http://host.openshell.internal:3002/v1" 2026.4.8 `firecrawl_search` fails with `Firecrawl baseUrl must use https` error with `"baseUrl": "http://host.openshell.internal:3002/v1"` # PR #63899: fix(firecrawl): allow http and custom hosts for self-hosted Firecrawl instances - Repository: openclaw/openclaw - Author: extrasmall0 - State: closed | merged: False - Link: https://github.com/openclaw/openclaw/pull/63899 ## Description (problem / solution / changelog) ## Summary Fixes #63877 The `resolveEndpoint` function in `extensions/firecrawl/src/firecrawl-client.ts` unconditionally rejects any `baseUrl` that: 1. Does not use `https:` 2. Has a hostname other than `api.firecrawl.dev` This prevents users who run a self-hosted Firecrawl service on a private/local network from using the firecrawl integration at all. ## Root Cause The https-only and host allowlist checks were written to protect against pointing the integration at untrusted external endpoints. These are reasonable constraints for the cloud API endpoint, but they break legitimate self-hosted deployments. ## Fix Relax both constraints so they only apply when the resolved hostname is the default cloud endpoint (`api.firecrawl.dev`). When a custom `baseUrl` is configured, the operator is explicitly overriding the default and is assumed to know what they're doing: ```ts const isDefaultHost = url.hostname === DEFAULT_FIRECRAWL_BASE_URL_HOSTNAME; // https enforcement and host allowlist only apply to the cloud endpoint if (isDefaultHost && url.protocol !== 'https:') { throw ... } if (isDefaultHost && !ALLOWED_FIRECRAWL_HOSTS.has(url.hostname)) { throw ... } ``` ## Before / After **Before:** ``` baseUrl: "http://host.internal:3002/v1" → Error: Firecrawl baseUrl must use https. baseUrl: "https://host.internal:444/v1" → Error: Firecrawl baseUrl host is not allowed: host.internal ``` **After:** ``` baseUrl: "http://host.internal:3002/v1" → ✅ works baseUrl: "https://host.internal:444/v1" → ✅ works baseUrl: "https://api.firecrawl.dev" → ✅ unchanged behavior ``` ## Changed files - `.agents/skills/openclaw-ghsa-maintainer/SKILL.md` (added, +87/-0) - `.agents/skills/openclaw-parallels-smoke/SKILL.md` (added, +129/-0) - `.agents/skills/openclaw-pr-maintainer/SKILL.md` (added, +75/-0) - `.agents/skills/openclaw-qa-testing/SKILL.md` (added, +148/-0) - `.agents/skills/openclaw-qa-testing/agents/openai.yaml` (added, +4/-0) - `.agents/skills/openclaw-release-maintainer/SKILL.md` (added, +267/-0) - `.agents/skills/openclaw-test-heap-leaks/SKILL.md` (added, +75/-0) - `.agents/skills/openclaw-test-heap-leaks/agents/openai.yaml` (added, +4/-0) - `.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs` (added, +553/-0) - `.agents/skills/parallels-discord-roundtrip/SKILL.md` (added, +62/-0) - `.agents/skills/security-triage/SKILL.md` (added, +111/-0) - `.codex` (added, +0/-0) - `.detect-secrets.cfg` (modified, +19/-4) - `.dockerignore` (modified, +8/-0) - `.github/CODEOWNERS` (added, +54/-0) - `.github/ISSUE_TEMPLATE/bug_report.yml` (modified, +61/-19) - `.github/actions/ensure-base-commit/action.yml` (modified, +16/-2) - `.github/actions/setup-node-env/action.yml` (modified, +12/-22) - `.github/actions/setup-pnpm-store-cache/action.yml` (modified, +12/-10) - `.github/codeql/codeql-javascript-typescript.yml` (added, +18/-0) - `.github/labeler.yml` (modified, +121/-16) - `.github/pr-assets/compaction-checkpoints/sessions-checkpoints-inline.png` (added, +0/-0) - `.github/pr-assets/compaction-checkpoints/sessions-overview-inline.png` (added, +0/-0) - `.github/pull_request_template.md` (modified, +46/-7) - `.github/workflows/auto-response.yml` (modified, +61/-9) - `.github/workflows/ci.yml` (modified, +921/-438) - `.github/workflows/codeql.yml` (added, +137/-0) - `.github/workflows/control-ui-locale-refresh.yml` (added, +172/-0) - `.github/workflows/docker-release.yml` (modified, +196/-48) - `.github/workflows/docs-sync-publish.yml` (added, +70/-0) - `.github/workflows/docs-translate-trigger-release.yml` (added, +42/-0) - `.github/workflows/install-smoke.yml` (modified, +121/-32) - `.github/workflows/labeler.yml` (modified, +205/-16) - `.github/workflows/macos-release.yml` (added, +93/-0) - `.github/workflows/openclaw-npm-release.yml` (added, +503/-0) - `.github/workflows/plugin-clawhub-release.yml` (added, +276/-0) - `.github/workflows/plugin-npm-release.yml` (added, +217/-0) - `.github/workflows/sandbox-common-smoke.yml` (modified, +8/-3) - `.github/workflows/stale.yml` (modified, +12/-9) - `.github/workflows/workflow-sanity.yml` (modified, +36/-5) - `.gitignore` (modified, +31/-2) - `.jscpd.json` (added, +16/-0) - `.markdownlint-cli2.jsonc` (modified, +3/-0) - `.npmignore` (added, +3/-0) - `.npmrc` (modified, +3/-0) - `.oxlintrc.json` (modified, +32/-2) - `.pi/prompts/reviewpr.md` (modified, +37/-8) - `.pre-commit-config.yaml` (modified, +27/-1) - `.prettierignore`

openclaw2026-04-09 18:16:48

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

openclaw/openclaw#63877•Fetched 2026-04-10 03:41:42

View on GitHub

Comments

Participants

Timeline

Reactions

Author

jhthompson12

Participants

Artyomkun

jhthompson12

Timeline (top)

cross-referenced ×2labeled ×2commented ×1referenced ×1

2026.4.8 firecrawl_search fails with Firecrawl baseUrl must use https error with "baseUrl": "http://host.openshell.internal:3002/v1"

Error Message

2026.4.8 firecrawl_search fails with Firecrawl baseUrl must use https error with "baseUrl": "http://host.openshell.internal:3002/v1" 3. ask openclaw to do a basic websearch, fails with baseUrl must use https error 4. try again with firecrawl service behind https reverse proxy (https://host.openshell.internal:444/v1), fails this time with Firecrawl baseUrl host is not allowed: host.openshell.internal error

ask openclaw to do a basic websearch, fails with baseUrl must use https error
try again with firecrawl service behind https reverse proxy (https://host.openshell.internal:444/v1), fails this time with Firecrawl baseUrl host is not allowed: host.openshell.internal error

Root Cause

2026.4.8 firecrawl_search fails with Firecrawl baseUrl must use https error with "baseUrl": "http://host.openshell.internal:3002/v1"

Fix Action

Fixed

Fixed by PR: fix(firecrawl): allow http and custom hosts for self-hosted Firecrawl instances (https://github.com/openclaw/openclaw/pull/63899)
Fixed by PR: Fix: allow self-hosted Firecrawl baseUrl with http on private networks (Resolves #63877) (https://github.com/openclaw/openclaw/pull/63941)

PR fix notes

PR #63899: fix(firecrawl): allow http and custom hosts for self-hosted Firecrawl instances

Repository: openclaw/openclaw
Author: extrasmall0
State: closed | merged: False
Link: https://github.com/openclaw/openclaw/pull/63899

Description (problem / solution / changelog)

Summary

Fixes #63877

The resolveEndpoint function in extensions/firecrawl/src/firecrawl-client.ts unconditionally rejects any baseUrl that:

Does not use https:
Has a hostname other than api.firecrawl.dev

This prevents users who run a self-hosted Firecrawl service on a private/local network from using the firecrawl integration at all.

Root Cause

The https-only and host allowlist checks were written to protect against pointing the integration at untrusted external endpoints. These are reasonable constraints for the cloud API endpoint, but they break legitimate self-hosted deployments.

Fix

Relax both constraints so they only apply when the resolved hostname is the default cloud endpoint (api.firecrawl.dev). When a custom baseUrl is configured, the operator is explicitly overriding the default and is assumed to know what they're doing:

const isDefaultHost = url.hostname === DEFAULT_FIRECRAWL_BASE_URL_HOSTNAME;
// https enforcement and host allowlist only apply to the cloud endpoint
if (isDefaultHost && url.protocol !== 'https:') { throw ... }
if (isDefaultHost && !ALLOWED_FIRECRAWL_HOSTS.has(url.hostname)) { throw ... }

Before / After

Before:

baseUrl: "http://host.internal:3002/v1"  →  Error: Firecrawl baseUrl must use https.
baseUrl: "https://host.internal:444/v1"  →  Error: Firecrawl baseUrl host is not allowed: host.internal

After:

baseUrl: "http://host.internal:3002/v1"  →  ✅ works
baseUrl: "https://host.internal:444/v1"  →  ✅ works
baseUrl: "https://api.firecrawl.dev"      →  ✅ unchanged behavior

Changed files

.agents/skills/openclaw-ghsa-maintainer/SKILL.md (added, +87/-0)
.agents/skills/openclaw-parallels-smoke/SKILL.md (added, +129/-0)
.agents/skills/openclaw-pr-maintainer/SKILL.md (added, +75/-0)
.agents/skills/openclaw-qa-testing/SKILL.md (added, +148/-0)
.agents/skills/openclaw-qa-testing/agents/openai.yaml (added, +4/-0)
.agents/skills/openclaw-release-maintainer/SKILL.md (added, +267/-0)
.agents/skills/openclaw-test-heap-leaks/SKILL.md (added, +75/-0)
.agents/skills/openclaw-test-heap-leaks/agents/openai.yaml (added, +4/-0)
.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs (added, +553/-0)
.agents/skills/parallels-discord-roundtrip/SKILL.md (added, +62/-0)
.agents/skills/security-triage/SKILL.md (added, +111/-0)
.codex (added, +0/-0)
.detect-secrets.cfg (modified, +19/-4)
.dockerignore (modified, +8/-0)
.github/CODEOWNERS (added, +54/-0)
.github/ISSUE_TEMPLATE/bug_report.yml (modified, +61/-19)
.github/actions/ensure-base-commit/action.yml (modified, +16/-2)
.github/actions/setup-node-env/action.yml (modified, +12/-22)
.github/actions/setup-pnpm-store-cache/action.yml (modified, +12/-10)
.github/codeql/codeql-javascript-typescript.yml (added, +18/-0)
.github/labeler.yml (modified, +121/-16)
.github/pr-assets/compaction-checkpoints/sessions-checkpoints-inline.png (added, +0/-0)
.github/pr-assets/compaction-checkpoints/sessions-overview-inline.png (added, +0/-0)
.github/pull_request_template.md (modified, +46/-7)
.github/workflows/auto-response.yml (modified, +61/-9)
.github/workflows/ci.yml (modified, +921/-438)
.github/workflows/codeql.yml (added, +137/-0)
.github/workflows/control-ui-locale-refresh.yml (added, +172/-0)
.github/workflows/docker-release.yml (modified, +196/-48)
.github/workflows/docs-sync-publish.yml (added, +70/-0)
.github/workflows/docs-translate-trigger-release.yml (added, +42/-0)
.github/workflows/install-smoke.yml (modified, +121/-32)
.github/workflows/labeler.yml (modified, +205/-16)
.github/workflows/macos-release.yml (added, +93/-0)
.github/workflows/openclaw-npm-release.yml (added, +503/-0)
.github/workflows/plugin-clawhub-release.yml (added, +276/-0)
.github/workflows/plugin-npm-release.yml (added, +217/-0)
.github/workflows/sandbox-common-smoke.yml (modified, +8/-3)
.github/workflows/stale.yml (modified, +12/-9)
.github/workflows/workflow-sanity.yml (modified, +36/-5)
.gitignore (modified, +31/-2)
.jscpd.json (added, +16/-0)
.markdownlint-cli2.jsonc (modified, +3/-0)
.npmignore (added, +3/-0)
.npmrc (modified, +3/-0)
.oxlintrc.json (modified, +32/-2)
.pi/prompts/reviewpr.md (modified, +37/-8)
.pre-commit-config.yaml (modified, +27/-1)
.prettierignore (added, +1/-0)
.secrets.baseline (modified, +229/-316)
.swiftformat (modified, +1/-1)
.swiftlint.yml (modified, +3/-1)
AGENTS.md (modified, +181/-120)
CHANGELOG.md (modified, +2563/-490)
CONTRIBUTING.md (modified, +63/-8)
Dockerfile (modified, +181/-60)
Dockerfile.sandbox (modified, +7/-3)
Dockerfile.sandbox-browser (modified, +9/-5)
Dockerfile.sandbox-common (modified, +7/-4)
Makefile (added, +4/-0)
README.md (modified, +79/-24)
SECURITY.md (modified, +41/-1)
Swabble/Sources/SwabbleKit/WakeWordGate.swift (modified, +7/-13)
Swabble/Tests/SwabbleKitTests/WakeWordGateTests.swift (modified, +19/-0)
appcast.xml (modified, +161/-331)
apps/android/README.md (modified, +70/-4)
apps/android/app/build.gradle.kts (modified, +131/-9)
apps/android/app/proguard-rules.pro (modified, +0/-20)
apps/android/app/src/main/AndroidManifest.xml (modified, +17/-9)
apps/android/app/src/main/java/ai/openclaw/app/AssistantLaunch.kt (added, +43/-0)
apps/android/app/src/main/java/ai/openclaw/app/CameraHudState.kt (added, +14/-0)
apps/android/app/src/main/java/ai/openclaw/app/DeviceNames.kt (added, +26/-0)
apps/android/app/src/main/java/ai/openclaw/app/LocationMode.kt (added, +15/-0)
apps/android/app/src/main/java/ai/openclaw/app/MainActivity.kt (added, +85/-0)
apps/android/app/src/main/java/ai/openclaw/app/MainViewModel.kt (added, +385/-0)
apps/android/app/src/main/java/ai/openclaw/app/NodeApp.kt (added, +37/-0)
apps/android/app/src/main/java/ai/openclaw/app/NodeForegroundService.kt (added, +162/-0)
apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt (added, +1395/-0)
apps/android/app/src/main/java/ai/openclaw/app/NotificationForwardingPolicy.kt (added, +102/-0)
apps/android/app/src/main/java/ai/openclaw/app/PermissionRequester.kt (added, +200/-0)
apps/android/app/src/main/java/ai/openclaw/app/SecurePrefs.kt (added, +553/-0)
apps/android/app/src/main/java/ai/openclaw/app/SessionKey.kt (added, +24/-0)
apps/android/app/src/main/java/ai/openclaw/app/VoiceWakeMode.kt (added, +14/-0)
apps/android/app/src/main/java/ai/openclaw/app/WakeWords.kt (added, +21/-0)
apps/android/app/src/main/java/ai/openclaw/app/chat/ChatController.kt (added, +641/-0)
apps/android/app/src/main/java/ai/openclaw/app/chat/ChatModels.kt (added, +44/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/BonjourEscapes.kt (added, +35/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/DeviceAuthPayload.kt (added, +52/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/DeviceAuthStore.kt (added, +92/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/DeviceIdentityStore.kt (added, +174/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayDiscovery.kt (added, +521/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayEndpoint.kt (added, +26/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayHostSecurity.kt (added, +124/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayProtocol.kt (added, +3/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewaySession.kt (added, +1039/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayTls.kt (added, +185/-0)
apps/android/app/src/main/java/ai/openclaw/app/gateway/InvokeErrorParser.kt (added, +39/-0)
apps/android/app/src/main/java/ai/openclaw/app/node/A2UIHandler.kt (added, +153/-0)
apps/android/app/src/main/java/ai/openclaw/app/node/CalendarHandler.kt (added, +384/-0)
apps/android/app/src/main/java/ai/openclaw/app/node/CallLogHandler.kt (added, +247/-0)

PR #63941: Fix: allow self-hosted Firecrawl baseUrl with http on private networks (Resolves #63877)

Repository: openclaw/openclaw
Author: Mlightsnow
State: open | merged: False
Link: https://github.com/openclaw/openclaw/pull/63941

Description (problem / solution / changelog)

Fixes #63877.

Problem

resolveEndpoint() in the Firecrawl extension hardcodes ALLOWED_FIRECRAWL_HOSTS to only api.firecrawl.dev and requires HTTPS unconditionally. This blocks users who self-host Firecrawl on their local/private network from using it, even when they explicitly configure a baseUrl.

Changes

Removed the ALLOWED_FIRECRAWL_HOSTS allowlist — the baseUrl is admin-configured, not untrusted user input, so any host should be allowed
Added isPrivateOrLocalHost() helper that detects private/local network addresses (loopback, RFC 1918 IPs, .local/.internal/.localhost suffixes)
HTTP is now allowed for private/local network hosts; HTTPS is still required for public hosts to prevent credential leakage
URL sanitization (credential/query/hash stripping) is preserved

Test plan

Updated existing endpoint validation test to reflect new behavior
Added tests for: custom public hosts over HTTPS, HTTP on private/local hosts (localhost, 127.0.0.1, 192.168.x.x, 10.x.x.x, .internal, .local)
Verified HTTPS is still required for public hosts
Ran pnpm test extensions/firecrawl/src/firecrawl-tools.test.ts — 21 passed, 2 failed (pre-existing DNS failures on main, unrelated)

AI-assisted (Claude). Fully tested. I understand what the code does.

Changed files

extensions/firecrawl/src/firecrawl-client.ts (modified, +52/-5)
extensions/firecrawl/src/firecrawl-tools.test.ts (modified, +55/-7)

Code Example

"tools": {
      "web": {
        "search": {
          "provider": "firecrawl"
        },
        "fetch": {
          "firecrawl": {
            "apiKey": "dummy",
            "baseUrl": "http://host.openshell.internal:3002/v1",
            "onlyMainContent": true,
            "maxAgeMs": 86400000,
            "timeoutSeconds": 60
          }
        }
      }
    },
    "plugins": {
      "entries": {
        "firecrawl": {
          "enabled": true,
          "config": {
            "webSearch": {
              "apiKey": "dummy",
              "baseUrl": "http://host.openshell.internal:3002/v1"
            }
          }
        }
      }
    }

---

"baseUrl": "https://host.openshell.internal:444/v1"

---

RAW_BUFFERClick to expand / collapse

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

Summary

2026.4.8 firecrawl_search fails with Firecrawl baseUrl must use https error with "baseUrl": "http://host.openshell.internal:3002/v1"

Steps to reproduce

Firecrawl service running on same host
configure web search plugin to use local firecrawl service over http (http://host.openshell.internal:3002/v1)

  "tools": {
    "web": {
      "search": {
        "provider": "firecrawl"
      },
      "fetch": {
        "firecrawl": {
          "apiKey": "dummy",
          "baseUrl": "http://host.openshell.internal:3002/v1",
          "onlyMainContent": true,
          "maxAgeMs": 86400000,
          "timeoutSeconds": 60
        }
      }
    }
  },
  "plugins": {
    "entries": {
      "firecrawl": {
        "enabled": true,
        "config": {
          "webSearch": {
            "apiKey": "dummy",
            "baseUrl": "http://host.openshell.internal:3002/v1"
          }
        }
      }
    }
  }

ask openclaw to do a basic websearch, fails with baseUrl must use https error
try again with firecrawl service behind https reverse proxy (https://host.openshell.internal:444/v1), fails this time with Firecrawl baseUrl host is not allowed: host.openshell.internal error

"baseUrl": "https://host.openshell.internal:444/v1"

Expected behavior

openclaw successfully uses the local self-hosted Firecrawl instance, even without https

Actual behavior

ask openclaw to do a basic websearch, fails with baseUrl must use https error
try again with firecrawl service behind https reverse proxy (https://host.openshell.internal:444/v1), fails this time with Firecrawl baseUrl host is not allowed: host.openshell.internal error

OpenClaw version

2026.4.8

Operating system

NemoClaw v0.0.9 sandbox

Install method

NemoClaw v0.0.9

Model

openai/gpt-oss-120b

Provider / routing chain

not sure

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Impact and severity

No response

Additional information

could there be a config option that would allow the user to use firecrawl over http instead of https?

extent analysis

TL;DR

The most likely fix is to configure the Firecrawl service to use HTTPS or find a configuration option to allow HTTP connections.

Guidance

Check the Firecrawl documentation for a configuration option to allow HTTP connections, as the current setup requires HTTPS.
Consider setting up an HTTPS reverse proxy for the Firecrawl service to meet the HTTPS requirement.
Verify if the host.openshell.internal domain is allowed in the Firecrawl configuration, as the error message suggests it is not.
Investigate the NemoClaw v0.0.9 sandbox and OpenClaw version 2026.4.8 configurations to see if there are any specific settings that can be adjusted to allow HTTP connections.

Example

No code snippet is provided, as the issue is related to configuration and not code.

Notes

The issue seems to be related to the security configuration of the Firecrawl service, which requires HTTPS connections.
The use of a self-hosted Firecrawl instance without HTTPS may pose security risks.

Recommendation

Apply workaround: Consider setting up an HTTPS reverse proxy for the Firecrawl service or finding a configuration option to allow HTTP connections, as upgrading to a fixed version is not mentioned in the issue.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

FAQ

Expected behavior

openclaw successfully uses the local self-hosted Firecrawl instance, even without https

#api #indexing error #inference speed #output truncation #response parsing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

openclaw - ✅(Solved) Fix [Bug]: firecrawl_search fails with local, self-hosted Firecrawl service using http [2 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fixed

PR fix notes

PR #63899: fix(firecrawl): allow http and custom hosts for self-hosted Firecrawl instances

Description (problem / solution / changelog)

Summary

Root Cause

Fix

Before / After

Changed files

PR #63941: Fix: allow self-hosted Firecrawl baseUrl with http on private networks (Resolves #63877)

Description (problem / solution / changelog)

Problem

Changes

Test plan

Changed files

Code Example

Bug type

Beta release blocker

Summary

Steps to reproduce

Expected behavior

Actual behavior

OpenClaw version

Operating system

Install method

Model

Provider / routing chain

Additional provider/model setup details

Logs, screenshots, and evidence

Impact and severity

Additional information

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

FAQ

Expected behavior

Still need to ship something?

RELATED_DISCOVERY

TRENDING