openclaw - 💡(How to fix) Fix [Bug]: request.headers SecretRefs on model providers fail in embedded agent context with "unresolved SecretRef" error [2 comments, 2 participants]

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

In OpenClaw 2026.5.3, embedded agent turns fail with an "unresolved SecretRef" error for every provider in the model fallback chain — including providers that have no SecretRef configured — when any single provider has a request.headers SecretRef.

Steps to reproduce

1. Configure one model provider (e.g. ollama) with a request.headers SecretRef in openclaw.json under models.providers.ollama.request.headers.
2. Configure agents.defaults.model.fallbacks with a chain that includes that provider plus at least one provider that has no SecretRef (e.g. openai-codex/gpt-5.5, github-copilot/gpt-5.5).
3. Start the OpenClaw gateway and confirm openclaw secrets audit --check --json reports zero unresolved refs and openclaw infer model run --model ollama/... --gateway succeeds.
4. Send an inbound message from any embedded channel (Discord, WhatsApp, chat) to trigger an embedded agent turn.
5. Observe that the turn fails before any model output with an "unresolved SecretRef" error logged against every provider in the fallback chain, including providers with no SecretRef.

Expected behavior

SecretRefs in models.providers.<name>.request.headers should be resolved lazily, per-provider, only when that provider is actually selected for a request. A SecretRef on provider A should never cause provider B (which has no SecretRef) to fail. The embedded agent pipeline should also attach a runtime snapshot before any config resolution occurs, consistent with the gateway transport path (which works correctly).

Actual behavior

All embedded agent turns fail with an error of the form:

Error: request.headers.X-Pomerium-Authorization: unresolved SecretRef "file:filemain:/providers/ollama/token". Resolve this command against an active gateway runtime snapshot before reading it.

This error fires for every provider in the fallback chain — including openai-codex and github-copilot, which have no request.headers SecretRef at all. The agent fails before producing any reply.

openclaw infer model run --model ollama/gemma4:e2b --gateway works correctly; SecretRefs resolve fine via the gateway WebSocket transport.

OpenClaw version

2026.5.3 (06d46f7)

Operating system

Ubuntu 24.04.4 LTS (GNU/Linux 6.8.0-111-generic x86_64)

Install method

Docker (custom docker-compose deployment)

Model

openai-codex/gpt-5.5 (primary); github-copilot/gpt-5.5, ollama/gemma4:e2b (fallbacks)

Provider / routing chain

embedded agent -> model fallback chain -> openai-codex / github-copilot / ollama

Additional provider/model setup details

Relevant config excerpt:

{
  "models": {
    "providers": {
      "ollama": {
        "baseUrl": "http://localhost:8080",
        "api": "ollama",
        "request": {
          "headers": {
            "X-Pomerium-Authorization": {
              "source": "file",
              "provider": "filemain",
              "id": "/providers/ollama/token"
            }
          }
        }
      }
    }
  },
  "agents": {
    "defaults": {
      "model": {
        "primary": "openai-codex/gpt-5.5",
        "fallbacks": ["github-copilot/gpt-5.5", "ollama/gemma4:e2b"]
      }
    }
  }
}

openclaw secrets audit --check --json reports zero unresolved refs. openclaw secrets reload --json succeeds with no warnings. Only embedded agent turns are affected; the gateway transport path resolves SecretRefs correctly.

Logs, screenshots, and evidence

[diagnostic] lane task error: lane=main durationMs=... error="Error: request.headers.X-Pomerium-Authorization: unresolved SecretRef \"file:filemain:/providers/ollama/token\". Resolve this command against an active gateway runtime snapshot before reading it."
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai-codex/gpt-5.5 candidate=openai-codex/gpt-5.5 reason=unknown next=github-copilot/gpt-5.5 detail=request.headers.X-Pomerium-Authorization: unresolved SecretRef "file:filemain:/providers/ollama/token". Resolve this command against an active gateway runtime snapshot before reading it.
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=github-copilot/gpt-5.5 candidate=github-copilot/gpt-5.5 reason=unknown next=ollama/gemma4:e2b detail=request.headers.X-Pomerium-Authorization: unresolved SecretRef "file:filemain:/providers/ollama/token". Resolve this command against an active gateway runtime snapshot before reading it.

The error subsystem logged is diagnostic, not the model provider itself, suggesting the SecretRef is being resolved eagerly at request-init or diagnostic time rather than at the point the ollama provider is actually invoked.

Impact and severity

• Affected: Any operator using models.providers.<name>.request.headers with a SecretRef and that provider anywhere in agents.defaults.model.fallbacks (or per-agent fallback chains).
• Severity: High — blocks all embedded agent turns from chat channels entirely, even when the SecretRef provider is last in the fallback chain and would never have been called.
• Frequency: Reproducible on every embedded agent turn with the above config.
• Consequence: Agents are completely non-functional from embedded channels (Discord, WhatsApp, etc.) despite the gateway and channel transports being healthy.

Additional information

Workarounds (both have significant downsides):

1. Remove any provider with request.headers SecretRefs from the fallback chain entirely. This restores embedded agent function but makes those providers permanently unavailable from chat channels.
2. Replace the SecretRef with a hardcoded plaintext header value. This unblocks embedded agent turns but stores the credential in plain text in openclaw.json, which is not recommended for production deployments.

Related issues with the same root-cause class (eager SecretRef resolution outside a live snapshot in the embedded agent pipeline):

• #75433 — BUG: Embedded channel reply runs crash on SecretRef-backed Telegram/Discord credentials (channel credentials path)
• #47519 — message tool crashes on unrelated channel SecretRef failures during schema probing (cross-contamination pattern)

The gateway transport path works because it runs inside the proper WebSocket request pipeline where a runtime snapshot is always available. The embedded agent pipeline appears to eagerly resolve SecretRefs across all model providers at request-init or diagnostic time, before a snapshot is attached to the execution context.