openclaw - 💡(How to fix) Fix Fix SecretRef integration so external secret providers batch/memoize lookups across all vault backends [1 comments, 2 participants]

Summary

OpenClaw's SecretRef integration needs a product-level fix for external secret providers as a class, not a 1Password-specific workaround and not a third-party plugin punt.

Follow-up to closed #70952. That issue used 1Password Connect as the backend where the duplicate traffic was visible, but the underlying problem applies to any external vault/secret entity backend that may need container lookup, item lookup, version lookup, or payload fetches before returning a field value.

Examples include, but are not limited to:

• 1Password / 1Password Connect
• HashiCorp Vault
• AWS Secrets Manager
• GCP Secret Manager
• Azure Key Vault
• pass / gopass-style stores
• any custom exec resolver backed by an external secret service

Core problem

OpenClaw currently has batching, but it is too low-level to solve the external-provider problem as exposed to users.

Core batching only dedupes exact SecretRef keys and groups refs by source/provider. That works if users already have one shared provider/resolver where each SecretRef id is a meaningful backend secret id.

But OpenClaw's current documented and configured usage can model each external secret as its own exec provider command. For example, the documented 1Password setup uses one provider command such as op read op://... and then a SecretRef id like value. With that shape, OpenClaw cannot batch across multiple secrets because it has already lost the shared backend semantics. Every logical secret becomes a separate provider command, so duplicate backend lookups are inevitable.

This is not actually specific to op or 1Password. The same design problem exists for any external vault integration where OpenClaw cannot see that multiple configured SecretRefs share the same backend account/provider/container/item path. If OpenClaw represents those as separate provider commands, it cannot dedupe or memoize the expensive backend work.

Why this belongs in OpenClaw

This should be fixed in OpenClaw's SecretRef integration because OpenClaw owns:

• the SecretRef contract exposed to users
• the provider config shape under secrets.providers
• openclaw secrets configure / apply / audit behavior
• the official docs and examples that tell users how to wire external secret stores
• gateway startup resolution behavior and active-surface collection

Telling users to build or find a separate resolver/plugin is not an adequate answer when the supported integration shape can itself force one external call path per secret. OpenClaw needs a first-class supported pattern that preserves backend identity and enables batching/memoization before secrets reach the runtime snapshot.

Observed impact

#70952 reported gateway startup producing repeated external secret-provider traffic in the first 60 seconds:

• 84 backend calls
• 28 unique provider paths
• 56 duplicate calls
• repeated container/vault lookups, item/secret name lookups, and item fetches
• top duplicate: 26 calls to one container/vault-title lookup path

That report happened with 1Password Connect, but the failure mode is generic: OpenClaw can multiply startup traffic against any quota-limited external secret backend when repeated SecretRefs require the same intermediate provider lookups.

This also combines badly with the crash-loop/rate-limit behavior already tracked in #56217.

Current core behavior

Core does useful but insufficient batching:

• src/secrets/runtime.ts collects active config/auth SecretRefs and calls resolveSecretRefValues for the runtime snapshot.
• src/secrets/resolve.ts dedupes exact SecretRef keys and groups by source/provider.
• For exec providers, src/secrets/resolve.ts sends one JSON payload with ids for that provider.

The gap is that OpenClaw does not provide a general external-secret-provider integration model that keeps all refs for one backend/account/provider in one batchable provider group with ids that represent backend secret locations. Without that, batching is technically present but practically bypassed by the way external secrets are configured.

Expected behavior

OpenClaw should provide a first-class SecretRef integration path for external secret backends that is efficient by default.

The important properties are:

• one configured provider should represent one external backend/account/context, not one individual secret value
• SecretRef ids should carry the backend secret identity/path needed for batching
• one startup resolution pass should batch refs per external backend provider
• repeated backend container/item/entity/payload lookups during that pass should be memoized in memory
• duplicate logical SecretRefs should not repeatedly call the external service
• this should apply generically across external vault providers, not only 1Password

Possible implementation directions

Any of these could work; the key is fixing the SecretRef integration model rather than pushing users to invent their own resolver:

1. Add a first-party external secret provider abstraction under secrets.providers where backend adapters can batch and use a pass-scoped cache.
2. Extend the exec provider flow and openclaw secrets configure so the supported external-vault pattern is one shared batched resolver per backend/account, with backend paths in SecretRef ids.
3. Provide bundled resolver adapters for common backends, while keeping the core contract generic enough for third-party backends.
4. Update docs so examples no longer recommend one-provider-command-per-secret for external vaults.

Acceptance criteria

• OpenClaw has a supported SecretRef pattern for external vault/entity-backed secret stores that batches multiple refs through one backend provider context.
• Repeated references to the same backend container/item/entity in one startup resolution pass reuse in-memory results.
• The solution is generic and applies to external secret providers as a class, with 1Password only serving as one reproduced backend.
• Docs and configure/apply flows guide users toward the batchable pattern by default.
• Regression coverage proves the supported path avoids duplicate backend lookups for repeated shared external entities.
• The fix keeps secrets in memory only and does not add disk caching or plaintext secret logging.

Related

• #70952: original duplicate SecretRef provider lookup report, closed with too-narrow external/plugin framing
• #56217: crash-loop exhaustion of external secret-provider rate limits, observed with 1Password service-account limits
• #55459: op CLI daemon/cache behavior on macOS, one backend-specific symptom
• #17311: broader env/keyring/1Password SecretsProvider plan