openclaw - ✅(Solved) Fix [Bug]: Bedrock aws-sdk auth intermittently fails in main session after 2026.3.23 upgrade (EC2 instance role) [4 pull requests, 1 participants]

openclaw2026-03-25 06:02:01

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

openclaw/openclaw#54274•Fetched 2026-04-08 01:29:50

View on GitHub

Comments

Participants

Timeline

Reactions

Author

yubingjiaocn

Participants

yubingjiaocn

Timeline (top)

cross-referenced ×4referenced ×4closed ×1subscribed ×1

Error Message

Error:

Cron jobs also affected: memory-digest cron jobs for 3/6 agents fail with the same error (the other 3 succeed, suggesting a timing/race condition).

Root Cause

Suspected root cause

Fix Action

Fix / Workaround

Configure amazon-bedrock provider with auth: "aws-sdk" on an EC2 instance (no static API keys)
Upgrade to 2026.3.23-2
Chat normally — works most of the time
Intermittently, the main session fails with "No API key found for amazon-bedrock"
Only workaround is /new to start a fresh session

Workaround attempted

PR fix notes

PR #1: fix(bedrock): stop injecting fake apiKey marker for aws-sdk auth when no env vars exist

Repository: wirjo/openclaw
Author: wirjo
State: closed | merged: False
Link: https://github.com/wirjo/openclaw/pull/1

Description (problem / solution / changelog)

fix(bedrock): stop injecting fake `apiKey` marker for `aws-sdk` auth when no env vars exist

Problem

When the Bedrock provider uses auth: "aws-sdk" with no AWS environment variables set (EC2 instance roles, ECS task roles, Lambda, etc.), the gateway startup injects apiKey: "AWS_PROFILE" into the provider config — even though AWS_PROFILE is not set.

This happens because resolveAwsSdkApiKeyVarName() unconditionally falls back to "AWS_PROFILE":

// Before
function resolveAwsSdkApiKeyVarName(env): string {
  return resolveAwsSdkEnvVarName(env) ?? "AWS_PROFILE"; // always returns a string
}

The same pattern exists in the extension's resolveBedrockConfigApiKey().

The downstream auth resolver then sees apiKey: "AWS_PROFILE", recognizes it as an AWS SDK marker via isNonSecretApiKeyMarker(), but isKnownEnvApiKeyMarker() excludes AWS SDK markers — so resolveUsableCustomProviderApiKey() returns null. The auth chain eventually fails with:

No API key found for amazon-bedrock.

This is the root cause behind three separate bug reports:

#49891 — Gateway injects apiKey: "AWS_PROFILE" on every startup, breaking auth
#50699 — Bedrock provider does not recognize environment AWS credentials
#54274 — Intermittent main session failures after upgrade (instance role)

Root Cause

Two functions (resolveAwsSdkApiKeyVarName and resolveBedrockConfigApiKey) fall back to "AWS_PROFILE" when resolveAwsSdkEnvVarName() returns undefined (no AWS env vars detected). The write path then persists this string as an apiKey marker, but the read path doesn't resolve it to a usable credential — creating a permanent auth failure state.

Fix

resolveAwsSdkApiKeyVarName() (src/agents/models-config.providers.secrets.ts):

Return undefined instead of "AWS_PROFILE" when no AWS env vars exist

resolveBedrockConfigApiKey() (extensions/amazon-bedrock/discovery.ts):

Same fix — return undefined instead of falling back to "AWS_PROFILE"

resolveMissingProviderApiKey() (src/agents/models-config.providers.secrets.ts):

Guard both the providerApiKeyResolver branch and the direct aws-sdk branch
If the resolver returns nothing, return the provider config unchanged (no apiKey injected)
The AWS SDK credential chain then resolves credentials at request time via IMDS/ECS/etc.

When AWS env vars are present (AWS_ACCESS_KEY_ID, AWS_PROFILE, AWS_BEARER_TOKEN_BEDROCK), the marker is still injected correctly — no behavior change for those setups.

Verification

New test file: src/agents/models-config.providers.secrets.bedrock-apikey.test.ts (7 tests)
- ✅ No apiKey injection when no env vars set (instance role scenario)
- ✅ No apiKey injection when providerApiKeyResolver returns undefined
- ✅ Correct marker injection for AWS_ACCESS_KEY_ID
- ✅ Correct marker injection for AWS_PROFILE
- ✅ Correct marker injection for AWS_BEARER_TOKEN_BEDROCK
- ✅ Skip injection when apiKey already configured
- ✅ providerApiKeyResolver result used when it returns a value
Existing normalize tests: pass
Existing auth-provenance tests: pass
Existing model-auth-markers tests: pass
Lint: clean

Files changed

File	Change
`src/agents/models-config.providers.secrets.ts`	Fix `resolveAwsSdkApiKeyVarName` + guard `resolveMissingProviderApiKey`
`extensions/amazon-bedrock/discovery.ts`	Fix `resolveBedrockConfigApiKey`
`src/agents/models-config.providers.secrets.bedrock-apikey.test.ts`	New regression tests

Closes #49891 Closes #50699 Fixes #54274

Changed files

extensions/amazon-bedrock/discovery.ts (modified, +4/-2)
src/agents/models-config.providers.secrets.bedrock-apikey.test.ts (added, +135/-0)
src/agents/models-config.providers.secrets.ts (modified, +19/-4)

PR #61194: Fix/bedrock aws sdk apikey injection

Repository: openclaw/openclaw
Author: wirjo
State: closed | merged: True
Link: https://github.com/openclaw/openclaw/pull/61194

Description (problem / solution / changelog)

fix(bedrock): stop injecting fake apiKey marker for aws-sdk auth when no env vars exist

Problem When the Bedrock provider uses auth: "aws-sdk" with no AWS environment variables set (EC2 instance roles, ECS task roles, Lambda, etc.), the gateway startup injects apiKey: "AWS_PROFILE" into the provider config — even though AWS_PROFILE is not set.

This happens because resolveAwsSdkApiKeyVarName() unconditionally falls back to "AWS_PROFILE": // Before function resolveAwsSdkApiKeyVarName(env): string { return resolveAwsSdkEnvVarName(env) ?? "AWS_PROFILE"; // always returns a string }

The same pattern exists in the extension's resolveBedrockConfigApiKey().

The downstream auth resolver then sees apiKey: "AWS_PROFILE", recognizes it as an AWS SDK marker via isNonSecretApiKeyMarker(), but isKnownEnvApiKeyMarker() excludes AWS SDK markers — so resolveUsableCustomProviderApiKey() returns null. The auth chain eventually fails with:

No API key found for amazon-bedrock.

This is the root cause behind three separate bug reports:

• #49891 — Gateway injects apiKey: "AWS_PROFILE" on every startup, breaking auth • #50699 — Bedrock provider does not recognize environment AWS credentials • #54274 — Intermittent main session failures after upgrade (instance role)

Root Cause

Two functions (resolveAwsSdkApiKeyVarName and resolveBedrockConfigApiKey) fall back to "AWS_PROFILE" when resolveAwsSdkEnvVarName() returns undefined (no AWS env vars detected). The write path then persists this string as an apiKey marker, but the read path doesn't resolve it to a usable credential — creating a permanent auth failure state.

Fix

resolveAwsSdkApiKeyVarName() (src/agents/models-config.providers.secrets.ts):

• Return undefined instead of "AWS_PROFILE" when no AWS env vars exist

resolveBedrockConfigApiKey() (extensions/amazon-bedrock/discovery.ts):

• Same fix — return undefined instead of falling back to "AWS_PROFILE"

resolveMissingProviderApiKey() (src/agents/models-config.providers.secrets.ts): • Guard both the providerApiKeyResolver branch and the direct aws-sdk branch • If the resolver returns nothing, return the provider config unchanged (no apiKey injected) • The AWS SDK credential chain then resolves credentials at request time via IMDS/ECS/etc.

When AWS env vars are present (AWS_ACCESS_KEY_ID, AWS_PROFILE, AWS_BEARER_TOKEN_BEDROCK), the marker is still injected correctly — no behavior change for those setups.

Verification

• New test file: src/agents/models-config.providers.secrets.bedrock-apikey.test.ts (7 tests) • ✅ No apiKey injection when no env vars set (instance role scenario) • ✅ No apiKey injection when providerApiKeyResolver returns undefined • ✅ Correct marker injection for AWS_ACCESS_KEY_ID • ✅ Correct marker injection for AWS_PROFILE • ✅ Correct marker injection for AWS_BEARER_TOKEN_BEDROCK • ✅ Skip injection when apiKey already configured • ✅ providerApiKeyResolver result used when it returns a value • Existing normalize tests: pass • Existing auth-provenance tests: pass • Existing model-auth-markers tests: pass • Lint: clean

Files changed

File	Change
src/agents/models-config.providers.secrets.ts	Fix resolveAwsSdkApiKeyVarName + guard resolveMissingProviderApiKey
extensions/amazon-bedrock/discovery.ts	Fix resolveBedrockConfigApiKey
src/agents/models-config.providers.secrets.bedrock-apikey.test.ts	New regression tests

Closes #49891 Closes #50699 Fixes #54274

Changed files

CHANGELOG.md (modified, +1/-0)
extensions/amazon-bedrock/discovery.test.ts (modified, +8/-1)
extensions/amazon-bedrock/discovery.ts (modified, +6/-2)
src/agents/models-config.providers.secrets.bedrock-apikey.test.ts (added, +158/-0)
src/agents/models-config.providers.secrets.ts (modified, +19/-4)

PR #62009: fix: propagate AWS SDK auth to pi's authStorage for Bedrock IMDS

Repository: openclaw/openclaw
Author: royosherove
State: open | merged: False
Link: https://github.com/openclaw/openclaw/pull/62009

Description (problem / solution / changelog)

Problem

When Bedrock auth resolves via AWS SDK (IMDS, env vars, or profiles), the auth controller in pi-embedded-runner does an early return without calling setRuntimeApiKey(). This means pi's authStorage never learns about Bedrock credentials, causing "No API key found for amazon-bedrock" errors even though AWS SDK auth was successfully resolved.

This is the root cause of Bedrock auth failures on EC2 instances using IAM instance profiles.

Fix

auth-controller.ts: When aws-sdk auth mode is detected and no API key is present, call setRuntimeApiKey() with a sentinel value (__aws_sdk_auth__) so pi's hasConfiguredAuth() recognizes the provider as authenticated.
model-auth-runtime-shared.ts: Detect AWS_EXECUTION_ENV and ECS_CONTAINER_METADATA_URI as indicators that IMDS/container credentials are available, improving fallback detection for EC2/ECS environments where no explicit AWS env vars are set.

Environment

Amazon Linux 2023 on EC2 with IAM instance profile
Auth: Bedrock via IMDS (no API keys, no env vars)
Config: models.providers.amazon-bedrock.auth: "aws-sdk"
Current workaround: patching pi's dist files at install time (overwritten on every update)

A companion PR has been submitted to pi-mono with defensive checks in pi's hasConfiguredAuth() and _getRequiredRequestAuth().

Changed files

src/agents/model-auth-runtime-shared.ts (modified, +6/-0)
src/agents/pi-embedded-runner/run/auth-controller.ts (modified, +5/-0)

Code Example

⚠️ Agent failed before reply: All models failed (2): 
amazon-bedrock/global.anthropic.claude-opus-4-6-v1: No API key found for amazon-bedrock.
amazon-bedrock/global.anthropic.claude-sonnet-4-6: No API key found for amazon-bedrock.

RAW_BUFFERClick to expand / collapse

Describe the bug

After upgrading to 2026.3.23-2, Amazon Bedrock with auth: aws-sdk (EC2 instance role) intermittently fails in the main chat session — not just isolated/cron contexts.

Error:

⚠️ Agent failed before reply: All models failed (2): 
amazon-bedrock/global.anthropic.claude-opus-4-6-v1: No API key found for amazon-bedrock.
amazon-bedrock/global.anthropic.claude-sonnet-4-6: No API key found for amazon-bedrock.

Environment

OpenClaw version: 2026.3.23-2 (worked on 2026.3.13)
Platform: Linux x64, AWS EC2 (c8i.2xlarge, us-west-2)
Provider: amazon-bedrock with auth: "aws-sdk"
Credential source: EC2 instance role (arn:aws:sts::600413481647:assumed-role/Openclaw/i-...)
Channel: Telegram
Gateway: systemd user service

Steps to reproduce

Configure amazon-bedrock provider with auth: "aws-sdk" on an EC2 instance (no static API keys)
Upgrade to 2026.3.23-2
Chat normally — works most of the time
Intermittently, the main session fails with "No API key found for amazon-bedrock"
Only workaround is /new to start a fresh session

Key observations

Not the same as #49891: This affects the main chat session, not just models.json injection. The apiKey: "AWS_PROFILE" injection issue from #49891 is a separate problem.
Not the same as #53592: That issue is about /btw isolated contexts. This bug affects the primary agent turn.
Regression: Did not occur on 2026.3.13. Started immediately after upgrading to 2026.3.23-2.
Intermittent: Sometimes works, sometimes fails. No pattern found — EC2 instance metadata is healthy (aws sts get-caller-identity always succeeds).
Gateway process has no AWS env vars: The systemd service does not set AWS_ACCESS_KEY_ID or AWS_PROFILE — it relies entirely on IMDS via the AWS SDK credential chain.
Cron jobs also affected: memory-digest cron jobs for 3/6 agents fail with the same error (the other 3 succeed, suggesting a timing/race condition).

Suspected root cause

Something in the 2026.3.23 auth refactoring (possibly the "Auth/OpenAI tokens: stop live gateway auth-profile writes from reverting freshly saved credentials" change) may have altered the credential resolution path for aws-sdk providers, causing the SDK credential chain to be bypassed or cached incorrectly.

The gateway process relies on EC2 IMDS for credentials (no env vars). The resolveAwsSdkEnvVarName() function returns undefined when no AWS env vars are set, which falls back to "AWS_PROFILE" — but AWS_PROFILE is also not set in the gateway environment.

Workaround attempted

Added AWS_PROFILE=default to gateway systemd environment + created ~/.aws/config with credential_source = Ec2InstanceMetadata
Removed injected apiKey: "AWS_PROFILE" from all agent models.json files
Results pending gateway restart

Related issues

#49891 (apiKey: "AWS_PROFILE" injection — same reporter, different symptom)
#53592 (/btw fails with aws-sdk auth — same version, isolated context)
#30374 (Model failover never triggers for Bedrock aws-sdk auth)

extent analysis

Fix Plan

To resolve the intermittent "No API key found for amazon-bedrock" error, follow these steps:

Update the resolveAwsSdkEnvVarName() function: Modify the function to correctly handle cases where no AWS environment variables are set. This can be achieved by checking for the presence of AWS_ACCESS_KEY_ID or AWS_PROFILE and returning the correct credential source.
Set the AWS_PROFILE environment variable: Add AWS_PROFILE=default to the gateway systemd environment file to ensure the aws-sdk provider uses the correct credential source.
Create a ~/.aws/config file: Create a ~/.aws/config file with the following content:

[default]
credential_source = Ec2InstanceMetadata

This will configure the aws-sdk provider to use the EC2 instance metadata as the credential source. 4. Remove injected apiKey from models.json files: Remove any injected apiKey: "AWS_PROFILE" from the agent models.json files to prevent conflicts with the updated credential resolution.

Example Code

// Update the resolveAwsSdkEnvVarName() function
function resolveAwsSdkEnvVarName() {
  if (process.env.AWS_ACCESS_KEY_ID) {
    return 'AWS_ACCESS_KEY_ID';
  } else if (process.env.AWS_PROFILE) {
    return 'AWS_PROFILE';
  } else {
    // If no AWS env vars are set, return the default credential source
    return 'Ec2InstanceMetadata';
  }
}

Verification

To verify the fix, restart the gateway service and test the chat session. The "No API key found for amazon-bedrock" error should no longer occur intermittently.

Extra Tips

Ensure the EC2 instance metadata is healthy and accessible by running aws sts get-caller-identity.
If issues persist, check the gateway logs for any errors related to credential resolution or the aws-sdk provider.
Consider implementing a retry mechanism for the aws-sdk provider to handle any temporary credential resolution issues.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #inference speed #output truncation #response parsing #generation error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

openclaw - ✅(Solved) Fix [Bug]: Bedrock aws-sdk auth intermittently fails in main session after 2026.3.23 upgrade (EC2 instance role) [4 pull requests, 1 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Suspected root cause

Fix Action

Fix / Workaround

Workaround attempted

PR fix notes

PR #1: fix(bedrock): stop injecting fake apiKey marker for aws-sdk auth when no env vars exist

Description (problem / solution / changelog)

fix(bedrock): stop injecting fake apiKey marker for aws-sdk auth when no env vars exist

Problem

Root Cause

Fix

Verification

Files changed

Changed files

PR #61194: Fix/bedrock aws sdk apikey injection

Description (problem / solution / changelog)

fix(bedrock): stop injecting fake apiKey marker for aws-sdk auth when no env vars exist

Changed files

PR #62009: fix: propagate AWS SDK auth to pi's authStorage for Bedrock IMDS

Description (problem / solution / changelog)

Problem

Fix

Environment

Related

Changed files

Code Example

Describe the bug

Environment

Steps to reproduce

Key observations

Suspected root cause

Workaround attempted

Related issues

extent analysis

Fix Plan

Example Code

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

fix(bedrock): stop injecting fake `apiKey` marker for `aws-sdk` auth when no env vars exist