openclaw - 💡(How to fix) Fix v2026.3.28: Runaway API calls burned entire monthly budget across all providers without user action [1 participants]

Summary

OpenClaw release 2026.3.28 (March 31, 2026) caused uncontrolled runaway API calls across all configured LLM providers simultaneously on March 31 and April 1, 2026. No user interaction triggered this — it happened autonomously. The entire monthly budget across Anthropic, OpenAI, Google (Gemini), and xAI (Grok) was consumed in approximately 2 days.

Environment

• OpenClaw version: 2026.3.28 (upgraded from 2026.3.24 on March 31)
• OS: macOS Darwin 24.6.0 (arm64) — Mac Studio
• Gateway: launchd-managed, running locally
• Providers affected: Anthropic, OpenAI, Google Gemini, xAI Grok, Ollama

What happened

1. Upgraded from 2026.3.24 → 2026.3.28 on March 31
2. Shortly after, observed massive API spend across all providers
3. Auth profiles began returning 401 errors (tokens invalidated or rotated unexpectedly)
4. OpenClaw's failover system cycled through all providers repeatedly in a loop
5. Each failed attempt billed the next provider in the fallback chain
6. This continued autonomously over ~48 hours (March 31 – April 1)
7. Monthly budget caps were hit on ALL providers

Evidence from logs

Repeated pattern observed in gateway logs:

embedded_run_agent_end: HTTP 401 authentication_error: Invalid bearer token
auth_profile_failure_state_updated: cooldown set
embedded_run_failover_decision: rotate_profile
model_fallback_decision: candidate_failed → trying next provider

This cycle repeated hundreds of times across all providers simultaneously with no user-facing alert or pause.

Root cause hypothesis

The 2026.3.28 release appears to have:

1. Invalidated or corrupted existing auth profile tokens causing immediate 401s on all Anthropic profiles
2. The failover/retry system had no effective rate limiting or spend guard — it cycled through all providers indefinitely
3. No user-facing alert was generated despite runaway spending across all providers

Impact

• Financial: Entire monthly API budget consumed across 4 paid providers in ~48 hours
• Service: Assistant was non-functional for 2+ days requiring extensive manual repair
• Trust: Required deep diagnosis of auth-profiles.json, gateway logs, and manual config surgery to restore

Suggested fixes

1. Hard failover limit: Stop after N total cross-provider failover attempts per session
2. Spend rate guard: If N failed auth attempts occur within a short window, pause all API calls and alert the user immediately
3. Auth invalidation alert: If ALL profiles for a provider return 401 simultaneously, surface an immediate user-facing alert rather than silently retrying
4. Billing lockout default: Default billing lockout should be 24h not 5h — we had to set this manually
5. Release notes: Breaking auth changes should be prominently flagged in release notes

Workarounds applied

• Manually identified and replaced corrupted auth profiles in ~/.openclaw/agents/*/agent/auth-profiles.json
• Set auth.cooldowns.billingBackoffHours: 24
• Set auth.cooldowns.overloadedProfileRotations: 1
• Added daily spend monitoring cron job as a custom guard
• Set hard monthly caps at each provider billing console

Stable version

2026.3.24 — confirmed running without issues before the March 31 upgrade.