openclaw - 💡(How to fix) Fix Feature: cache-aware sticky fallback to prevent prompt cache bouncing across providers [1 participants]

Summary

When the primary provider (e.g. Anthropic direct) intermittently times out and falls back to a secondary provider (e.g. OpenRouter/Bedrock), the per-request fallback design causes prompt cache bouncing: each successful primary request warms cache on Provider A, while each fallback request needs a full cache reload on Provider B. The result is near-zero cache hit rates on both providers simultaneously.

Observed impact

In a real session with ~160K context on Opus 4.6:

• Primary (Anthropic) intermittently times out
• Fallback (OpenRouter/Bedrock) handles those requests
• Bedrock caches 22,935 tokens (14%) instead of 160K+ (99%)
• Every fallback turn costs ~$0.90 instead of ~$0.09
• 36 Opus requests in one session cost $37.39 — estimated $5-7 with proper caching
• Pattern confirmed: OR CSV shows time gaps (primary succeeded) followed by immediate cache drops on fallback

Root cause

Model fallback is per-request: each new turn independently tries the primary provider first. There is no mechanism to "stick" to the fallback provider long enough for its prompt cache to warm and persist across turns.

• overloadedBackoffMs only controls delay within a single request's retry chain, not across requests
• Auth profile cooldown (1min) is too short and not cache-aware
• No config exists for "stay on fallback for N minutes across future requests"

Proposed solution

Add a sticky fallback window that keeps subsequent requests on the fallback provider after a failover event:

{
  auth: {
    cooldowns: {
      // After failover, stay on fallback provider for this duration
      // before probing primary again. Aligns with prompt cache TTL.
      stickyFallbackMs: 300000,  // 5 minutes (matches cache TTL)
      // Or: "cache-ttl" to auto-derive from cacheRetention setting
      stickyFallbackMode: "fixed",  // "fixed" | "cache-ttl"
    }
  }
}

Ideal behavior:

1. Primary times out → fall to secondary
2. Secondary succeeds → lock to secondary for stickyFallbackMs
3. During sticky window, all new requests go directly to secondary (cache builds)
4. After window expires, probe primary once
5. If primary succeeds → switch back (cache will warm on primary over next turns)
6. If primary fails → extend sticky window

This ensures the prompt cache on whichever provider is active stays warm.

Environment

• OpenClaw 4.5 (3e72c03)
• Primary: Anthropic direct (Opus 4.6, cacheRetention: "long")
• Fallback: OpenRouter Anthropic (cacheRetention: "short")
• Context size: ~160-190K tokens
• Cache TTL: 5min (short) / 1hr (long)

Workaround

Currently using manual provider switching — disable fallback entirely and switch providers manually during outages. Works but not automated.