openclaw - 💡(How to fix) Fix Feature Request: Per-candidate retry count for model fallback (support pool-based/proxy providers) [6 comments, 2 participants]

Problem

When using pool-based/proxy API providers (e.g., third-party Anthropic resellers that rotate through a pool of API keys), OpenClaw's model fallback mechanism switches to the next candidate after a single failure instead of retrying the same candidate. This makes pool-based providers nearly unusable as fallback options.

Real-World Scenario

We use a proxy provider (TicketPro) that forwards requests to Anthropic's API through a rotating key pool. We ran a test of 30 consecutive requests:

With OpenClaw's current behavior, the proxy provider is tried once, gets a 502, and immediately falls back to the next provider — never getting a chance to hit a healthy key in the pool.

What Happens in Practice

TP fails (502) → switch to GLM → GLM rate_limit → switch to TP
→ TP in cooldown (30s skip) → probe TP → TP still 502
→ switch to GLM → GLM also cooldown → dead loop for 4-5 minutes

Every provider ends up in cooldown, and the fallback chain bounces between them until all are exhausted.

Root Cause Analysis

Based on source code inspection (model-fallback.ts):

1. No per-candidate retry: The fallback loop (for (candidate of candidates)) calls runFallbackAttempt() once per candidate, then continues to the next on failure
2. 502 classified as "timeout": classifyFailoverReasonFromHttpStatus(502) → "timeout", same as real network timeouts
3. Aggressive cooldown: calculateAuthProfileCooldownMs applies 30s → 60s → 5min escalating cooldown after failures
4. Cooldown skip + probe: During cooldown, candidates are skipped with a single probe attempt — but pool providers need multiple retries, not a probe

For pool-based providers, a single failure means "this particular key is bad, try another" — not "this provider is down, avoid it."

Expected Behavior

• Allow configuring per-candidate retry count (e.g., maxRetriesPerCandidate: 3) so pool-based providers get multiple attempts before being marked as failed
• Alternatively, treat timeout-classified errors differently from rate_limit/auth — timeouts from pool providers are often transient and worth retrying immediately
• Optionally, support a retryBeforeFallback: true/false flag per provider or per model entry

Suggested Configuration

agents:
  defaults:
    models:
      fallbacks:
        - provider: ticketpro
          model: claude-opus-4-6
          retriesBeforeFallback: 3  # NEW: retry this candidate up to 3 times
          retryDelayMs: 2000        # NEW: delay between retries
        - provider: zhipu-coding
          model: glm-5-turbo
          retriesBeforeFallback: 1  # default behavior: try once

Or a simpler global setting:

agents:
  defaults:
    models:
      fallbackMaxRetriesPerCandidate: 3  # applies to all fallback candidates

Environment

• OpenClaw 2026.3.x
• 17 agents in multi-tier hierarchy
• Proxy provider: TicketPro (Anthropic reseller with key pool rotation)
• Fallback chain: ticketpro/claude-opus-4-6 → zhipu-coding/glm-5-turbo

Related Issues

• #57906 — proposes reducing retries (opposite direction, but same config surface)

Tested with real production data: 30 consecutive requests to TicketPro, 40% success rate, consistent pool rotation behavior observed.