openclaw - 💡(How to fix) Fix Failover loop: timeout-heavy candidates starve later fallbacks in run time budget [1 participants]

1. The run itself times out and is marked as error [agent/embedded] embedded run agent end: isError=true model=gemini-3.1-pro-preview error=LLM request failed: network connection error.

Fix Plan

To address the infinite retry loop issue, we will implement a circuit-breaker mechanism for candidates that fail with a timeout. This will prevent the fallback chain from repeatedly attempting to use a candidate that is not responding within the allotted time.

Step-by-Step Solution:

1. Implement Circuit Breaker:

   • Introduce a circuitBreaker object to track the status of each candidate.
   • When a candidate times out, mark it as unavailable in the circuitBreaker for a cooldown period.

2. Modify Fallback Logic:

   • Before attempting a candidate, check its status in the circuitBreaker.
   • If a candidate is marked as unavailable, skip it and proceed to the next candidate in the fallback chain.

3. Example Code Snippet:

   const circuitBreaker = {};
   const cooldownPeriod = 30000; // 30 seconds
   
   function isCandidateAvailable(candidate) {
     return !(candidate in circuitBreaker) || circuitBreaker[candidate] < Date.now();
   }
   
   function markCandidateUnavailable(candidate) {
     circuitBreaker[candidate] = Date.now() + cooldownPeriod;
   }
   
   // Example usage in the fallback chain evaluation
   function evaluateFallbackChain(candidates) {
     for (const candidate of candidates) {
       if (isCandidateAvailable(candidate)) {
         try {
           // Attempt to use the candidate
           // ...
         } catch (error) {
           if (error.reason === 'timeout') {
             markCandidateUnavailable(candidate);
           }
           // Proceed to the next candidate
         }
       }
     }
   }

Verification

To verify that the fix worked:

• Set up a test scenario with a model override pointing to an unavailable model.
• Monitor the logs for the candidate_failed and candidate_succeeded messages.
• The working candidate (e.g., Sonnet) should succeed within a reasonable time frame without the infinite retry loop.

Extra Tips

• Consider implementing a more sophisticated circuit-breaker strategy, such as one that adapts the cooldown period based on the number of consecutive failures.
• Review the fallback chain configuration to ensure that the order of candidates makes sense and that there are no unnecessary or redundant candidates.