hermes - 💡(How to fix) Fix [Feature]: Cron job stage persistence + partial retry mechanism - Real world case: 2 million tokens wasted due to push failures [2 comments, 2 participants]

 1. Run script to check Volcengine API routing                
 2. Search news from 3 engines using multi-search-engine      
 skill                                                        
 3. LLM generates the briefing content                        
 4. Push to WeChat                                            
                                                              
 The Problem                                                  
 Today step 4 (push to WeChat) failed, and Hermes             
 automatically retried 10 times.                              
                                                              
 But instead of only retrying step 4, it re-executed the      
 ENTIRE job from step 1!                                      
                                                              
 Result:                                                      
 - Each retry costs ~80k tokens                               
 - 2 million tokens burned in one day                         
 - The push action itself costs almost 0 tokens               
 My Pain Point                                                
 Push failure is a WeChat API issue, it has nothing to do     
 with search and generation. The content was already          
 generated, why search and generate it all again?             
                                                              
 This is like asking an employee to write a proposal,         
 print it, and mail it. If the mail fails, and they start     
 over by re-writing the entire proposal again. It's just      
 not logical.                                                 
 My Temporary Workaround                                      
 I split the job into 4 separate cron jobs:                   
 1. 08:55 Routing check                                       
 2. 08:56 Search news → save to local JSON                    
 3. 08:59 Read JSON, generate briefing → save to local MD     
 4. 09:00 Read MD and push to WeChat                          
                                                              
 Now even if push fails 100 times, the tokens spent           
 earlier are never wasted.

 yaml                                                         
 workflow:                                                    
   name: Insurance Tech Daily Briefing                        
   stages:                                                    
     - name: Routing Check                                    
       run: python3 ~/.hermes/scripts/add-volc-route.sh       
       retry: 2                                               
                                                              
     - name: News Search                                      
       skill: multi-search-engine                             
       retry: 2                                               
       depends_on: Routing Check                              
       output: materials.json  # Output saved, downstream     
 stages can read                                              
                                                              
     - name: Content Generation                               
       skill: zhibao-insurance-tech-daily-briefing            
       retry: 1                                               
       depends_on: News Search                                
       input: materials.json                                  
       output: briefing.md                                    
                                                              
     - name: WeChat Push                                      
       run: hermes send --platform weixin --file              
 briefing.md                                                  
       retry: 5                                               
       depends_on: Content Generation                         
       input: briefing.md                                     
                                                              
 Core Requirements                                            
 1. Stage Isolation: Each stage can have independent          
 retry counts                                                 
 2. Output Persistence: Each stage's output can be saved      
 to file for downstream stages                                
 3. Partial Retry: If a stage fails, only retry that          
 specific stage, successful stages are preserved              
 4. Traceability: All intermediate outputs are preserved      
 for debugging                                                
 One Sentence Summary                                         
 > Don't let a push failure waste all the tokens spent on     
 search and generation. The cost of failure should be         
 limited to the stage that failed.                            
                                                              
 Thanks for building such an amazing product! 🙏