hermes - 💡(How to fix) Fix [OPS] CPU spikes to 144%+ due to uncoordinated kanban worker CI parallelism [1 participants]

Root Cause Analysis

Observed

Load average 7.58–8.93 on a 4-core VM (189–223% utilization), with CPU pressure some avg10 at 14.88 and sustained avg300 at 48.16.

Top CPU Consumers (sampled Mon 13:34 UTC)

17 Hermes kanban worker Python processes active across 10 boards. Multiple boards running CI simultaneously — each spawning tsc --noEmit + vitest with no inter-board coordination.

Memory Pressure

• 6.5G / 7.8G used (83%)
• 6.4G / 14G swap used — significant swap thrashing adding IO pressure (IO some avg10: 1.93)

Root Cause Chain

1. max_spawn=5 allows 5 concurrent kanban workers (per board? or global?)
2. Multiple boards run project-ci workflow simultaneously → each spawns tsc --noEmit + vitest
3. tsc processes are CPU-bound (~130% each), vitest workers compete for remaining cores
4. No admission control to throttle CPU-intensive CI steps when system is overloaded
5. Memory exhaustion leads to swap → compounding IO wait

Impact

• Worker timeouts and slowdowns
• Increased failure rate from resource starvation
• Potential cascading failures when timed-out workers retry

Recommended Actions

1. Immediate: Reduce max_spawn to 2–3 until per-worker CPU cgroup limits are in place
2. Short-term: Add CPU load guard — workers should check loadavg before spawning CI steps
3. Short-term: Serialize tsc across boards (single file-lock or queue for type-checking)
4. Medium-term: cgroup CPU limits per kanban worker process
5. Medium-term: vitest workers should be capped at 1 when system load > 75%

System Details

• Host: Linux 6.8.0-117-generic, 4 CPUs, 7.8G RAM, 14G swap
• Disk: 72G, 75% used
• 10 active kanban boards