PR #70990: feat(plugins): add model failover and terminal failure hooks

• Repository: openclaw/openclaw
• Author: mantisai-bot
• State: open | merged: False
• Link: https://github.com/openclaw/openclaw/pull/70990

Description (problem / solution / changelog)

PR Draft — #70976 model failover hooks

Title

feat(plugins): add model failover and terminal failure hooks

Summary

• Problem: OpenClaw exposes model failover and terminal all-models-failed outcomes mostly through logs, so plugins and operators cannot react in-process without log scraping.
• Why it matters: This makes alerting and automation brittle, delayed, and hard to build on top of the existing failover path.
• What changed: Added two fire-and-forget plugin hooks, model_failover and model_failure_terminal, and emitted them from the existing failover-decision and terminal before-reply failure seams.
• What did NOT change (scope boundary): This PR does not add non-verbose fallback notices, workspace-context re-injection, or circuit-breaker/policy behavior.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #70976
• Related #65824
• This PR fixes a bug or regression

Root Cause (if applicable)

• Root cause: N/A
• Missing detection / guardrail: N/A
• Contributing context (if known): N/A

Regression Test Plan (if applicable)

N/A

User-visible / Behavior Changes

• Plugins can now register model_failover hooks and receive structured failover decision events.
• Plugins can now register model_failure_terminal hooks and receive structured before-reply terminal failure events, including fallback-attempt summaries for all-models-failed outcomes.
• No default user-facing fallback messaging behavior changed.

Diagram (if applicable)

Before:
[model error] -> [log failover decision] -> [fallback or final failure]
                     \-> plugins must scrape logs

After:
[model error] -> [log failover decision + model_failover hook] -> [fallback or final failure]
                                                               \-> [model_failure_terminal hook on before-reply collapse]

Security Impact (required)

• New permissions/capabilities? (Yes)
• Secrets/tokens handling changed? (No)
• New/changed network calls? (No)
• Command/tool execution surface changed? (No)
• Data access scope changed? (No)
• If any Yes, explain risk + mitigation:
  • This adds new plugin observability hooks. To reduce leakage risk, the new event surface does not expose raw provider errors and uses the sanitized terminal message path for model_failure_terminal.

Repro + Verification

Environment

• OS: macOS
• Runtime/container: local source checkout
• Model/provider: N/A (hook plumbing)
• Integration/channel (if any): N/A
• Relevant config (redacted): default local test config

Steps

1. Register a plugin hook for model_failover or model_failure_terminal using the hook runner test helpers.
2. Trigger the corresponding hook runner path or failover seam.
3. Verify the handler receives the structured event payload and that hook failures do not break the main flow.

Expected

• model_failover fires on failover decisions.
• model_failure_terminal fires on terminal before-reply failures.
• Hook execution is fire-and-forget / fail-open.

Actual

• Added targeted tests for both hook types and failover-observation hook emission.
• Full plugin Vitest suite passed after the change.

Evidence

Attach at least one:

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

• Verified scenarios:
  • reviewed the source seams where failover decisions and terminal failures are emitted
  • verified the new hook names are registered in the plugin hook surface
  • verified targeted tests pass for runner plumbing and failover-observation emission
  • verified the full plugin Vitest suite passes
• Edge cases checked:
  • no-op when no hook is registered
  • fail-open behavior when a hook throws
  • repeated decision emissions still invoke hooks correctly
• What you did not verify:
  • full-project tsc --noEmit on this host, because it OOMs due to an existing machine limitation
  • a live end-to-end provider outage scenario outside the test suite

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

• Backward compatible? (Yes)
• Config/env changes? (No)
• Migration needed? (No)
• If yes, exact upgrade steps:

Risks and Mitigations

• Risk: Hook payload scope may be seen as too narrow for v1 because it omits richer source-model context.
  • Mitigation: Keep this PR intentionally minimal and source-backed; richer payload fields can be added in follow-up work if maintainers want them.
• Risk: Hook emission could accidentally interfere with failover paths if blocking.
  • Mitigation: Both hooks are wired through existing fire-and-forget runVoidHook behavior.
• Risk: Terminal failure event content could leak too much provider detail.
  • Mitigation: model_failure_terminal uses the sanitized message path and structured attempt summaries instead of raw provider error blobs.

Changed files

• src/agents/pi-embedded-runner/run.ts (modified, +8/-0)
• src/agents/pi-embedded-runner/run/failover-observation.model-failover-hook.test.ts (added, +138/-0)
• src/agents/pi-embedded-runner/run/failover-observation.ts (modified, +51/-1)
• src/auto-reply/reply/agent-runner-execution.ts (modified, +36/-1)
• src/plugins/hook-types.ts (modified, +45/-0)
• src/plugins/hooks.model-failover.test.ts (added, +231/-0)
• src/plugins/hooks.ts (modified, +30/-0)