openclaw - ✅(Solved) Fix image/video/music generate tools eagerly list capability providers, causing ~15-18s reload per turn [2 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#74096Fetched 2026-04-30 06:28:29
View on GitHub
Comments
1
Participants
2
Timeline
7
Reactions
0
Timeline (top)
cross-referenced ×6commented ×1

The image-generate, video-generate, and music-generate agent tools eagerly list runtime capability providers during tool registration, even when the agent has an explicit and complete modelConfig and the provider list goes unused. On hosted gateways with plugins.entries non-empty, each provider listing triggers a full plugin-registry reload (~5–6s per call), adding ~15–18s of wall-clock latency to every agent turn for capabilities the agent isn't using.

Root Cause

The image-generate, video-generate, and music-generate agent tools eagerly list runtime capability providers during tool registration, even when the agent has an explicit and complete modelConfig and the provider list goes unused. On hosted gateways with plugins.entries non-empty, each provider listing triggers a full plugin-registry reload (~5–6s per call), adding ~15–18s of wall-clock latency to every agent turn for capabilities the agent isn't using.

Fix Action

Fixed

PR fix notes

PR #74097: perf(agents/tools): lazily list capability providers in tool model-config resolution

Description (problem / solution / changelog)

Summary

  • Problem: resolveCapabilityModelConfigForTool (src/agents/tools/media-tool-shared.ts:246) takes providers: CapabilityProvider[] as a required eagerly-evaluated parameter, and short-circuits via hasToolModelConfig(explicit) after the caller has already computed the provider list. The 3 callers (image-generate-tool.ts:202, video-generate-tool.ts:232, music-generate-tool.ts:138) call listRuntime{Image,Video,Music}GenerationProviders({ config: cfg }) eagerly. On hosted gateways with plugins.entries populated, each provider listing triggers a full plugin-registry reload through resolvePluginCapabilityProviders → loadOpenClawPlugins (~5–6s per call).
  • Why it matters: For agents with an explicit, complete modelConfig for image / video / music generation, the provider list is pure waste — resolveCapabilityModelConfigForTool returns at the hasToolModelConfig(explicit) check without using it. With 3 generation tools registered on every agent, that's ~15–18s of wasted wall-clock per turn on hosted gateways. Repro and instrumented stack traces in #74096.
  • What changed: providers is now () => CapabilityProvider[] (a thunk). It's only invoked when the function falls through to candidate resolution (i.e. when modelConfig is incomplete). The 3 internal callers pass thunks. Two regression tests added in media-tool-shared.test.ts.
  • What did NOT change (scope boundary): Behavior when modelConfig is incomplete (still lists providers and resolves candidates). The provider listing logic itself (listRuntime{Image,Video,Music}GenerationProviders). The capability-provider runtime's caching behavior (separate concern, see #73793). All existing image/video/music tool tests pass unchanged.

Change Type (select all)

  • Bug fix
  • Refactor required for the fix

Scope (select all touched areas)

  • Skills / tool execution

Linked Issue/PR

  • Closes #74096
  • Related #73793 (independent capability-provider cache fix; this PR helps even without that one)
  • This PR fixes a bug or regression

Root Cause

resolveCapabilityModelConfigForTool short-circuits via hasToolModelConfig(explicit) before using the provider list, but the provider list is computed by callers before calling the function. The eager evaluation was the natural choice when the helper was first written, but on hosted gateways it has a hidden cost of ~5–6s per call because listRuntime*GenerationProviders triggers a plugin-registry reload (see #73793 for the deeper cache-miss issue on the registry side). Making the parameter lazy lets the helper choose to evaluate it only when needed.

Tests

  • Added 2 tests in src/agents/tools/media-tool-shared.test.ts:
    • does not invoke the providers thunk when an explicit modelConfig resolves
    • invokes the providers thunk when modelConfig is incomplete
  • All 116 existing tests in image-generate-tool.test.ts / video-generate-tool.test.ts / music-generate-tool.test.ts pass.

Verification

$ pnpm vitest run src/agents/tools/media-tool-shared.test.ts \
                  src/agents/tools/image-generate-tool.test.ts \
                  src/agents/tools/video-generate-tool.test.ts \
                  src/agents/tools/music-generate-tool.test.ts
 Test Files  6 passed (6)
      Tests  116 passed (116)

Changed files

  • src/agents/tools/image-generate-tool.ts (modified, +1/-1)
  • src/agents/tools/media-tool-shared.test.ts (modified, +37/-1)
  • src/agents/tools/media-tool-shared.ts (modified, +16/-3)
  • src/agents/tools/music-generate-tool.ts (modified, +1/-1)
  • src/agents/tools/video-generate-tool.ts (modified, +1/-1)

PR #74118: perf(agents/runtime): short-circuit ensureRuntimePluginsLoaded when active registry exists

Description (problem / solution / changelog)

Summary

  • Problem: ensureRuntimePluginsLoaded (src/agents/runtime-plugins.ts:6) is called from dispatchReplyFromConfig on every inbound message and rebuilds the entire plugin registry on each call. It builds a 3-field options object (config, workspaceDir, runtimeOptions) and hands it to resolveRuntimePluginRegistry. getCompatibleActivePluginRegistry's strict cacheKey equality comparison fails — boot's options set has 9+ fields (onlyPluginIds, activationSourceConfig, autoEnabledReasons, etc.), so the two hashes always differ. The fall-through runs a full loadOpenClawPlugins: Jiti-loads every plugin module, re-validates manifests, re-runs each plugin's register().
  • Why it matters: ~5–6s of wasted wall-clock per inbound message on hosted gateways. The active registry is already a valid answer; rebuilding it produces no useful state change. Repro and instrumented stack traces in #74117.
  • What changed: Fast path in ensureRuntimePluginsLoaded — if getActivePluginRegistry() is populated, return immediately. The function's intent is to ensure plugins are loaded; if they already are, the goal is met. Updates the existing test (which was asserting one call — that assertion was wrong; the prior behavior reactivated every time) into a proper regression test that asserts zero calls when the active registry exists.
  • What did NOT change (scope boundary): Behavior when no active registry exists (still builds load options and calls resolveRuntimePluginRegistry). loadGatewayPlugins's boot path. getCompatibleActivePluginRegistry's strict-equality logic — that's a more intrusive change to file separately if desired. Plugin reconfiguration paths that explicitly invalidate the active registry (setActivePluginRegistry, gateway restart on config write) — those continue to work as before.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution

Linked Issue/PR

  • Closes #74117
  • Related #73793 / #74096 (independent fixes for the same class of strict-cache-key issue in different code paths)
  • This PR fixes a bug or regression

Root Cause

ensureRuntimePluginsLoaded is the simplest member of a family of "ensure registry is loaded" helpers (cf. ensurePluginRegistryLoaded in plugins/runtime/runtime-registry-loader.ts, which has its own scope-tracking short-circuit via pluginRegistryLoaded rank). The agent-side ensureRuntimePluginsLoaded is a thin wrapper that defers all caching responsibility to resolveRuntimePluginRegistry → getCompatibleActivePluginRegistry. That worked when the cache-key derivation matched between boot and dispatch. It stopped working when the boot path grew additional options (onlyPluginIds, activationSourceConfig, etc.) that the dispatch path doesn't supply, but the strict-equality check was never relaxed to a "is-compatible-subset" check.

A more thorough fix would be to make getCompatibleActivePluginRegistry accept subset-compatibility, but that's a larger surface area touching multiple call sites. This PR takes the minimal approach for this specific call site: short-circuit at the helper level. The function's contract is "ensure plugins are loaded", not "always rebuild the registry".

Tests

  • Updated src/agents/runtime-plugins.test.ts:
    • Added getActivePluginRegistry mock to the existing vi.mock("../plugins/runtime.js", ...) block.
    • Renamed and rewrote the existing test (does not reactivate plugins when a process already has an active registry — which was incorrectly asserting one call) into short-circuits without rebuilding load options when an active registry exists, asserting zero calls to resolveRuntimePluginRegistry.
    • Other tests (resolves runtime plugins through the shared runtime helper when no active registry is present, etc.) now mock getActivePluginRegistry to return undefined so they exercise the fall-through path.

Verification

$ pnpm vitest run src/agents/runtime-plugins.test.ts
 Test Files  2 passed (2)
      Tests  8 passed (8)

Changed files

  • src/agents/runtime-plugins.test.ts (modified, +15/-4)
  • src/agents/runtime-plugins.ts (modified, +24/-1)

Code Example

register #3 caller: ... resolvePluginCapabilityProviders ← resolvePluginImageGenerationProviders ← buildProviderMaps ← listImageGenerationProviders
register #4 caller: ... resolvePluginCapabilityProviders ← resolvePluginVideoGenerationProviders ← buildProviderMaps ← listVideoGenerationProviders
register #5 caller: ... resolvePluginCapabilityProviders ← resolvePluginMusicGenerationProviders ← buildProviderMaps ← listMusicGenerationProviders

---

register #3 caller: runPluginRegisterSync ← loadOpenClawPlugins ← resolveRuntimePluginRegistry ← resolvePluginCapabilityProviders ← resolvePluginImageGenerationProviders ← buildProviderMaps ← listImageGenerationProviders
register #4 caller: ... ← resolvePluginVideoGenerationProviders ← buildProviderMaps ← listVideoGenerationProviders
register #5 caller: ... ← resolvePluginMusicGenerationProviders ← buildProviderMaps ← listMusicGenerationProviders
RAW_BUFFERClick to expand / collapse

Bug type

Behavior bug (incorrect output/state without crash) — performance.

Beta release blocker

No

Summary

The image-generate, video-generate, and music-generate agent tools eagerly list runtime capability providers during tool registration, even when the agent has an explicit and complete modelConfig and the provider list goes unused. On hosted gateways with plugins.entries non-empty, each provider listing triggers a full plugin-registry reload (~5–6s per call), adding ~15–18s of wall-clock latency to every agent turn for capabilities the agent isn't using.

Steps to reproduce

  1. Run any gateway with openclaw.json containing at least one plugin in plugins.entries. Confirmed against 2026.4.26-f53b52ad6d21.
  2. Connect any agent that registers the built-in image-generate / video-generate / music-generate tools (i.e. all agents, since these are core agent tools).
  3. Send a single inbound message.
  4. Observe register() being invoked 3 additional times during the turn (4 total counting the initial load), with stack runPluginRegisterSync ← loadOpenClawPlugins ← resolveRuntimePluginRegistry ← resolvePluginCapabilityProviders ← resolvePluginImageGenerationProviders ← buildProviderMaps ← listImageGenerationProviders (and likewise for video / music).

Observed log (top of stack on the 3 in-turn reloads):

register #3 caller: ... resolvePluginCapabilityProviders ← resolvePluginImageGenerationProviders ← buildProviderMaps ← listImageGenerationProviders
register #4 caller: ... resolvePluginCapabilityProviders ← resolvePluginVideoGenerationProviders ← buildProviderMaps ← listVideoGenerationProviders
register #5 caller: ... resolvePluginCapabilityProviders ← resolvePluginMusicGenerationProviders ← buildProviderMaps ← listMusicGenerationProviders

Expected behavior

When an agent has an explicit complete modelConfig for image / video / music generation, resolveCapabilityModelConfigForTool short-circuits at the hasToolModelConfig(explicit) check and returns without using the provider list. In that path, listing providers is pure waste — and on hosted gateways it's a 5–6s cache miss per call.

Actual behavior

resolveCapabilityModelConfigForTool (in src/agents/tools/media-tool-shared.ts:246) takes providers: CapabilityProvider[] as a required eagerly-evaluated parameter. The 3 callers (image-generate-tool.ts:202, video-generate-tool.ts:232, music-generate-tool.ts:138) call listRuntime{Image,Video,Music}GenerationProviders({ config: cfg }) to compute it before calling resolveCapabilityModelConfigForTool, so providers are listed even when hasToolModelConfig(explicit) would short-circuit.

listRuntime{...}GenerationProviders flows through resolvePluginCapabilityProviders (src/plugins/capability-provider-runtime.ts:316) which calls resolveRuntimePluginRegistry({config: compatConfig, activate: false}). That's a cache key not held by the active registry, so it falls through to loadOpenClawPlugins, re-imports every plugin, and re-runs each plugin's register() (~5–6s each in production).

This is independent of #73793 — it's about avoiding the lookup entirely when not needed.

Environment

  • OpenClaw 2026.4.26-f53b52ad6d21 (production gateway)
  • Node v24.14.0
  • Linux x64 (Elestio-hosted Docker container)
  • Active plugins: memory-core, plus a third-party channel plugin

Logs / evidence

Per-turn timing from instrumented third-party plugin (paths trimmed):

register #3 caller: runPluginRegisterSync ← loadOpenClawPlugins ← resolveRuntimePluginRegistry ← resolvePluginCapabilityProviders ← resolvePluginImageGenerationProviders ← buildProviderMaps ← listImageGenerationProviders
register #4 caller: ... ← resolvePluginVideoGenerationProviders ← buildProviderMaps ← listVideoGenerationProviders
register #5 caller: ... ← resolvePluginMusicGenerationProviders ← buildProviderMaps ← listMusicGenerationProviders

Proposed fix

Change the providers parameter on resolveCapabilityModelConfigForTool from CapabilityProvider[] to () => CapabilityProvider[] so the listing is deferred until the function actually needs it (i.e. when modelConfig is incomplete and candidate resolution must run). Update the 3 internal callers to pass thunks. PR with fix + regression tests filed alongside.

extent analysis

TL;DR

Change the providers parameter on resolveCapabilityModelConfigForTool to a function that returns CapabilityProvider[] to defer provider listing until necessary.

Guidance

  • Update the resolveCapabilityModelConfigForTool function to accept a thunk (a function that returns CapabilityProvider[]) instead of an eagerly evaluated array of providers.
  • Modify the internal callers (image-generate-tool.ts, video-generate-tool.ts, music-generate-tool.ts) to pass thunks that compute the providers only when needed.
  • Verify that the fix reduces the latency introduced by the unnecessary provider listing by measuring the time it takes to process a single inbound message.
  • Test the fix with different modelConfig scenarios to ensure it correctly short-circuits when the config is explicit and complete.

Example

// Before
function resolveCapabilityModelConfigForTool(providers: CapabilityProvider[]) {
  // ...
}

// After
function resolveCapabilityModelConfigForTool(getProviders: () => CapabilityProvider[]) {
  // ...
}

Notes

This fix assumes that the resolveCapabilityModelConfigForTool function is the only place where the providers are used. If there are other parts of the code that rely on the eager evaluation of the providers, additional changes may be necessary.

Recommendation

Apply the proposed fix to defer the provider listing until necessary, as it directly addresses the root cause of the issue and should significantly reduce the introduced latency.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

When an agent has an explicit complete modelConfig for image / video / music generation, resolveCapabilityModelConfigForTool short-circuits at the hasToolModelConfig(explicit) check and returns without using the provider list. In that path, listing providers is pure waste — and on hosted gateways it's a 5–6s cache miss per call.

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING