Issue Station
Sign In
Tools hereToolbox

openclaw - ✅(Solved) Fix [Test gap]: claude-cli backend has no integration test catching the registration regression class [2 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#72576•Fetched 2026-04-28 06:34:15
View on GitHub
Comments
2
Participants
2
Timeline
7
Reactions
0
Author
1fanwang1fanwang
Participants
1fanwang1fanwangMaiHHConnectMaiHHConnect
Timeline (top)
referenced ×3commented ×2cross-referenced ×2

The claude-cli agent backend has shipped multiple distinct regressions in close succession:

  • #61093 (CLOSED, 2026.4.2): claude-cli/* model catalog failed to register on gateway start; all claude-cli/* requests failed with model_not_found.
  • #72077 (CLOSED, 2026.4.21+): claude-cli harness regression (workaround = claude-max-api-proxy).
  • #72434 (OPEN, 2026.4.24): Requested agent harness "claude-cli" is not registered. Channel startup fails on every restart.

A simple integration test asserting "claude-cli backend is registered and resolvable after default-config gateway boot" would have caught all three. None exists today.

gateway-codex-harness.live.test.ts is the parallel test for the codex harness — comprehensive but heavy (866 lines, gated on OPENCLAW_LIVE_CODEX_HARNESS=1). The codex coverage pattern works because codex regressions are rare; the asymmetry with claude-cli (3 regressions in 25 days vs zero) suggests a lighter, default-on guardrail is worth its weight specifically for claude-cli.

Error Message

Each of the three regressions surfaced as a runtime error the plugin author / operator has to diagnose by reading loader-*.js source. Adding this guardrail to upstream CI would:

Root Cause

Test gap (CI guardrail missing). Not a behavior bug — but a behavior bug class that has shipped 3+ times in the past 25 days because there's no test asserting it.

Fix Action

Fix / Workaround

  • #61093 (CLOSED, 2026.4.2): claude-cli/* model catalog failed to register on gateway start; all claude-cli/* requests failed with model_not_found.
  • #72077 (CLOSED, 2026.4.21+): claude-cli harness regression (workaround = claude-max-api-proxy).
  • #72434 (OPEN, 2026.4.24): Requested agent harness "claude-cli" is not registered. Channel startup fails on every restart.

PR fix notes

PR #72590: test(anthropic): cover legacy embeddedHarness migration path

Description (problem / solution / changelog)

What

Adds a single test inside extensions/anthropic/cli-migration.test.ts that gives buildAnthropicCliMigrationResult a 4.23-carry-over config — agents.defaults.embeddedHarness.runtime: "claude-cli" — and asserts the output configPatch emits the canonical agentRuntime.id. ~27 net lines, no behavior change.

Why

Regression coverage for #72434. buildAnthropicCliMigrationResult is well-tested for clean-config inputs (the existing it("keeps anthropic defaults and selects the claude-cli runtime", ...) passes a config with no embeddedHarness block). The 5b9be2cdb1 fix introduced the canonical agentRuntime.id key, but the legacy-input-shape path it's actually fixing has no explicit test pinning it.

This addresses part of #72576 (claude-cli backend regression-prevention test). Test-only; no production code touched.

Reproducer / verification

Locally on this branch:

$ pnpm test extensions/anthropic/cli-migration.test.ts -- -t 'migrates a 4.23-carry-over'
 Test Files  1 passed (1)
      Tests  1 passed | 11 skipped (12)
   Duration  4.22s

Environment: macOS 25.3.0, Node 22.22.2, openclaw repo at 6c1cffa7f8 (origin/main head at PR open time).

Why this passes "test-only / refactor" per CLAUDE.md

CODEOWNERS line: "maint/refactor/tests ok". This is a test-only change inside the anthropic extension package boundary; no plugin SDK seams added; no public contract change.

Changed files

  • extensions/anthropic/cli-migration.test.ts (modified, +27/-0)

PR #73122: fix claude-cli runtime harness registration

Description (problem / solution / changelog)

Summary

  • Register CLI backends as same-id embedded agent harness runtimes, so claude-cli can be resolved when selected through agent runtime policy.
  • Treat manifest cliBackends as agent-harness activation owners, so gateway startup loads the owning plugin for forced CLI runtimes.
  • Preserve the existing guard that CLI runtime aliases only route through the CLI path when they match the selected provider.

Root Cause

The Anthropic plugin registered claude-cli as a CLI backend, but the forced embedded runtime path looked only at registered agent harnesses. Configurations that selected the claude-cli runtime could therefore load the owning plugin but still fail to resolve a registered harness, or fall back to PI in paths that should use the CLI backend.

Validation

  • corepack pnpm test src/agents/harness/selection.test.ts src/plugins/activation-planner.test.ts src/plugins/channel-plugin-ids.test.ts src/plugins/loader.test.ts -- -t "CLI runtime|agentRuntime|agentHarness|cli backend|registers cli backends as forced embedded"
  • PATH=/tmp/openclaw-pnpm-shim:$PATH corepack pnpm check:changed

Notes: the temporary PR worktree initially lacked node_modules; corepack pnpm install succeeded on retry after sandboxed DNS failures. The optional Matrix crypto postinstall download logged a transient network reset but the install command exited 0.

Fixes #72576

Changed files

  • src/agents/harness/selection.ts (modified, +10/-2)
  • src/plugins/activation-planner.test.ts (modified, +18/-0)
  • src/plugins/activation-planner.ts (modified, +9/-3)
  • src/plugins/channel-plugin-ids.test.ts (modified, +10/-0)
  • src/plugins/loader.test.ts (modified, +37/-0)
  • src/plugins/registry.ts (modified, +137/-0)

Code Example

import { describe, expect, it } from "vitest";
import { loadBundledPluginForTest } from "<existing test helper>";
import { listAgentHarnesses } from "<existing harness registry>";

describe("anthropic extension: claude-cli backend post-load contract", () => {
  it("registers an agent harness resolvable as 'claude-cli' (or its canonical replacement)", async () => {
    const record = await loadBundledPluginForTest("anthropic");
    const ids = listAgentHarnesses().map((h) => h.id);
    // Either the legacy id is still resolvable, OR a documented migration exists.
    expect(
      ids.includes("claude-cli") || record.config.migrations?.runtime?.from === "claude-cli",
    ).toBe(true);
  });

  it("default-config carry-over from a prior version still resolves to a valid harness", async () => {
    const cfg = { agents: { defaults: { embeddedHarness: { runtime: "claude-cli" } } } };
    const result = await resolveHarnessForConfig(cfg);
    expect(result).toBeDefined();
    expect(result.id).toBeTruthy();
  });
});
RAW_BUFFERClick to expand / collapse

Issue type

Test gap (CI guardrail missing). Not a behavior bug — but a behavior bug class that has shipped 3+ times in the past 25 days because there's no test asserting it.

Summary

The claude-cli agent backend has shipped multiple distinct regressions in close succession:

  • #61093 (CLOSED, 2026.4.2): claude-cli/* model catalog failed to register on gateway start; all claude-cli/* requests failed with model_not_found.
  • #72077 (CLOSED, 2026.4.21+): claude-cli harness regression (workaround = claude-max-api-proxy).
  • #72434 (OPEN, 2026.4.24): Requested agent harness "claude-cli" is not registered. Channel startup fails on every restart.

A simple integration test asserting "claude-cli backend is registered and resolvable after default-config gateway boot" would have caught all three. None exists today.

gateway-codex-harness.live.test.ts is the parallel test for the codex harness — comprehensive but heavy (866 lines, gated on OPENCLAW_LIVE_CODEX_HARNESS=1). The codex coverage pattern works because codex regressions are rare; the asymmetry with claude-cli (3 regressions in 25 days vs zero) suggests a lighter, default-on guardrail is worth its weight specifically for claude-cli.

Proposed test (sketch, ~30 lines)

A non-live unit-level test that:

  1. Loads the bundled extensions/anthropic plugin into a minimal in-memory gateway record.
  2. Asserts that after register() completes, the agent-harness registry contains an entry whose id matches the value agents.defaults.embeddedHarness.runtime was historically set to (i.e. claude-cli for configs carried over from 4.23, or whatever the new canonical id is).
  3. Asserts that cli-migration.js either migrates embeddedHarness.runtime: "claude-cli" to the new value OR the old id remains a registered alias — choice belongs to the upstream design, not this test.

Pseudocode:

import { describe, expect, it } from "vitest";
import { loadBundledPluginForTest } from "<existing test helper>";
import { listAgentHarnesses } from "<existing harness registry>";

describe("anthropic extension: claude-cli backend post-load contract", () => {
  it("registers an agent harness resolvable as 'claude-cli' (or its canonical replacement)", async () => {
    const record = await loadBundledPluginForTest("anthropic");
    const ids = listAgentHarnesses().map((h) => h.id);
    // Either the legacy id is still resolvable, OR a documented migration exists.
    expect(
      ids.includes("claude-cli") || record.config.migrations?.runtime?.from === "claude-cli",
    ).toBe(true);
  });

  it("default-config carry-over from a prior version still resolves to a valid harness", async () => {
    const cfg = { agents: { defaults: { embeddedHarness: { runtime: "claude-cli" } } } };
    const result = await resolveHarnessForConfig(cfg);
    expect(result).toBeDefined();
    expect(result.id).toBeTruthy();
  });
});

Why now

Each of the three regressions surfaced as a runtime error the plugin author / operator has to diagnose by reading loader-*.js source. Adding this guardrail to upstream CI would:

  • Fail the next PR that breaks the registration before it merges.
  • Document the contract: "the claude-cli backend is part of the bundled anthropic extension's public surface".
  • Save downstream operators (us, and others affected by the same 4.24 regression) the upgrade-day hour.

Offer to contribute

Happy to send the PR if you confirm:

  1. The test should live at extensions/anthropic/cli-harness-registration.test.ts (or wherever you prefer).
  2. The exact harness-id contract you want preserved post-claude-cli/<model> deprecation. Per the diff insight on #72434, the migration moves model refs to anthropic/<model> with a CLI runtime modifier — but the runtime id (used by embeddedHarness.runtime) appears not to be part of cli-migration.js. If there's a canonical new id (anthropic-cli? claude-cli retained as alias? something else), pin it; otherwise the test should assert "either the legacy id resolves or a migration is registered" so it stays green across either choice.

OpenClaw version

2026.4.24 (cbcfdf6). Verified against installed /opt/homebrew/lib/node_modules/openclaw/dist/extensions/anthropic/ — only extensions/codex/index.js calls registerAgentHarness in the stock bundle.

extent analysis

TL;DR

Add a unit test to assert the claude-cli backend is registered and resolvable after default-config gateway boot to prevent similar regressions.

Guidance

  • Create a new test file extensions/anthropic/cli-harness-registration.test.ts to contain the proposed test.
  • Implement the test using the provided pseudocode as a starting point, ensuring to cover both the legacy claude-cli id and any potential migration to a new id.
  • Verify the test fails when the claude-cli backend is not registered or resolvable, and passes when it is.
  • Consider adding additional test cases to cover different configuration scenarios and edge cases.

Example

The provided pseudocode can be used as a starting point for the test implementation:

describe("anthropic extension: claude-cli backend post-load contract", () => {
  it("registers an agent harness resolvable as 'claude-cli' (or its canonical replacement)", async () => {
    const record = await loadBundledPluginForTest("anthropic");
    const ids = listAgentHarnesses().map((h) => h.id);
    // Either the legacy id is still resolvable, OR a documented migration exists.
    expect(
      ids.includes("claude-cli") || record.config.migrations?.runtime?.from === "claude-cli",
    ).toBe(true);
  });
});

Notes

The test implementation should be reviewed and refined to ensure it accurately covers the required scenarios and does not introduce any false positives or negatives.

Recommendation

Apply the proposed test to the openclaw codebase to prevent similar regressions and ensure the claude-cli backend is properly registered and resolvable. This will help catch any future issues before they are merged and affect downstream operators.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#installation#tensor shape#autograd error#runtime error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

openclaw - ✅(Solved) Fix [Test gap]: claude-cli backend has no integration test catching the registration regression class [2 pull requests, 2 comments, 2 participants]