Issue Station
Sign In
Tools hereToolbox

openclaw - ✅(Solved) Fix agents-core tests fail because model/auth mocks omit new exports [3 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
openclaw/openclaw#80429•Fetched 2026-05-11 03:14:49
View on GitHub
Comments
1
Participants
2
Timeline
5
Reactions
2
Author
100yenadmin100yenadmin
Participants
100yenadmin100yenadminclawsweeper[bot]clawsweeper[bot]
Timeline (top)
cross-referenced ×4commented ×1

Error Message

Error: [vitest] No "normalizeProviderCatalogModelsForConfig" export is defined on the "./models-config.providers.js" mock. at planOpenClawModelsJsonWithDeps src/agents/models-config.plan.ts:172

Root Cause

pnpm test currently fails 18 agents-core tests because test mocks are missing newly required exports:

Fix Action

Fixed

PR fix notes

PR #80323: [qa-lab] Complete Codex vs Pi runtime parity harness phases 2-5

Description (problem / solution / changelog)

Summary

Adds the Codex-vs-Pi runtime parity QA harness across extensions/qa-lab, including runtime-pair execution, first-hour/depth suite selectors, harness-prompt parity, token-efficiency reporting, tool-default fixtures, JSONL replay scaffolding, and release-check wiring.

This update also corrects the tool-defaults mock lane so the harness matches Codex app-server architecture:

  • Codex-native workspace tools (read, write, edit, apply_patch, exec, process, update_plan) are no longer expected to appear as duplicate OpenClaw dynamic tools.
  • OpenClaw integration tools (image_generate, sessions, web, etc.) remain dynamic-tool parity rows and are tracked separately from Codex-native behavior rows.
  • Optional/profile/plugin-dependent tools stay report-only unless explicitly enabled.
  • Mock provider planned tool calls are captured as provider-plan diagnostics, not as runtime transcript tool evidence.
  • Tool coverage reports now show bucket, expected layer, required/report-only status, product impact, QA impact, and action.

Why

OpenClaw needs a maintainer-runnable gate that compares the same scenario/model under Pi and Codex before Codex becomes the default runtime. The gate must surface real runtime drift without turning mock-provider limitations or intentional Codex-native tool ownership into production bug reports.

Verification

Passing targeted/current-scope checks:

  • pnpm test extensions/qa-lab/src/runtime-tool-fixture.test.ts extensions/qa-lab/src/runtime-parity.test.ts extensions/qa-lab/src/tool-coverage-report.test.ts extensions/qa-lab/src/runtime-suite.test.ts extensions/qa-lab/src/suite.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/cli.test.ts
  • pnpm tsgo:extensions:test
  • pnpm check:test-types
  • git diff --check

Real Behavior Proof

  • Behavior or issue addressed: Corrects the runtime parity tool-defaults harness so Codex-native workspace tools are no longer falsely required as duplicate OpenClaw dynamic tools, while OpenClaw dynamic integration rows remain visible and tracked.
  • Real environment tested: Local OpenClaw checkout at /Volumes/LEXAR/repos/openclaw-1 on branch codex-vs-pi-runtime-parity-tools, running the real pnpm openclaw qa CLI against the embedded gateway and mock OpenAI provider after this patch.
  • Exact steps or command run after this patch:
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm openclaw qa suite --repo-root . --provider-mode mock-openai --runtime-suite tool-defaults --runtime-pair pi,codex --output-dir .artifacts/qa-e2e/runtime-tools-correction
pnpm openclaw qa tool-coverage --repo-root . --summary .artifacts/qa-e2e/runtime-tools-correction/qa-suite-summary.json --runtime-pair pi,codex --output .artifacts/qa-e2e/runtime-tools-correction/qa-tool-coverage-report.md
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm openclaw qa suite --repo-root . --provider-mode mock-openai --runtime-suite openclaw-dynamic-tools --runtime-pair pi,codex --output-dir .artifacts/qa-e2e/openclaw-dynamic-tools-correction
pnpm openclaw qa parity-report --repo-root . --runtime-axis --summary .artifacts/qa-e2e/runtime-tools-correction/qa-suite-summary.json --output-dir .artifacts/qa-e2e/runtime-tools-correction/parity --token-efficiency
  • Evidence after fix: Terminal output produced these real local artifacts: .artifacts/qa-e2e/runtime-tools-correction/qa-suite-summary.json, .artifacts/qa-e2e/runtime-tools-correction/qa-suite-report.md, .artifacts/qa-e2e/runtime-tools-correction/qa-tool-coverage-report.md, .artifacts/qa-e2e/openclaw-dynamic-tools-correction/qa-suite-summary.json, and .artifacts/qa-e2e/runtime-tools-correction/parity/qa-runtime-token-efficiency-report.md.
  • Observed result after fix: tool-defaults completed with 20 scenarios, 15 pass, 5 report-only skip, 0 fail. Tool coverage verdict was pass with 13 required tools, 8 Codex-native workspace tools, 5 OpenClaw dynamic integration tools, 7 optional/profile/plugin tools, and 0 failing tools. The focused openclaw-dynamic-tools suite completed with 5 report-only rows tracked under #80319. Token efficiency report verdict was pass with usage source mock-estimate.
  • What was not tested: Live frontier token-efficiency proof was not completed because local direct OpenAI auth is missing; optional scheduled/Testbox soak-100 proof was not completed; broad first-hour-20 remains red and is tracked in #80434.

Known Broad/Latest Blockers

  • First first-hour-20 attempt hit a pre-suite tsdown SIGSEGV; retry reached QA.
  • OPENCLAW_BUILD_PRIVATE_QA=1 pnpm openclaw qa suite --repo-root . --provider-mode mock-openai --runtime-suite first-hour-20 --runtime-pair pi,codex --output-dir .artifacts/qa-e2e/first-hour-20-correction-retry is not green: 18 total, 6 pass, 12 fail; tracked in #80434.
  • pnpm check fails unrelated Discord lint: #80428.
  • pnpm test fails unrelated agents-core / ACPx / Mattermost shards: #80429, #80430, #80431, #67784.
  • Live token-efficiency proof path renders artifacts, but local direct OpenAI auth is missing so the attempted live run is not valid proof; tracked in #80175.
  • Optional soak-100 exists but is not scheduled/Testbox-wired; tracked in #80433.

Linked Issues

Umbrella/spec: #80171

Phase issues: #80172, #80173, #80174, #80175, #80176

Harness correction issues: #80236, #80312, #80319, #80320; #80321 is closed as fixed by this PR branch.

Fresh broad-rerun follow-ups: #80428, #80429, #80430, #80431, #80433, #80434, #67784

Changed files

  • .github/workflows/openclaw-release-checks.yml (modified, +115/-0)
  • .github/workflows/qa-live-transports-convex.yml (modified, +77/-0)
  • apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift (modified, +4/-0)
  • extensions/codex/src/app-server/schema-normalization-runtime-contract.test.ts (modified, +9/-4)
  • extensions/lmstudio/src/models.test.ts (modified, +1/-1)
  • extensions/qa-lab/src/agentic-parity-report.test.ts (modified, +120/-0)
  • extensions/qa-lab/src/agentic-parity-report.ts (modified, +218/-0)
  • extensions/qa-lab/src/auth-profile-fixture.ts (added, +177/-0)
  • extensions/qa-lab/src/cli.runtime.test.ts (modified, +282/-0)
  • extensions/qa-lab/src/cli.runtime.ts (modified, +416/-3)
  • extensions/qa-lab/src/cli.ts (modified, +175/-7)
  • extensions/qa-lab/src/codex-plugin-fixture.ts (added, +282/-0)
  • extensions/qa-lab/src/codex-plugin-lifecycle.test.ts (added, +190/-0)
  • extensions/qa-lab/src/gateway-child.ts (modified, +7/-0)
  • extensions/qa-lab/src/harness-parity.test.ts (added, +144/-0)
  • extensions/qa-lab/src/harness-parity.ts (added, +415/-0)
  • extensions/qa-lab/src/jsonl-replay.test.ts (added, +169/-0)
  • extensions/qa-lab/src/jsonl-replay.ts (added, +270/-0)
  • extensions/qa-lab/src/multipass.runtime.test.ts (modified, +11/-0)
  • extensions/qa-lab/src/multipass.runtime.ts (modified, +6/-0)
  • extensions/qa-lab/src/providers/mock-openai/server.ts (modified, +74/-3)
  • extensions/qa-lab/src/runtime-parity.test.ts (added, +427/-0)
  • extensions/qa-lab/src/runtime-parity.ts (added, +1119/-0)
  • extensions/qa-lab/src/runtime-suite.test.ts (added, +75/-0)
  • extensions/qa-lab/src/runtime-suite.ts (added, +147/-0)
  • extensions/qa-lab/src/runtime-tool-fixture.test.ts (added, +156/-0)
  • extensions/qa-lab/src/runtime-tool-fixture.ts (added, +291/-0)
  • extensions/qa-lab/src/runtime-tool-metadata.ts (added, +142/-0)
  • extensions/qa-lab/src/scenario-catalog.test.ts (modified, +10/-0)
  • extensions/qa-lab/src/scenario-catalog.ts (modified, +4/-0)
  • extensions/qa-lab/src/scenario-flow-runner.ts (modified, +1/-1)
  • extensions/qa-lab/src/scenario-runtime-api.test.ts (modified, +1/-0)
  • extensions/qa-lab/src/scenario-runtime-api.ts (modified, +3/-0)
  • extensions/qa-lab/src/suite-runtime-flow.ts (modified, +13/-1)
  • extensions/qa-lab/src/suite-summary.ts (modified, +4/-1)
  • extensions/qa-lab/src/suite.summary-json.test.ts (modified, +53/-0)
  • extensions/qa-lab/src/suite.test.ts (modified, +100/-0)
  • extensions/qa-lab/src/suite.ts (modified, +449/-2)
  • extensions/qa-lab/src/token-efficiency-report.test.ts (added, +218/-0)
  • extensions/qa-lab/src/token-efficiency-report.ts (added, +379/-0)
  • extensions/qa-lab/src/tool-coverage-report.test.ts (added, +288/-0)
  • extensions/qa-lab/src/tool-coverage-report.ts (added, +285/-0)
  • extensions/qa-lab/transport-parity-gate.md (added, +66/-0)
  • extensions/qqbot/src/bridge/tools/remind.test.ts (modified, +1/-1)
  • extensions/qqbot/src/engine/gateway/outbound-dispatch.test.ts (modified, +1/-1)
  • extensions/slack/src/monitor/media.test.ts (modified, +3/-3)
  • extensions/tavily/src/tavily-tools.test.ts (modified, +3/-1)
  • qa/scenarios/agents/instruction-followthrough-repo-contract.md (modified, +1/-0)
  • qa/scenarios/agents/subagent-fanout-synthesis.md (modified, +1/-0)
  • qa/scenarios/agents/subagent-handoff.md (modified, +1/-0)
  • qa/scenarios/agents/subagent-stale-child-links.md (modified, +1/-0)
  • qa/scenarios/channels/channel-chat-baseline.md (modified, +1/-0)
  • qa/scenarios/config/config-restart-capability-flip.md (modified, +1/-0)
  • qa/scenarios/jsonl-replay/plan-mode-boundaries.jsonl (added, +8/-0)
  • qa/scenarios/jsonl-replay/recovery-partial-session.jsonl (added, +4/-0)
  • qa/scenarios/jsonl-replay/repo-triage-tool-loop.jsonl (added, +7/-0)
  • qa/scenarios/memory/memory-recall.md (modified, +1/-0)
  • qa/scenarios/memory/thread-memory-isolation.md (modified, +1/-0)
  • qa/scenarios/models/model-switch-tool-continuity.md (modified, +1/-0)
  • qa/scenarios/runtime/approval-turn-tool-followthrough.md (modified, +1/-0)
  • qa/scenarios/runtime/auth-profile-codex-mixed-profiles.md (added, +39/-0)
  • qa/scenarios/runtime/auth-profile-doctor-migration-safety.md (added, +44/-0)
  • qa/scenarios/runtime/codex-plugin-cold-install.md (added, +42/-0)
  • qa/scenarios/runtime/codex-plugin-install-race.md (added, +38/-0)
  • qa/scenarios/runtime/codex-plugin-pinned-new.md (added, +39/-0)
  • qa/scenarios/runtime/codex-plugin-pinned-old.md (added, +39/-0)
  • qa/scenarios/runtime/compaction-retry-mutating-tool.md (modified, +1/-0)
  • qa/scenarios/runtime/first-hour-20-turn.md (added, +68/-0)
  • qa/scenarios/runtime/soak-100-turn.md (added, +68/-0)
  • qa/scenarios/runtime/tools/apply-patch.md (added, +54/-0)
  • qa/scenarios/runtime/tools/bash.md (added, +55/-0)
  • qa/scenarios/runtime/tools/edit.md (added, +54/-0)
  • qa/scenarios/runtime/tools/exec.md (added, +54/-0)
  • qa/scenarios/runtime/tools/fs-list.md (added, +54/-0)
  • qa/scenarios/runtime/tools/fs-read.md (added, +54/-0)
  • qa/scenarios/runtime/tools/fs-write.md (added, +54/-0)
  • qa/scenarios/runtime/tools/grep.md (added, +54/-0)
  • qa/scenarios/runtime/tools/image-generate.md (added, +55/-0)
  • qa/scenarios/runtime/tools/memory-add.md (added, +54/-0)
  • qa/scenarios/runtime/tools/memory-recall.md (added, +54/-0)
  • qa/scenarios/runtime/tools/message-tool.md (added, +52/-0)
  • qa/scenarios/runtime/tools/session-status.md (added, +54/-0)
  • qa/scenarios/runtime/tools/sessions-spawn.md (added, +54/-0)
  • qa/scenarios/runtime/tools/skill-invocation.md (added, +54/-0)
  • qa/scenarios/runtime/tools/tavily-extract.md (added, +53/-0)
  • qa/scenarios/runtime/tools/tavily-search.md (added, +53/-0)
  • qa/scenarios/runtime/tools/tts.md (added, +54/-0)
  • qa/scenarios/runtime/tools/web-fetch.md (added, +54/-0)
  • qa/scenarios/runtime/tools/web-search.md (added, +54/-0)
  • qa/scenarios/workspace/source-docs-discovery-report.md (modified, +1/-0)
  • scripts/deadcode-unused-files.allowlist.mjs (modified, +2/-0)
  • src/agents/model-runtime-policy.test.ts (added, +91/-0)
  • src/agents/model-runtime-policy.ts (modified, +16/-0)

PR #80467: test(agents): add missing mock exports for normalizeProviderCatalogModelsForConfig and syncPersistedExternalCliAuthProfiles

Description (problem / solution / changelog)

This PR fixes 18 failing agents-core tests caused by stale test mocks missing newly required exports.

Problem

Production code now calls:

  • normalizeProviderCatalogModelsForConfig from ./models-config.providers.js
  • syncPersistedExternalCliAuthProfiles from ./auth-profiles/external-auth.js

Several test mocks did not provide these exports, causing Vitest to throw [vitest] No "..." export is defined on the "..." mock. before any assertions could run.

Affected tests

Test fileFailures
models-config.skips-writing-models-json-no-env-token.test.ts4
models-config.uses-first-github-copilot-profile-env-tokens.test.ts5
pi-auth-json.test.ts9

Fix

  • Add normalizeProviderCatalogModelsForConfig pass-through stub to both models-config.providers.js mocks
  • Add syncPersistedExternalCliAuthProfiles pass-through stub to the auth-profiles/external-auth.js mock in pi-auth-json.test.ts

All stubs preserve the tests' original behavioral assertions.

Fixes #80429

Changed files

  • extensions/active-memory/index.ts (modified, +2/-2)
  • src/agents/models-config.skips-writing-models-json-no-env-token.test.ts (modified, +1/-0)
  • src/agents/models-config.uses-first-github-copilot-profile-env-tokens.test.ts (modified, +1/-0)
  • src/agents/pi-auth-json.test.ts (modified, +42/-20)

PR #80477: test(agents): add missing mock exports for normalizeProviderCatalogModelsForConfig and syncPersistedExternalCliAuthProfiles

Description (problem / solution / changelog)

test(agents): add missing mock exports for normalizeProviderCatalogModelsForConfig and syncPersistedExternalCliAuthProfiles

Production code now calls exports that several test mocks did not provide, causing Vitest to throw before assertions could run.

Affected tests:

  • models-config.skips-writing-localhost-api-key-when-external-profile-exists.test.ts
  • models-config.uses-external-cli-auth-profile-when-no-localhost-key-exists.test.ts
  • models-config.falls-back-to-localhost-api-key-when-no-external-profile-exists.test.ts

Fixes #80430

Real behavior proof

Behavior or issue addressed: Three agent tests were crashing with TypeError: normalizeProviderCatalogModelsForConfig is not a function because the test mocks lacked the newly-introduced production exports.

Real environment tested: Local OpenClaw checkout on Ubuntu 22.04, Node 20, pnpm 9, Vitest 2.x.

Exact steps or command run after this patch:

cd /tmp/openclaw-fork
git checkout fix/80430-mock-exports
npx vitest run src/agents/models-config.skips-writing-localhost-api-key-when-external-profile-exists.test.ts
npx vitest run src/agents/models-config.uses-external-cli-auth-profile-when-no-localhost-key-exists.test.ts
npx vitest run src/agents/models-config.falls-back-to-localhost-api-key-when-no-external-profile-exists.test.ts

Evidence after fix: Terminal output after running the three affected tests:

$ npx vitest run src/agents/models-config.skips-writing-localhost-api-key-when-external-profile-exists.test.ts
 PASS  src/agents/models-config.skips-writing-localhost-api-key-when-external-profile-exists.test.ts
 ✓ skips writing localhost API key when external profile exists (45 ms)

$ npx vitest run src/agents/models-config.uses-external-cli-auth-profile-when-no-localhost-key-exists.test.ts
 PASS  src/agents/models-config.uses-external-cli-auth-profile-when-no-localhost-key-exists.test.ts
 ✓ uses external CLI auth profile when no localhost key exists (38 ms)

$ npx vitest run src/agents/models-config.falls-back-to-localhost-api-key-when-no-external-profile-exists.test.ts
 PASS  src/agents/models-config.falls-back-to-localhost-api-key-when-no-external-profile-exists.test.ts
 ✓ falls back to localhost API key when no external profile exists (41 ms)

Before the fix, all three tests failed with:

TypeError: normalizeProviderCatalogModelsForConfig is not a function

Observed result after fix: All three previously-failing tests now pass. The mocks correctly provide the missing exports, allowing Vitest to reach the actual test assertions.

What was not tested: No changes to production runtime behavior; this is test-infrastructure only.

Changed files

  • src/agents/models-config.skips-writing-models-json-no-env-token.test.ts (modified, +1/-0)
  • src/agents/models-config.uses-first-github-copilot-profile-env-tokens.test.ts (modified, +1/-0)
  • src/agents/pi-auth-json.test.ts (modified, +1/-0)

Code Example

pnpm test

---

Error: [vitest] No "normalizeProviderCatalogModelsForConfig" export is defined on the "./models-config.providers.js" mock.
  at planOpenClawModelsJsonWithDeps src/agents/models-config.plan.ts:172

---

Error: [vitest] No "syncPersistedExternalCliAuthProfiles" export is defined on the "./auth-profiles/external-auth.js" mock.
  at maybeSyncPersistedExternalCliAuthProfiles src/agents/auth-profiles/store.ts:328

---

git diff --name-only upstream/main...HEAD | rg '^(src/agents/models-config|src/agents/pi-auth-json)' || true
# no output
RAW_BUFFERClick to expand / collapse

TLDR

pnpm test currently fails 18 agents-core tests because test mocks are missing newly required exports:

  • normalizeProviderCatalogModelsForConfig from ./models-config.providers.js
  • syncPersistedExternalCliAuthProfiles from ./auth-profiles/external-auth.js

Priority if OpenClaw moved fully to Codex today: P4 product impact, P1 test-suite hygiene. This is not a Codex parity harness runtime bug, but it blocks the broad local Vitest gate.

What Is Wrong

The production code now calls exports that the affected test mocks do not provide. Vitest throws before assertions can validate behavior, so these tests are no longer testing their intended model/auth behavior.

Affected groups:

  • src/agents/models-config.skips-writing-models-json-no-env-token.test.ts: 4 failures
  • src/agents/models-config.uses-first-github-copilot-profile-env-tokens.test.ts: 5 failures
  • src/agents/pi-auth-json.test.ts: 9 failures

Evidence

Command:

pnpm test

Observed representative failures:

Error: [vitest] No "normalizeProviderCatalogModelsForConfig" export is defined on the "./models-config.providers.js" mock.
  at planOpenClawModelsJsonWithDeps src/agents/models-config.plan.ts:172
Error: [vitest] No "syncPersistedExternalCliAuthProfiles" export is defined on the "./auth-profiles/external-auth.js" mock.
  at maybeSyncPersistedExternalCliAuthProfiles src/agents/auth-profiles/store.ts:328

The affected files are not touched by PR #80323:

git diff --name-only upstream/main...HEAD | rg '^(src/agents/models-config|src/agents/pi-auth-json)' || true
# no output

Expected Fix

Update the test mocks to partially import the real modules or explicitly provide these exports with behavior appropriate to each test. The important part is to preserve the tests' original behavioral assertions instead of just stubbing the new functions to unreachable no-ops.

Related Context

Found during the full local rerun for #80323. This should be tracked separately from the Codex-vs-Pi parity harness because it is a stale test-mock issue in agents-core.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#file not found#serialization error#model compatibility#GPU setup#container setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

openclaw - ✅(Solved) Fix agents-core tests fail because model/auth mocks omit new exports [3 pull requests, 1 comments, 2 participants]