Issue Station
Sign In
Tools hereToolbox

hermes - ✅(Solved) Fix Shared CI test baseline failures affect unrelated PRs [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#23056•Fetched 2026-05-11 03:31:33
View on GitHub
Comments
0
Participants
1
Timeline
4
Reactions
0
Author
LeonSGP43LeonSGP43
Participants
LeonSGP43LeonSGP43
Timeline (top)
labeled ×3cross-referenced ×1

While preparing several narrow fixes from the latest origin/main, I noticed that unrelated PRs were repeatedly failing the same shared CI lanes. The failures were not tied to the individual feature/fix patches; they came from repository-level CI/test baseline assumptions that made otherwise scoped PRs look red.

Affected examples include #22911, #22917, #22933, #22943, #22945, #22954, and #22965. These PRs touched different areas, but inherited overlapping failures in shared acceptance checks.

Root Cause

When shared CI lanes are already unstable, reviewers cannot reliably distinguish a real regression in a small PR from inherited baseline failures. This lowers merge confidence for otherwise narrow fixes and creates repeated red PRs even when the PR-specific code path is sound.

Fix Action

Fix / Workaround

While preparing several narrow fixes from the latest origin/main, I noticed that unrelated PRs were repeatedly failing the same shared CI lanes. The failures were not tied to the individual feature/fix patches; they came from repository-level CI/test baseline assumptions that made otherwise scoped PRs look red.

PR fix notes

PR #22984: fix(ci): restore shared acceptance lanes

Description (problem / solution / changelog)

Addresses #23056.

Summary

This PR restores the shared CI/test baseline that has been making unrelated fork PRs appear red.

  • Replace the os.killpg(..., signal.SIGKILL) cleanup path in tools/process_registry.py with a Windows-safe hard-kill helper.
  • Make the lint-diff PR comment step advisory when fork-token permissions cannot comment (403 Resource not accessible by integration).
  • Disable destructive slash confirmation in the e2e harness so /new command assertions keep testing the immediate-reset path.
  • Refresh stale shared lane fixtures around TTS media routing, gateway restart helpers, and async HTTP cleanup.
  • Reset worker-local i18n/config caches between tests so one test cannot poison unrelated gateway/tool tests.
  • Make the native vision fast-path test explicit about its config, so it is not affected by previous auxiliary vision settings.

Verification

Focused local checks run on the current branch:

uv run --frozen python -m pytest tests/gateway/test_restart_drain.py::test_restart_command_while_busy_requests_drain_without_interrupt tests/tools/test_vision_native_fast_path.py::TestHandleVisionAnalyzeFastPath::test_vision_capable_main_model_uses_fast_path -q --tb=short -n auto
# 2 passed

uv run --frozen python -m pytest tests/agent/test_i18n.py tests/gateway/test_restart_drain.py tests/tools/test_vision_native_fast_path.py -q --tb=short -n auto
# 61 passed

uv run --frozen ruff check tests/conftest.py tests/tools/test_vision_native_fast_path.py
# All checks passed

Earlier targeted checks for the first repair commit also passed:

uv run --frozen python -m pytest \
  tests/ci/test_lint_diff_workflow.py \
  tests/e2e/test_agent_workflow.py::test_agent_workflow_new_command_resets_session \
  tests/e2e/test_platform_commands.py::test_platform_commands_new_resets_session \
  tests/tools/test_process_registry.py \
  -q --tb=short

Attribution

The failures addressed here were shared CI/test-lane issues observed across unrelated PRs, not regressions in those individual PR bodies. See #23056 for the issue-level summary and affected examples.

Changed files

  • .github/workflows/lint.yml (modified, +27/-17)
  • tests/conftest.py (modified, +17/-0)
  • tests/e2e/conftest.py (modified, +6/-2)
  • tests/gateway/test_tts_media_routing.py (modified, +24/-6)
  • tests/hermes_cli/test_update_gateway_restart.py (modified, +18/-0)
  • tests/run_agent/test_async_httpx_del_neuter.py (modified, +2/-1)
  • tests/test_lint_config.py (modified, +7/-0)
  • tests/tools/test_vision_native_fast_path.py (modified, +11/-0)
  • tools/process_registry.py (modified, +21/-7)
RAW_BUFFERClick to expand / collapse

Summary

While preparing several narrow fixes from the latest origin/main, I noticed that unrelated PRs were repeatedly failing the same shared CI lanes. The failures were not tied to the individual feature/fix patches; they came from repository-level CI/test baseline assumptions that made otherwise scoped PRs look red.

Affected examples include #22911, #22917, #22933, #22943, #22945, #22954, and #22965. These PRs touched different areas, but inherited overlapping failures in shared acceptance checks.

Observed failure cluster

The repeated failures included:

  • Lint (ruff + ty) / ruff + ty diff: fork PR comment update path failed with 403 Resource not accessible by integration.
  • Windows footguns (blocking): flagged bare process-group kill patterns that were not safe on Windows.
  • Tests / e2e: /new command assertions were stale after destructive slash confirmation became the default path.
  • Tests / test: after the first shared-lane fixes, the remaining red checks were still shared test-baseline issues rather than regressions from the unrelated PR bodies.

Current repair PR

I opened #22984 as the shared repair PR for this cluster.

At the current head, these lanes are already green on #22984:

  • ruff + ty diff
  • ruff enforcement (blocking)
  • Windows footguns (blocking)
  • e2e
  • Docker, Nix, attribution, and supply-chain checks

The latest remaining Tests / test failures on #22984 are:

  • tests/gateway/test_restart_drain.py::test_restart_command_while_busy_requests_drain_without_interrupt
  • tests/tools/test_vision_native_fast_path.py::TestHandleVisionAnalyzeFastPath::test_vision_capable_main_model_uses_fast_path

Both point to test isolation/config-cache assumptions, not to the business logic of the earlier unrelated fix PRs.

Why this matters

When shared CI lanes are already unstable, reviewers cannot reliably distinguish a real regression in a small PR from inherited baseline failures. This lowers merge confidence for otherwise narrow fixes and creates repeated red PRs even when the PR-specific code path is sound.

Proposed path

Use this issue to track restoration of the shared CI/test baseline. I am continuing #22984 with focused fixes for the remaining shared failures. If maintainers prefer a different shape, I can split the remaining Tests / test cleanup into smaller PRs.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#tool integration#LLM response#prompt template#agent execution#callback error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - ✅(Solved) Fix Shared CI test baseline failures affect unrelated PRs [1 pull requests, 1 participants]