hermes - ✅(Solved) Fix delegate_task: subagent side-effect verification — structural mechanism beyond the tool-description warning [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#16357Fetched 2026-04-28 06:53:50
View on GitHub
Comments
0
Participants
1
Timeline
8
Reactions
0
Author
Participants
Timeline (top)
labeled ×4cross-referenced ×2mentioned ×1subscribed ×1

Fix Action

Fix / Workaround

PR #16325 shipped the cheap mitigation — a new bullet in the delegate_task tool description warning the calling model that subagent summaries are self-reports, and asking it to verify external-side-effect claims (HTTP uploads, remote writes, file creation at shared paths) by requiring a verifiable handle (URL, ID, path) from the subagent and fetching/stat'ing it before claiming success to the user.

  • Tracking: #16328 (the original docx triage — this is the one deferred item)
  • Mitigation shipped: #16325 (delegate_task description warning)
  • Code: tools/delegate_tool.pyDELEGATE_TASK_SCHEMA description and the per-task subagent result synthesis.

Priority: P3 (the mitigation covers the common case; structural fix is a design question, not a bug).

PR fix notes

PR #16789: feat(delegate): add side-effect manifest verification

Description (problem / solution / changelog)

What does this PR do?

Adds an opt-in structural side-effect verification path for delegated subagents. Instead of trusting a child agent’s final summary alone, delegate_task can now grant a record_side_effect tool, collect a typed manifest, and verify recorded file/URL side effects before reporting the task as fully completed.

The implementation downgrades unverifiable claims to completed_unverified, verifies expected side effects when provided, blocks unsafe URL redirects during verification, requires concrete evidence for remote mutation checks, and bounds recorded manifest fields to keep parent context controlled.

Related Issue

Fixes #16357

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/side_effects.py: adds the side-effect manifest registry, record_side_effect tool, file/URL verification, redirect safety checks, remote mutation evidence requirements, and manifest field limits.
  • tools/delegate_tool.py: adds verify_side_effects and expected_side_effects support, grants the side-effect recorder only when requested, and returns verification metadata.
  • run_agent.py: forwards verification fields through delegate dispatch.
  • toolsets.py: registers the new side_effects toolset.
  • tests/tools/test_delegate.py: adds regression coverage for missing manifests, verified file writes, URL failures, unsafe redirects, remote mutation evidence, expected side effects, schema support, and batch isolation.

How to Test

  1. .venv/bin/python -m pytest -o "addopts=" -n 0 --ignore=tests/integration --ignore=tests/e2e -m "not integration" tests/tools/test_delegate.py::TestDelegateSideEffectVerification tests/tools/test_delegate.py::TestDelegateRequirements::test_schema_valid
  2. scripts/run_tests.sh tests/tools/test_delegate.py tests/tools/test_file_state_registry.py
  3. .venv/bin/python -m ruff check tools/side_effects.py
  4. .venv/bin/python -m ruff format --check tools/side_effects.py
  5. .venv/bin/python -m compileall -q tools/side_effects.py tools/delegate_tool.py run_agent.py toolsets.py tests/tools/test_delegate.py
  6. git diff --check

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux container, Python 3.11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Targeted verification passed:

  • 15 passed for side-effect verification/schema tests
  • 145 passed for tests/tools/test_delegate.py and tests/tools/test_file_state_registry.py

Changed files

  • run_agent.py (modified, +2/-0)
  • tests/tools/test_delegate.py (modified, +719/-0)
  • tools/delegate_tool.py (modified, +171/-4)
  • tools/side_effects.py (added, +562/-0)
  • toolsets.py (modified, +6/-0)
RAW_BUFFERClick to expand / collapse

Split out from #16328 (@versun docx triage). The docx flagged a real failure mode:

User used the sharethis-chat skill via delegation. Subagent reported success; page was empty except "test". User asked "are you sure?", Hermes rechecked, found only "test" had been uploaded.

PR #16325 shipped the cheap mitigation — a new bullet in the delegate_task tool description warning the calling model that subagent summaries are self-reports, and asking it to verify external-side-effect claims (HTTP uploads, remote writes, file creation at shared paths) by requiring a verifiable handle (URL, ID, path) from the subagent and fetching/stat'ing it before claiming success to the user.

That's a prompt-level nudge. It relies on the model reading and following the guidance. This issue tracks the structural fix.

The underlying problem

tools/delegate_tool.py returns a summary string to the parent. By design, the parent has zero visibility into intermediate tool results in the subagent's context — that's the whole point of context-isolated delegation. But it means:

  1. A subagent can claim "uploaded" when the last HTTP response was a 400.
  2. A subagent can claim "wrote file X" when it actually wrote to /tmp/… and the shared path still has the old content.
  3. A subagent can claim "the deploy succeeded" when the deploy script exited 0 on a no-op.

Each individual tool (terminal, write_file, send_message, etc.) has its own success/failure semantics the subagent sees. None of that bubbles up in the summary. The parent only gets the subagent's narrative reconstruction of what happened.

What a structural fix would look like

Not obvious, and that's why we deferred it. Some possible shapes:

  • Side-effect manifest. Subagent records a typed list of side-effects it claims to have performed ({"kind": "http_post", "url": "…", "status": 200}, {"kind": "file_write", "path": "…", "bytes": 14823}). Parent gets the manifest alongside the summary and can verify each entry with a corresponding tool call. Needs: a recording mechanism, a taxonomy of side-effect kinds, a verification library per kind, and opt-in per-call (not every delegation has external effects worth verifying).

  • Capability-scoped subagent grants. Parent declares what external-effect capabilities the subagent is granted (network.post, fs.write_outside_cwd) and the manifest records what was used. Lets the parent decide whether to verify.

  • Tool-level result propagation. Specific tools (HTTP clients, upload helpers) return a "verifiable handle" as part of their normal result; delegate_task surfaces those handles in a structured field on the summary object.

  • Stay with the warning. Accept the structural limitation and lean harder on the tool-description warning + skill-level verification (sharethis-chat skill does a GET <url> and checks content length before returning).

Non-goals

  • Preventing a malicious subagent from lying. This is about honest mistakes, not adversarial cases. Parent-side verification raises the bar but doesn't eliminate the class.
  • Verifying every subagent call. Most delegations (research, code review, debugging) have no external side-effects worth verifying. The mechanism should be opt-in.
  • Breaking the context-isolation invariant. The reason delegate_task exists is to keep intermediate noise out of the parent's context. Any mechanism here needs to avoid leaking that.

Related

  • Tracking: #16328 (the original docx triage — this is the one deferred item)
  • Mitigation shipped: #16325 (delegate_task description warning)
  • Code: tools/delegate_tool.pyDELEGATE_TASK_SCHEMA description and the per-task subagent result synthesis.

Priority: P3 (the mitigation covers the common case; structural fix is a design question, not a bug).

extent analysis

TL;DR

Implement a side-effect manifest or capability-scoped subagent grants to enable parent-side verification of subagent claims.

Guidance

  • Consider implementing a side-effect manifest that records a typed list of side-effects performed by the subagent, allowing the parent to verify each entry with a corresponding tool call.
  • Explore capability-scoped subagent grants, where the parent declares external-effect capabilities granted to the subagent, and the manifest records what was used.
  • Evaluate tool-level result propagation, where specific tools return a "verifiable handle" as part of their normal result, and delegate_task surfaces those handles in a structured field on the summary object.
  • Assess the feasibility of each approach, considering the trade-offs between complexity, security, and usability.

Example

No code snippet is provided, as the issue requires a design-level solution rather than a specific code change.

Notes

The chosen solution should balance the need for verification with the context-isolation invariant and opt-in requirements. It's essential to consider the limitations and potential drawbacks of each approach.

Recommendation

Apply a side-effect manifest or capability-scoped subagent grants, as these approaches seem to offer a good balance between verification and context isolation. This recommendation is based on the provided information and may need to be adjusted as the design evolves.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING