Issue Station
Sign In
Tools hereToolbox

hermes - ✅(Solved) Fix [Feature Proposal]: Post-response hook extension point for pluggable response quality enforcement [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#11719•Fetched 2026-04-18 05:59:12
View on GitHub
Comments
0
Participants
1
Timeline
8
Reactions
0
Author
zons-zhaozhyzons-zhaozhy
Participants
zons-zhaozhyzons-zhaozhy
Timeline (top)
mentioned ×2referenced ×2subscribed ×2cross-referenced ×1

Error Message

  • Fail-open: hook exception → skip, never blocks the agent

Fix Action

Fixed

PR fix notes

PR #11747: feat(agent): add post-response hook extension point for pluggable response quality enforcement

Description (problem / solution / changelog)

What does this PR do?

Adds a lightweight post-response hook extension point that lets users drop .py files into ~/.hermes/hooks/ to inspect and optionally nudge the agent's final text response. Each hook can:

  1. Inject system prompt additions (pre-response guidance)
  2. Validate the final response via a check(response, context) gate
  3. Trigger a single re-generation with a nudge message when check() returns False

This decouples domain-specific response quality enforcement (compliance checks, style gates, toxicity filters, depth analysis) from the core agent loop — users configure hooks in config.yaml and drop Python files into ~/.hermes/hooks/ without patching core files.

Related Issue

Fixes #11719

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Security fix
  • Documentation update
  • Tests (adding or improving test coverage)
  • Refactor (no behavior change)
  • New skill (bundled or hub)

Changes Made

  • agent/post_response_hooks.py (NEW, ~130 LOC) — Hook loader framework: Hook dataclass, load_hooks(), build_system_prompt_additions(), run_post_response_checks()
  • run_agent.py (~30 LOC changed): init hook loading, _build_system_prompt injection, main loop hook check + nudge re-generation (max 1 per response)
  • tests/run_agent/test_post_response_hooks.py (NEW, 19 tests) — Full coverage

Config schema (in ~/.hermes/config.yaml):

agent: post_response_hooks: - module: bottom_logic_check enabled: true - module: toxicity_filter enabled: false

User hook example (in ~/.hermes/hooks/bottom_logic_check.py):

class Hook: module_name = "bottom_logic_check" system_prompt_addition = "Always provide deep analytical reasoning." nudge_message = "Your response lacks analytical depth. Please elaborate." def check(self, response, context): return len(response) > 200

How to Test

  1. python -m pytest tests/run_agent/test_post_response_hooks.py -v — all 19 tests pass
  2. python -m pytest tests/run_agent/test_run_agent.py -v — 249/250 pass (1 pre-existing env failure unrelated to this PR)
  3. Create ~/.hermes/hooks/ dir, drop a .py hook file, add config, run a conversation

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this feature
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture — or N/A
  • I've considered cross-platform impact (Windows, macOS)
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

19 passed in 2.26s (test_post_response_hooks.py) 249 passed, 1 failed in 151.48s (test_run_agent.py — pre-existing tools.memory_tool import error)

Changed files

  • agent/post_response_hooks.py (added, +206/-0)
  • run_agent.py (modified, +50/-1)
  • tests/run_agent/test_post_response_hooks.py (added, +280/-0)

Code Example

class PIIFilterHook:
    system_prompt_addition = ""  # optional pre-response guidance
    
    def check(self, response: str, context: dict) -> HookResult:
        if contains_pii(response):
            return HookResult(passed=False, action="block", message="Response blocked: PII detected")
        return HookResult(passed=True)

---

class JSONFormatHook:
    nudge_message = "Your response must be valid JSON. Try again."
    
    def check(self, response: str, context: dict) -> HookResult:
        if not is_valid_json(response):
            return HookResult(passed=False, action="nudge", message=self.nudge_message)
        return HookResult(passed=True)

---

class AuditLogHook:
    def check(self, response: str, context: dict) -> HookResult:
        log_to_audit_trail(response, context)
        return HookResult(passed=True)  # always pass

---

@dataclass
class HookResult:
    passed: bool                           # True = let response through
    action: str = "block"                  # "block" | "nudge"
    message: str = ""                      # block: refusal text | nudge: corrective hint

class Hook:
    system_prompt_addition: str = ""       # injected into system prompt (pre-response)
    
    def check(self, response: str, context: dict) -> HookResult:
        ...

---

# After strip_think_blocks(), before _build_assistant_message():
result = run_post_response_checks(self._hooks, response, context)
if not result.passed:
    if result.action == "block":
        # Replace response with block message, deliver once, no re-generation
        response = result.message
    elif result.action == "nudge" and self._empty_content_retries < 1:
        self._empty_content_retries += 1
        # Append nudge, continue main loop for one re-generation

---

agent:
  post_response_hooks:
    - module: pii_filter
      enabled: true
    - module: content_safety
      enabled: true
    - module: json_format
      enabled: false
    - module: audit_log
      enabled: true
RAW_BUFFERClick to expand / collapse

The Gap

Hermes has 11+ callbacks (tool_start_callback, thinking_callback, stream_delta_callback, etc.), but they are all fire-and-forget — none can intercept the completed response before delivery.

This means there is no extension point between "LLM finishes generating" and "response reaches the user." Any logic that inspects the final response — safety gates, compliance checks, format validation, audit logging — must be hard-coded into prompt_builder.py or run_agent.py.

Every pip install --upgrade wipes out these customizations.

Why This Matters More Now

Reasoning-capable models (DeepSeek R1, Claude extended thinking, o1/o3) change the calculus for response enforcement:

  • You cannot nudge a reasoning model into deeper thinking. Its reasoning depth is a function of model capability, not prompt engineering. Asking it to "think harder" via re-generation produces different words, not deeper thought.
  • What you CAN do is gate the output. Check for safety violations, format compliance, information leaks — and either block or let through.

This shifts the primary value from "re-generate to improve quality" to "inspect and gate what gets delivered."

Proposed: Post-Response Hook Framework

A declarative extension point where users drop .py files in ~/.hermes/hooks/ and the framework runs them after each response. Hooks support two modes:

Mode 1: Block (primary value)

Inspect the response. If it violates a rule, block delivery and return a refusal message instead. No re-generation, no wasted tokens.

class PIIFilterHook:
    system_prompt_addition = ""  # optional pre-response guidance
    
    def check(self, response: str, context: dict) -> HookResult:
        if contains_pii(response):
            return HookResult(passed=False, action="block", message="Response blocked: PII detected")
        return HookResult(passed=True)

Use cases:

  • PII / secret leak detection — scan for API keys, passwords, ID numbers
  • Content safety — block harmful, discriminatory, or violent outputs
  • Compliance gating — ensure responses meet regulatory requirements

Mode 2: Nudge (for cases where re-generation helps)

For non-reasoning models or format issues, trigger one re-generation with a corrective hint.

class JSONFormatHook:
    nudge_message = "Your response must be valid JSON. Try again."
    
    def check(self, response: str, context: dict) -> HookResult:
        if not is_valid_json(response):
            return HookResult(passed=False, action="nudge", message=self.nudge_message)
        return HookResult(passed=True)

Use cases:

  • Structured output validation — enforce JSON/XML/schema compliance
  • Format enforcement — ensure markdown tables, code blocks, or templates are correct

Mode 0: Passive (observe-only)

For monitoring and logging — hook runs but never blocks or nudges.

class AuditLogHook:
    def check(self, response: str, context: dict) -> HookResult:
        log_to_audit_trail(response, context)
        return HookResult(passed=True)  # always pass

Use cases:

  • Audit trail — log every response for compliance
  • Quality scoring — rate responses, feed data back into prompt/model optimization
  • Cost monitoring — alert on unusually long responses

API Design

Hook Interface

@dataclass
class HookResult:
    passed: bool                           # True = let response through
    action: str = "block"                  # "block" | "nudge"
    message: str = ""                      # block: refusal text | nudge: corrective hint

class Hook:
    system_prompt_addition: str = ""       # injected into system prompt (pre-response)
    
    def check(self, response: str, context: dict) -> HookResult:
        ...

Core Integration (~30 LOC in run_agent.py)

# After strip_think_blocks(), before _build_assistant_message():
result = run_post_response_checks(self._hooks, response, context)
if not result.passed:
    if result.action == "block":
        # Replace response with block message, deliver once, no re-generation
        response = result.message
    elif result.action == "nudge" and self._empty_content_retries < 1:
        self._empty_content_retries += 1
        # Append nudge, continue main loop for one re-generation

Config (config.yaml)

agent:
  post_response_hooks:
    - module: pii_filter
      enabled: true
    - module: content_safety
      enabled: true
    - module: json_format
      enabled: false
    - module: audit_log
      enabled: true

What I've Built

Working implementation in my fork, 94 automated tests passing:

ComponentLinesDescription
agent/post_response_hooks.py~130 NEWHook loader + check pipeline
run_agent.py~30 changed4 small edits (import, init, prompt injection, main loop gate)

Core change is 4 small edits to run_agent.py. All hook logic lives in ~/.hermes/hooks/, survives upgrades.

Test Coverage

CategoryTestsKey scenarios
Framework robustness20Empty lists, missing modules, exceptions (fail-open), unicode, 10k chars
Execution semantics12Multi-hook ordering, first-fail-wins, disabled hooks, prompt aggregation
Response classification32Depth keywords (CN+EN), structure patterns, code detection
Intent-aware decisions18User intent + response quality interaction
Cross-language8Mixed CN/EN/JA, code+prose
Real-world12AML analysis, architecture, code gen, debugging

Design Properties

  • Fail-open: hook exception → skip, never blocks the agent
  • Max one nudge: prevents infinite re-generation loops
  • Block is instant: no extra token cost
  • Zero coupling: hooks live outside the codebase, survive upgrades

Ask

Would you accept a PR for this? I would submit:

  • Core framework with HookResult (block/nudge/pass)
  • Hook loader and config parsing
  • Tests for the framework
  • No domain-specific hooks — those are userland

Tagging: @schizohub @benpais

extent analysis

TL;DR

Implement a post-response hook framework to enable custom inspection and gating of responses before delivery.

Guidance

  • Review the proposed Hook interface and HookResult dataclass to ensure they meet the requirements for response inspection and gating.
  • Evaluate the core integration changes in run_agent.py to understand how the hook framework will be invoked and how responses will be modified or blocked.
  • Consider the implications of the "fail-open" design property, where hook exceptions will cause the hook to be skipped, and determine if this aligns with the desired behavior.
  • Examine the test coverage to ensure that key scenarios, such as multi-hook ordering and response classification, are adequately tested.

Example

The provided code snippets, such as the PIIFilterHook and JSONFormatHook classes, demonstrate how users can implement custom hooks to inspect and gate responses.

Notes

The proposed framework appears to be well-designed, with a clear interface and robust test coverage. However, it is essential to carefully review the implementation and consider potential edge cases to ensure that the framework meets the requirements and is reliable in production.

Recommendation

Apply the proposed post-response hook framework, as it provides a flexible and extensible way to inspect and gate responses, addressing the current limitations of the fire-and-forget callback approach. This will enable users to implement custom logic for safety gates, compliance checks, and format validation, among other use cases.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#optimization#tokenizer error#prompt formatting#chain error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING