Issue Station
Sign In
Tools hereToolbox

hermes - 💡(How to fix) Fix Bundled skill SKILL.md files drift from codebase — need an automated update mechanism [1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#13737•Fetched 2026-04-22 08:04:26
View on GitHub
Comments
0
Participants
1
Timeline
2
Reactions
0
Author
piliplakerpiliplaker
Participants
piliplakerpiliplaker
Timeline (top)
labeled ×2

Fix Action

Fix / Workaround

  1. Periodic stale detection: Let the agent itself detect when a bundled skill is outdated (it already has a "patch nudge" mechanism) and surface a warning or auto-file an issue.

Code Example

| `terminal` | `backend` (local/docker/ssh/modal), ...

---

terminal:
  backend: local # local | docker | ssh | modal | daytona | singularity
RAW_BUFFERClick to expand / collapse

Problem

Bundled skills (e.g. skills/autonomous-ai-agents/hermes-agent/SKILL.md) are static documents that ship with the repo, but they gradually fall out of sync with the actual codebase.

For example, the hermes-agent skill currently lists 4 terminal backends (local/docker/ssh/modal) while the code and official config docs support 6 (missing daytona and singularity):

SKILL.md (stale):

| `terminal` | `backend` (local/docker/ssh/modal), ...

Actual config docs:

terminal:
  backend: local # local | docker | ssh | modal | daytona | singularity

This is not just about one missing backend — it's a systemic problem. As features are added, providers are added, CLI commands change, etc., the bundled skill docs don't get updated unless someone remembers to do it manually. The agent then relies on these stale docs as its primary reference and gives incorrect answers.

Existing sync mechanism

  • ~/.hermes/skills/.bundled_manifest tracks MD5 hashes of bundled skills
  • hermes update / hermes skills check can sync if upstream has changed
  • But this only works if the upstream SKILL.md itself is up to date — which it often isn't

Proposed solutions (pick or combine)

  1. CI check: Add a GitHub Actions workflow that compares key facts in SKILL.md against the actual config defaults / codebase (e.g. parse config.py for supported terminal backends and assert they match what the skill doc lists). Fail CI if there's a drift.

  2. Auto-generate parts of SKILL.md: Extract provider lists, backend options, toolset names, etc. from the source code and inject them into the skill doc at build time, so they're always accurate.

  3. Post-commit hook / pre-merge check: When a PR adds a new backend, provider, or toolset, require the corresponding skill doc to be updated.

  4. Periodic stale detection: Let the agent itself detect when a bundled skill is outdated (it already has a "patch nudge" mechanism) and surface a warning or auto-file an issue.

Option 1 (CI check) seems like the lowest-effort, highest-impact approach. Even a simple grep-based check would catch most drift.

extent analysis

TL;DR

Implement a CI check using GitHub Actions to compare key facts in SKILL.md against the actual config defaults/codebase to ensure accuracy and prevent drift.

Guidance

  • Identify the key facts that need to be synced between SKILL.md and the codebase, such as terminal backends and provider lists.
  • Develop a GitHub Actions workflow that parses the codebase (e.g., config.py) and compares the extracted information with the corresponding data in SKILL.md.
  • Implement a CI check that fails if discrepancies are found, ensuring that the skill docs are updated before changes are merged.
  • Consider starting with a simple grep-based check to catch most drift cases and refine the check as needed.

Example

name: Skill Doc Checker
on: [push]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Run CI check
        run: |
          # Parse config.py to extract supported terminal backends
          backends=$(grep "terminal.backend" config.py | cut -d':' -f2-)
          # Compare with SKILL.md
          if ! grep -q "$backends" SKILL.md; then
            echo "Skill doc is out of sync"
            exit 1
          fi

Notes

This approach assumes that the codebase and skill docs are structured in a way that allows for easy parsing and comparison. Additional complexity may arise if the codebase or skill docs are highly dynamic or complex.

Recommendation

Apply workaround: Implement a CI check using GitHub Actions, as it seems to be the lowest-effort, highest-impact approach to ensure the accuracy of skill docs.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#permission error#memory optimization#batch processing#GPU compatibility#latency issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

hermes - 💡(How to fix) Fix Bundled skill SKILL.md files drift from codebase — need an automated update mechanism [1 participants]