hermes - 💡(How to fix) Fix YAML serialization inconsistency across 3 serializers causes config corruption and Gateway crash

Bug Description

Hermes Agent uses 3 different YAML serializers (PyYAML default, PyYAML+IndentDumper, ruamel.yaml) across 7+ write paths to the same config.yaml file. These produce two incompatible indentation styles for list items under mapping keys:

• Style A (0-indent): - name: at column 0 (PyYAML default)
• Style B (2-indent): - name: at column 2 (IndentDumper / ruamel.yaml)

When different paths write to the same file, indentation toggles between styles. Each toggle silently destroys all user comments and formatting. When the file lands in a mixed-indent state that js-yaml rejects (stricter than PyYAML), Hermes Gateway crashes with "bad indentation of a mapping entry" — all custom_providers become invisible, and model switching silently fails.

Steps to Reproduce

1. Start with a fresh Hermes config.yaml (default PyYAML style, 0-indent lists)
2. Run a scan from Project 003's auto_add_models.py (writes with IndentDumper → 2-indent lists) — file changes to Style B
3. Use Web UI to save config (triggers save_config() → atomic_yaml_write() → default PyYAML → 0-indent lists) — file changes back to Style A
4. Next Hermes Startup: ruamel.yaml reads file in Style A, writes a single key change (see /skin, /compact) — both styles coexist → mixed indentation
5. On next load: js-yaml (Web UI) or PyYAML rejects the mixed file → YAMLException: bad indentation of a mapping entry

Observed impact cascade:

YAML write path A (0-indent) → write path B (2-indent) → mixed file 
→ js-yaml parse failure → Gateway ignores config → custom_providers invisible 
→ model switching broken → all requests fail

Expected Behavior

1. All write paths should produce identical, deterministic YAML output regardless of which serializer is used
2. Comments should be preserved on all single-key updates (not just /skin / /compact CLI commands)
3. Writes should be validated before commit — if the output is invalid YAML, the file should not be overwritten

Actual Behavior

Key conflict: ruamel.yaml is configured with indent(mapping=2, sequence=4, offset=2), which produces 2-space-indented list items. Default PyYAML produces 0-indent list items. These two styles are incompatible in the same file.

Root Cause

/usr/local/lib/hermes-agent/utils.py has two YAML write functions:

1. atomic_yaml_write() (line 139) — uses default yaml.dump(data, f, ...) with no custom Dumper → 0-indent lists
2. atomic_roundtrip_yaml_update() (line 191) — uses ruamel.yaml YAML(typ="rt") with indent(mapping=2, sequence=4, offset=2) → 2-indent lists

Both write to the same file, but neither is aware of the other's formatting.

Environment

• OS: Linux (5.15.0-177-generic)
• Hermes Version: 0.14.0 (pip package)
• Package installed at: /usr/local/lib/hermes-agent
• Repo: https://github.com/NousResearch/hermes-agent
• Affected files:
  • utils.py (lines 139, 191, 203-213) — both YAML writers
  • hermes_cli/config.py:save_config() (line 4551) — calls atomic_yaml_write()
  • hermes_cli/auth.py:_update_config_for_provider() (line 5901) — calls atomic_yaml_write()
  • tui_gateway/server.py:_save_cfg() (line 681) — non-atomic yaml.safe_dump()
• Python: 3.11.15
• Node.js: v24.16.0 (Web UI)

Concrete Code Evidence

utils.py line 139 — path A: default PyYAML (0-indent lists)

def atomic_yaml_write(path, data, *, sort_keys=False):
    """Atomically write a YAML file."""
    content = yaml.dump(data, default_flow_style=False, sort_keys=sort_keys)
    # ... atomic write with temp file

Output:

custom_providers:
- name: NVIDIA          # ← 0-indent list item (indentless)
  base_url: ...

utils.py lines 203-213 — path B: ruamel.yaml (2-indent lists)

yaml_rt = YAML(typ="rt")
yaml_rt.preserve_quotes = True
yaml_rt.allow_unicode = True
yaml_rt.default_flow_style = False
yaml_rt.indent(mapping=2, sequence=4, offset=2)

Output:

custom_providers:
  - name: NVIDIA        # ← 2-indent list item
    base_url: ...

utils.py line 242 — writes back the ruamel-parsed content

yaml_rt.dump(config, f)

Proposed Fix

Fix 1: Unify atomic_yaml_write() to use IndentDumper (aligns with ruamel.yaml output)

class IndentDumper(yaml.Dumper):
    """PyYAML Dumper that indents list items under mapping keys (2-space)."""
    def increase_indent(self, flow=False, indentless=False):
        return super().increase_indent(flow, False)

def atomic_yaml_write(path, data, *, sort_keys=False):
    content = yaml.dump(
        data,
        default_flow_style=False,
        sort_keys=sort_keys,
        Dumper=IndentDumper  # ← Added: produces 2-indent lists
    )

This makes path A produce identical output to path B (ruamel.yaml):

custom_providers:
  - name: NVIDIA        # ← 2-indent (same as ruamel.yaml!)
    base_url: ...

Fix 2: Add pre-write YAML validation (reject invalid output)

import subprocess
import json

def validate_yaml_with_js_yaml(content: str) -> bool:
    """Validate YAML output using js-yaml (stricter than PyYAML)."""
    result = subprocess.run(
        ["node", "-e", f"""
            const yaml = require('js-yaml');
            try {{
                yaml.load({json.dumps(content)});
                process.exit(0);
            }} catch(e) {{
                console.error(e.message);
                process.exit(1);
            }}
        """],
        capture_output=True, text=True, timeout=10
    )
    if result.returncode != 0:
        raise ValueError(f"js-yaml rejected output: {result.stderr.strip()}")
    return True

Then in atomic_yaml_write():

def atomic_yaml_write(path, data, *, sort_keys=False):
    content = yaml.dump(data, ...)
    validate_yaml_with_js_yaml(content)  # ← Pre-write validation
    # ... atomic write

Fix 3 (Optional, preferred but larger scope): Migrate ALL writes to ruamel.yaml

Replace atomic_yaml_write() entirely with the ruamel.yaml roundtrip approach. This would:

• Preserve comments on ALL writes (not just CLI single-key changes)
• Use a single, consistent serializer
• Eliminate the two-paradigm problem

from ruamel.yaml import YAML

def atomic_yaml_write(path, data, *, sort_keys=False):
    """Atomically write a YAML file using ruamel.yaml (comment-preserving)."""
    yaml_rt = YAML(typ="safe" if sort_keys else "rt")
    yaml_rt.default_flow_style = False
    yaml_rt.indent(mapping=2, sequence=4, offset=2)
    # ... atomic write

Workarounds (that we applied in production)

1. All project scanner scripts (001, 003) use IndentDumper to produce 2-indent output
2. Config file has a cron watchdog that auto-fixes indentation after any write
3. A pre-commit YAML validator checks the file before Hermes restarts

Additional Context

Why js-yaml rejects what PyYAML accepts:

When a file has mixed indentation after cross-serializer writes:

custom_providers:
  - name: NVIDIA       # ← 2-indent (from ruamel)
  base_url: ...        # ← 2-indent (from PyYAML sub-key) — but `name` is at col 4!

PyYAML is lenient and may parse this. js-yaml rejects it with:

YAMLException: bad indentation of a mapping entry (572:3)

Observed failure mode in production (actual May 25 2026 incident):

• config.yaml had duplicate fallback_providers lines (broken by a 0-indent→2-indent→0-indent cycle)
• Hermes Gateway failed to parse → all 13 custom_providers invisible
• User's model switching appeared to work (UI showed the option) but API always fell back to default
• 3 hours of debugging traced to yaml.safe_load() returning defaults

See local fix applied at /usr/local/lib/hermes-agent/utils.py:

• IndentDumper class unifying all yaml.dump() to 2-space list indent
• js-yaml pre-write validation in atomic_yaml_write()