hermes - ✅(Solved) Fix SSH backend: `tar xf - -C /` corrupts remote home directory mode, breaks sshd StrictModes [3 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
NousResearch/hermes-agent#17767Fetched 2026-05-01 05:56:01
View on GitHub
Comments
2
Participants
2
Timeline
13
Reactions
0
Timeline (top)
referenced ×4cross-referenced ×3labeled ×3commented ×2

The SSH terminal backend in tools/environments/ssh.py extracts a tar stream into the remote root with tar xf - -C /, which silently overwrites the mode of the remote user's home directory (e.g. /home/<user>). When the user's umask is 002 (Ubuntu / userprivate-group default on both ends), the home directory ends up as 0775 (group-writable), which causes OpenSSH's default StrictModes yes to refuse to read ~/.ssh/authorized_keys. Subsequent SSH connections fail with Permission denied (publickey) and the only diagnostic is in /var/log/auth.log:

sshd: Authentication refused: bad ownership or modes for directory /home/<user>

Root Cause

tools/environments/ssh.py:185:

ssh_cmd.append(\"tar xf - -C /\")

GNU tar's default is to overwrite existing directory metadata; the option to opt out (--no-overwrite-dir) is not passed. Because the staging path includes the remote user's home dir as an intermediate (staging/home/<user>/.hermes/...), that home dir appears as an archive entry with the local laptop's umask applied (0775 when umask is 002), and tar then chmods the live /home/<user> to that mode on extraction.

Fix Action

Fix / Workaround

Tested locally: the one-line patch above stops /home/deploy from being clobbered to 0775 on subsequent Hermes runs, and SSH publickey auth keeps working.

Workaround for affected users (until patched)

Either:

  • Run hermes with umask 022 set in the launching shell (avoids creating group-writable staging dirs in the first place), or
  • Patch tools/environments/ssh.py:185 locally with --no-overwrite-dir as above, and sudo chmod 755 /home/<user> on the remote.

PR fix notes

PR #17776: fix(ssh): pass --no-overwrite-dir to tar extract to prevent clobbering directory modes

Description (problem / solution / changelog)

Summary

Fixes #17767

The SSH bulk upload (_ssh_bulk_upload in tools/environments/ssh.py) pipes a tar archive into tar xf - -C / on the remote host. GNU tar's default behaviour overwrites existing directory metadata with what is in the archive. When the local umask produces group-writable dirs (0775, common on Ubuntu with umask 002), intermediate directories like the user's home (/home/<user>) get their mode overwritten to 0775. This causes OpenSSH StrictModes to reject ~/.ssh/authorized_keys, silently locking the user out of SSH with Permission denied (publickey).

Changes

  • tools/environments/ssh.py:185: Added --no-overwrite-dir flag to the remote tar xf invocation. This preserves the metadata of directories that already exist on the remote while still extracting all file content correctly. New directories created by tar still get reasonable modes via the remote umask.
  • tests/tools/test_ssh_bulk_upload.py: Updated assertion to match the new tar command.

Testing

  • All existing SSH environment tests pass (34 passed, 11 skipped).
  • The fix is a single flag addition to a shell command — minimal risk of regression.

Linked Issue

Closes #17767

Changed files

  • tests/tools/test_ssh_bulk_upload.py (modified, +1/-1)
  • tools/environments/ssh.py (modified, +1/-1)

PR #17867: fix(ssh): prevent tar from overwriting remote home dir permissions (#17767)

Description (problem / solution / changelog)

Summary

The SSH terminal backend's _ssh_bulk_upload() uses tar xf - -C / to extract files to the remote root. GNU tar's default behavior overwrites the metadata (including mode) of existing directories. When the local umask is 002 (Ubuntu default), staging directories are created with 0775 mode, and tar overwrites /home/<user> to 0775.

This breaks OpenSSH's StrictModes yes (default), which requires the home directory to NOT be group-writable. Subsequent SSH connections fail with Permission denied (publickey).

Fix

Add --no-overwrite-dir to the remote tar command:

# Before
tar xf - -C /

# After
tar xf - --no-overwrite-dir -C /

--no-overwrite-dir tells tar to skip updating the attributes of already-existing directories, preserving the remote system's correct 0755 mode.

Impact

  • Fixes SSH authentication breakage after first file sync
  • No functional change to file extraction — files are still written correctly
  • Only directory metadata preservation changes

Fixes #17767

Changed files

  • tools/environments/ssh.py (modified, +5/-1)

PR #17898: fix(ssh): prevent tar from overwriting remote home dir permissions (#17767)

Description (problem / solution / changelog)

Salvages #17867 by @vominh1919 onto current main. Closes #17767.

Problem

The SSH terminal backend's _ssh_bulk_upload() runs tar xf - -C / on the remote, extracting a staging directory tree to the remote root. GNU tar's default is to overwrite metadata (including mode) of existing directories. When the local umask is 002 (Ubuntu default), staging dirs are 0775, and tar chmods /home/<user> to 0775 on the remote.

That violates sshd StrictModes yes (which requires non-group-writable home dirs), so subsequent SSH connections fail with Permission denied (publickey). Recovery requires out-of-band console access to chmod the home dir back. Reporter (@luismartinezs) hit this on a Hetzner VPS and couldn't reconnect.

Confirmed on current main — tools/environments/ssh.py:185 still has the bare tar xf - -C /.

Fix (author: @vominh1919, 1 file, +5/-1)

Add --no-overwrite-dir to the remote tar command. Tar skips updating attributes of already-existing directories, so new entries (~/.hermes/, ~/.hermes/skills/) are still created with reasonable modes via remote umask, but /home/<user> keeps its original 0755.

Follow-up test update

tests/tools/test_ssh_bulk_upload.py::test_tar_pipe_commands asserted the literal substring "tar xf - -C /" in the ssh command — which is no longer present with --no-overwrite-dir between tar xf - and -C /. Split into three separate assertions; added an explicit check for the new flag as a regression guard.

Validation

scripts/run_tests.sh tests/tools/test_ssh_bulk_upload.py tests/tools/test_ssh_environment.py
34 passed, 11 skipped

Authorship preserved for @vominh1919 via plain cherry-pick.

Changed files

  • tests/tools/test_ssh_bulk_upload.py (modified, +4/-2)
  • tools/environments/ssh.py (modified, +5/-1)

Code Example

sshd: Authentication refused: bad ownership or modes for directory /home/<user>

---

ssh_cmd.append(\"tar xf - -C /\")

---

-            ssh_cmd.append(\"tar xf - -C /\")
+            ssh_cmd.append(\"tar xf - --no-overwrite-dir -C /\")
RAW_BUFFERClick to expand / collapse

Summary

The SSH terminal backend in tools/environments/ssh.py extracts a tar stream into the remote root with tar xf - -C /, which silently overwrites the mode of the remote user's home directory (e.g. /home/<user>). When the user's umask is 002 (Ubuntu / userprivate-group default on both ends), the home directory ends up as 0775 (group-writable), which causes OpenSSH's default StrictModes yes to refuse to read ~/.ssh/authorized_keys. Subsequent SSH connections fail with Permission denied (publickey) and the only diagnostic is in /var/log/auth.log:

sshd: Authentication refused: bad ownership or modes for directory /home/<user>

Reproduction

  1. Local laptop and remote VPS both running default Ubuntu (umask 002).
  2. Configure hermes setup terminal with the SSH backend, target user = the same user you SSH into the box as (e.g. deploy).
  3. First Hermes run that triggers a file sync: the _ssh_bulk_upload codepath archives a local staging tree (os.makedirs produces 0775 dirs because of umask 002) and runs tar xf - -C / on the remote.
  4. GNU tar default behavior overwrites metadata of existing directories with what's in the archive entry, so /home/<user> is chmod'd to 0775.
  5. New SSH connections to the same user fail. Existing ControlMaster sessions stay alive, masking the issue until the next reconnect.

stat /home/<user> after the break shows Modify (file added in dir) and Change (mode of dir itself overwritten) timestamps within seconds of each other, both at the moment of the Hermes sync — proving tar wrote the dir entry, not just the children.

/var/log/auth.log shows Authentication refused: bad ownership or modes for directory /home/<user> from the moment of the sync onward.

Root cause

tools/environments/ssh.py:185:

ssh_cmd.append(\"tar xf - -C /\")

GNU tar's default is to overwrite existing directory metadata; the option to opt out (--no-overwrite-dir) is not passed. Because the staging path includes the remote user's home dir as an intermediate (staging/home/<user>/.hermes/...), that home dir appears as an archive entry with the local laptop's umask applied (0775 when umask is 002), and tar then chmods the live /home/<user> to that mode on extraction.

Proposed fix

Pass --no-overwrite-dir to the remote tar invocation:

-            ssh_cmd.append(\"tar xf - -C /\")
+            ssh_cmd.append(\"tar xf - --no-overwrite-dir -C /\")

This preserves the metadata of existing directories on the remote while still extracting all file content correctly. New directories created by tar (e.g. ~/.hermes/, ~/.hermes/skills/) still get reasonable modes via remote umask.

Tested locally: the one-line patch above stops /home/deploy from being clobbered to 0775 on subsequent Hermes runs, and SSH publickey auth keeps working.

Severity

Silent privilege/access disruption. The user is locked out of SSH with a generic Permission denied (publickey) message, no client-side hint, and recovery requires console/KVM access (Hetzner Cloud Console, AWS Session Manager, etc.) to chmod the home dir. For users without out-of-band console access, this can be a hard lockout.

Workaround for affected users (until patched)

Either:

  • Run hermes with umask 022 set in the launching shell (avoids creating group-writable staging dirs in the first place), or
  • Patch tools/environments/ssh.py:185 locally with --no-overwrite-dir as above, and sudo chmod 755 /home/<user> on the remote.

Environment

  • Hermes installed via the published install.sh (latest as of 2026-04-30).
  • Local: Ubuntu 24.04, OpenSSH 9.6p1, umask 002.
  • Remote: Ubuntu 24.04 (Hetzner cloud image), OpenSSH 9.6p1, deploy user with umask 002, StrictModes yes (default).

extent analysis

TL;DR

Passing --no-overwrite-dir to the remote tar invocation in tools/environments/ssh.py should fix the issue by preserving the metadata of existing directories on the remote.

Guidance

  • The root cause is the default behavior of GNU tar overwriting existing directory metadata without the --no-overwrite-dir option.
  • To verify the fix, check the mode of the remote user's home directory after running Hermes with the patched ssh.py file.
  • As a temporary workaround, running Hermes with umask 022 set in the launching shell or patching tools/environments/ssh.py locally with --no-overwrite-dir and running sudo chmod 755 /home/<user> on the remote can resolve the issue.
  • To mitigate the issue, ensure that the remote user's home directory has the correct permissions after running Hermes.

Example

The proposed fix involves changing the line in tools/environments/ssh.py from:

ssh_cmd.append("tar xf - -C /")

to:

ssh_cmd.append("tar xf - --no-overwrite-dir -C /")

Notes

This fix assumes that the issue is caused by the default behavior of GNU tar and that passing --no-overwrite-dir will resolve the issue. If the issue persists, further investigation may be necessary.

Recommendation

Apply the workaround by patching tools/environments/ssh.py locally with --no-overwrite-dir and running sudo chmod 755 /home/<user> on the remote, as this provides a temporary solution until a permanent fix can be implemented.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING