hermes - 💡(How to fix) Fix [Bug] asyncio subprocess StreamReader default 64 KiB limit truncates long output lines

Summary

asyncio subprocess spawns in Hermes that read NDJSON / line-oriented output from child processes will raise ValueError: Separator is not found, and chunk exceed the limit when a single output line exceeds 64 KiB, because asyncio's StreamReader uses a 64 KiB default limit. This silently breaks any agent that spawns a subprocess whose output can contain a long line — file dumps, SQL result rows, base64 blobs, tool results from external CLIs that quote large content, etc.

I hit this consistently while building a subprocess-based executor (separate proposal in #31385); easy to reproduce in isolation.

Severity

• Frequency: any subprocess emitting a single line > 64 KiB. Triggers reliably with claude / codex / other CLIs when the agent asks the child to quote a large file, paste tool output, dump a wide JSON, etc.
• Symptom: subprocess read raises ValueError; depending on caller, the turn either fails with a confusing error or hangs (if the read loop swallows the exception and the process never sees EOF).
• Platforms: all (Linux / macOS / Windows). Python stdlib default, not OS-specific.

Reproduction (standalone, no Hermes needed)

import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        "python", "-c", "print('x' * 70_000)",   # one line >64 KiB
        stdout=asyncio.subprocess.PIPE,
    )
    line = await proc.stdout.readline()
    print(len(line))

asyncio.run(main())

Raises:

ValueError: Separator is not found, and chunk exceed the limit

Why the default is wrong for agent subprocess use

StreamReader's 64 KiB default exists because, historically, asyncio targeted network protocols where 64 KiB is a reasonable bound. For local subprocess output from CLIs that the agent itself drives, that bound is wildly too small:

• A single base64-encoded image: ~100–500 KiB on one line
• A single SQL result row with a wide JSON column: easily > 200 KiB
• An LLM-quoted file or stack trace: easily multi-MiB
• A cat'd 10 MB log file: one process, but readline() will fail at 64 KiB

The fix is one line per subprocess spawn site:

# Use a larger StreamReader limit when spawning subprocesses we control.
loop = asyncio.get_event_loop()
transport, protocol = await loop.subprocess_exec(
    lambda: asyncio.subprocess.SubprocessStreamProtocol(
        limit=16 * 1024 * 1024,   # 16 MiB — generous but bounded
        loop=loop,
    ),
    *cmd, ...,
)

Or, for the more common create_subprocess_exec path, monkey-patch / wrap the protocol factory; or pass limit= through any helper that fronts it.

Proposed fix

1. Audit all asyncio.create_subprocess_* / asyncio.subprocess_exec callsites in Hermes (agent/, gateway/, tools/, etc.)
2. For each one whose stdout we read line-by-line: pass limit=16 * 1024 * 1024 (or expose as a config knob with that default)
3. Add a unit test that spawns a child emitting a 1 MiB single line and asserts readline() succeeds
4. Document the rationale next to the limit constant (so a future maintainer doesn't "clean up" the explicit limit thinking the default is fine)

In my own subprocess-executor work (#31385), I'm using a CLAUDE_CODE_STREAM_LIMIT = 16 * 1024 * 1024 constant for exactly this reason — but it should really be applied broadly inside Hermes, not just inside opt-in executors.

Why I'm filing it as a separate bug (not folded into #31385)

The StreamReader limit is a Hermes-wide concern that affects any agent code touching subprocess output, independent of whether the executor-bridge proposal lands. Filing here so it can be fixed and tracked on its own.

Happy to open a PR doing the audit + the fix, if there's interest. Want to gauge severity-confirmation first before sinking time into the codebase-wide audit.

Thanks!