hermes - 💡(How to fix) Fix Modal terminal backend: blocking Modal API called from a running event loop → "Task was destroyed but it is pending" / "no running event loop"

Summary

The Modal terminal backend calls Modal's blocking API from inside Hermes's running asyncio event loop. This is the documented Modal anti-pattern and produces a flood of asyncio: Task was destroyed but it is pending! errors (from Modal's synchronicity library) plus RuntimeError: no running event loop (from grpclib). Modal sandboxes do not work reliably in the normal agent / chat path as a result.

Environment

• Hermes v0.13.0; modal package v1.4.2
• terminal.backend: modal, direct mode (MODAL_TOKEN_ID/MODAL_TOKEN_SECRET valid; ~/.modal.toml present)

Observed

A chat / Kanban-worker run with terminal.backend: modal ran for ~11 minutes then collapsed with repeated:

ERROR asyncio: Task was destroyed but it is pending!
task: <Task pending coro=<Synchronizer._run_function_async.<locals>.wrapper_coro() ...>
task: <Task pending coro=<ModalEnvironment._modal_bulk_download.<locals>._download() ... tools/environments/modal.py:357>

and, in stderr, RuntimeError: no running event loop from grpclib. The shell command never reliably completed in the Modal sandbox.

Root cause

tools/environments/modal.py calls the synchronous/blocking Modal API (modal.Sandbox.create(...), sandbox.process.exec(...), App.lookup(...), etc.) from a thread that already has a running asyncio event loop. Modal's official docs (https://modal.com/docs/guide/async) state that from async code you must use the .aio variants, because Modal's synchronicity layer otherwise orphans tasks when the foreign loop reclaims the thread — producing exactly the Task was destroyed but it is pending / no running event loop symptoms.

Suggested fix

In tools/environments/modal.py, use the async .aio API throughout when running inside the event loop: await modal.Sandbox.create.aio(...), await App.lookup.aio(...), await sandbox.process.exec.aio(...), await sandbox.terminate.aio(), etc.

Note

Hermes's RL/Atropos pipeline appears to monkey-patch Modal's event-loop handling (HermesAgentBaseEnv), which is presumably why Modal works there but not in the normal agent path. A proper .aio integration in modal.py would fix it for all paths.