hermes - 💡(How to fix) Fix computer_use capture can blow context on Electron/Obsidian AX trees [1 pull requests]

Problem

computer_use captures can accidentally blow up the session context on dense macOS apps, especially Electron apps such as Obsidian.

In one real macOS session, computer_use(action="capture", mode="ax") against Obsidian returned about 597 accessibility elements. The human-readable summary is truncated, but the raw JSON response still appears to include the full elements array, which can rapidly consume context or trigger compression failures.

som mode is expected to be heavier because it includes screenshot/image payload, but ax mode feels like it should be the safer/lightweight option and can still become very large.

Why it matters

The docs position computer_use well as narrow, app-scoped background control of real Mac apps. In practice, a single broad capture on Obsidian/Electron can make the current session nearly unusable.

This is easy for agents/users to trigger accidentally when using computer_use for screen inspection rather than tightly scoped GUI control.

Relevant implementation detail

In tools/computer_use/tool.py, _format_elements(elements, max_lines=40) truncates the displayed summary, but _capture_response() for AX/text results returns:

"elements": [_element_to_dict(e) for e in cap.elements]

So the summary is compact, but the full AX element list still enters the tool result.

Suggested improvements

• Add max_elements for capture results, with a conservative default.
• Add a compact=true option that returns only summary/metadata, not raw elements.
• Truncate raw AX elements by default when element count exceeds a threshold.
• Warn on dense Electron/Chromium AX trees, e.g. “597 elements detected; returning first 40.”
• Consider file-backed screenshots or provider-aware image handling for CLI/OpenAI-Codex-style sessions.
• Update docs/skill guidance to emphasize: list_apps first, scope captures with app=..., avoid broad Obsidian/Electron captures when file/browser/terminal tools can answer instead.

Environment

• macOS
• Hermes Agent v0.13.0
• computer_use via cua-driver
• Target app observed: Obsidian / Electron