hermes - 💡(How to fix) Fix [Enhancement] computer_use: action=type should fall back to key events for terminal emulators (Ghostty/Terminal.app/iTerm2)

Bug Description

When using computer_use(action="type", text="...", app="Ghostty"), the tool reports success via AX insertion, but the typed characters never reach the terminal's pty. The text is silently dropped — the shell never sees it.

This is a silent failure: cua-driver returns ✅ Inserted N char(s) into focused AXTextArea, yet nothing appears in the terminal and no command is executed.

The correct workaround is to send individual keystrokes via action="key", which goes through the keyboard event path and correctly reaches the pty.

Steps to Reproduce

1. Open Ghostty (or Terminal.app) with a shell prompt
2. Call computer_use(action="type", text="ls", app="Ghostty")
3. Call computer_use(action="key", keys="return")
4. Observe: nothing happens in the terminal

Working alternative:

computer_use(action="key", keys="l")
computer_use(action="key", keys="s")
computer_use(action="key", keys="return")

This correctly types ls into the shell and executes it.

Expected Behavior

action="type" should detect that the target app is a terminal emulator (whose input path is pty, not an AX text field) and automatically fall back to sending keyboard events — the same path used by action="key".

Alternatively, the tool should surface a warning when AX insertion succeeds but the target element is a terminal pty surface, so the caller knows to use key instead.

Root Cause

Terminal emulators like Ghostty do expose an AXTextArea for accessibility, but writing to it via AX does not inject into the pty. Input must go through the keyboard event path to be received by the shell. The action="type" path uses AX insertion and has no fallback for this case.

Affected Component

• tools/computer_use/ — action="type" handler

Environment

• macOS 26.3
• Ghostty (latest)
• cua-driver 0.2.0
• Hermes Agent v0.14.0

Proposed Fix

In the type action handler, detect if the target app is a known terminal emulator (or detect that the focused element is a pty-backed surface) and route through key events instead of AX text insertion. Known terminal apps: Ghostty, Terminal, iTerm2, Alacritty, Kitty, WezTerm.