hermes - 💡(How to fix) Fix [Feature]: GUI Teaching for Hermes Agent – Record mouse/keyboard workflows as executable skills (macOS) [1 participants]

git clone https://github.com/VodoooFilms/Linka
cd Linka
npm install
npm run build:native:mac
npm run dev

### Proposed Solution

### Problem Statement

Hermes Agent currently excels at automating code and terminal tasks, but it lacks a native way to learn or automate repetitive **Graphical User Interface (GUI)** workflows (e.g., opening a browser, navigating a specific site, clicking a series of buttons).

| Before | After |
|--------|-------|
| Hermes could only automate code and terminal | Hermes can now learn and replay **any GUI workflow** |
| Skills required manual coding | Skills are **demonstrated once** and auto‑generated |
| No native GUI context | Events include **app name, window title, bounds** |

---

### Proposed Solution

I have developed **Linka**, an open‑source (MIT) remote control app that now includes a full Hermes integration with **Teach Mode** on macOS.

**Repository:** https://github.com/VodoooFilms/Linka

**How it works:**

1. Click **Teach** in Linka’s UI
2. Perform a GUI workflow once (e.g., open Safari → YouTube → play a mix)
3. Click **Teach** again → name your skill
4. Linka auto‑generates an executable skill file at `~/.hermes/skills/linka/<name>.md`
5. Hermes Agent can replay it anytime

**What this enables Hermes to do:**

- Ingest recordings via `linka_ingest`
- Detect repeated patterns with `linka_analyze`
- Load skills from `~/.hermes/skills/linka/`
- Replay workflows with window‑offset compensation

**Technical implementation (already working):**

| Component | Technology | Status |
|-----------|------------|--------|
| Event capture | CGEventTap (Swift) | ✅ |
| 60‑second circular buffer | Native Swift | ✅ |
| Teach markers | `teach_start/stop` | ✅ |
| HTTP endpoint | `GET /hermes/events` | ✅ |
| Hotkey | `Cmd+Shift+Option+L` | ✅ |
| Skill generation | Markdown + YAML + JSON | ✅ |
| Hermes tools | `linka_ingest`, `linka_analyze` | ✅ |
| Smoke tests | `node:test` (3/3 pass) | ✅ |

**Files changed (14 files, +3,920 −2,327 lines):**

- `main.js` – hotkey, inbox dump
- `server.js` – `/hermes/events`, `teach_save`, skill generator
- `native/mac-input/main.swift` – CGEventTap, buffer, markers
- `platform/input/macos.js` – `teachStart/Stop`, `dumpEvents`
- `index.html` – Teach button, modal, responsive UI
- `README.md` – full documentation
- `tests/smoke.test.js` – smoke test suite

---

### Alternatives Considered

| Approach | Why not chosen |
|----------|----------------|
| Manual scripting (PyAutoGUI, AppleScript) | Requires coding, no pattern detection |
| Robot.js | No native CGEvent, permission issues |
| Hammerspoon | Lua‑only, no Hermes integration |
| Commercial tools (Keyboard Maestro) | Closed source, no Hermes bridge |

**Why Linka is different:** It records **context** (app name, window bounds, screenshot) and generates **executable Hermes skills** with zero friction.

---

### Additional Context

**Testing instructions:**

```bash
git clone https://github.comVodoooFilms/Linka
cd Linka
npm install
npm run build:native:mac
npm run dev

### Alternatives Considered

_No response_

### Feature Type

New tool

### Scope

None

### Contribution

- [ ] I'd like to implement this myself and submit a PR

### Debug Report (optional)

```shell