openclaw - 💡(How to fix) Fix [Feature]: Clicky companion support - screen-aware voice assistant as node [1 participants]

Summary

Support Clicky (https://github.com/farzaa/clicky) as a native macOS node/client that connects to the Gateway via WS Protocol v3 — enabling a screen-aware voice companion with persistent memory, tools, and multi-model support.

Problem to solve

OpenClaw has Talk Mode and Voice Wake for voice interaction, but lacks a visual desktop companion that can see the user's screen and point at UI elements. Clicky (4.5K+ stars, trending) does exactly this — screen capture, push-to-talk, blue cursor overlay with [POINT:x,y:label] — but it's a stateless Claude wrapper with no memory, no tools, and no persistent context.

Users who want a screen-aware AI assistant currently have to choose: Clicky (great UX, no backend) or OpenClaw (great backend, no visual overlay). There's no way to combine them.

Related Clicky-side proposal: https://github.com/farzaa/clicky/issues/30

Proposed solution

1. Clicky companion skill — A skill (clicky-companion) that appends POINT coordinate instructions to the system prompt when the session originates from a Clicky client. This moves the POINT prompt from Clicky's hardcoded Swift string into a maintainable, editable skill file.
2. Session routing for Clicky clients — Clicky connects as a node with caps: ["screen", "voice"]. Sessions routed to a dedicated key pattern (e.g., clicky:<device-id>) with optional model/prompt overrides via sessionOverrides.
3. Verify multi-image attachments in chat.send — Clicky sends labeled screenshots from multiple monitors as base64 image attachments. The media pipeline should already handle this, but needs verification with the multi-image + label pattern.
4. TTS via talk.speak/tts.convert — Clicky would use existing Gateway TTS RPCs instead of direct ElevenLabs calls. Main concern: low-latency audio delivery for real-time voice responses.
5. (Optional future) POINT metadata extraction — Extract [POINT:x,y:label] tags from agent responses server-side and include as structured metadata in event:agent streaming payloads. Non-blocking — client-side parsing works fine initially.

Alternatives considered

• Clicky stays standalone: users configure their own Cloudflare Worker + 3 API keys. Works but no memory, no tools, no cross-device continuity. • OCPlatform builds its own overlay: duplicates the cursor/POINT/overlay work Clicky already does well. Not worth it when integration is simpler. • Generic "screen capture" node protocol: more abstract but overkill for the immediate use case. Clicky integration is concrete and achievable now.

Impact

• Affected: macOS users who want a visual desktop AI companion with full agent capabilities • Severity: New capability (not blocking existing workflows) • Frequency: Every session — this would be the primary interaction mode for desktop users • Consequence: Without this, OpenClaw desktop interaction is text/voice only (no screen awareness, no visual pointing). Clicky users get no memory or tools.

Evidence/examples

• Clicky repo: https://github.com/farzaa/clicky (4.5K+ stars, MIT, active development) • Clicky-side feature proposal: https://github.com/farzaa/clicky/issues/30 • OpenClaw already has the full node protocol, TTS integration, and media pipeline needed • The POINT system ([POINT:x,y:label]) is proven — Clicky uses it successfully for screen element pointing

Additional information

https://github.com/farzaa/clicky

The Clicky-side PR would add OpenClawClient.swift implementing Gateway WS Protocol v3 (connect, chat.send, event:agent streaming, tts.convert). The current Cloudflare Worker mode remains the default — OpenClaw mode is opt-in.

Key design question: should Clicky connect as role: "node" (natural fit for screen/voice caps) or would a lighter-weight client role be more appropriate?

Detailed architecture doc available on request.