Summary

Add support for the xAI Realtime Voice Agent (grok-voice-think-fast-1.0) to OpenClaw's /voiceclaw/realtime gateway surface, implemented as a shared OpenAIRealtimeProtocolAdapter that can also serve OpenAI Realtime (issue #71195) when added.

xAI's Voice Agent API is OpenAI-Realtime-protocol-compatible (verbatim from xAI docs: "compatible with the OpenAI Realtime API. Most OpenAI client libraries and SDKs work with the xAI endpoint by changing the base URL"), so a single protocol adapter can serve both providers via parameterization.

Motivation

• xAI Realtime is GA with grok-voice-think-fast-1.0 as the flagship "think fast" voice model. The Voice Agent API is documented at https://docs.x.ai/developers/model-capabilities/audio/voice-agent and uses the OpenAI-Realtime protocol with minor wire deltas.
• OpenClaw already has the right surface — /voiceclaw/realtime (added in #70938), the VoiceClawRealtimeAdapter interface, and the provider field in VoiceClawSessionConfigEvent types — but the only wired adapter today is gemini-live.ts.
• Shared work with #71195 — issue #71195 requests OpenAI Realtime in /voiceclaw/realtime. Because xAI's API is OpenAI-Realtime-protocol-compatible, a shared OpenAIRealtimeProtocolAdapter base class would address both providers with one body of code, parameterized by URL / model defaults / voices / a small event-name override table.
• Community signal — issue #12911 also requests Grok voice integration (in the web chat / Live Voice Mode). This proposal addresses the gateway-level capability that would unblock that and similar use cases.
• Real adopter — a downstream private-AI-operator project is targeting /voiceclaw/realtime with provider: "xai" as its primary realtime voice route, preserving the credential boundary by keeping the xAI key inside OpenClaw's gateway environment. Downstream adopters are interested in driving this if maintainers are open to the design.

If maintainers prefer a smaller first PR, the initial implementation can target xAI only, with the shared adapter structured so OpenAI Realtime support can follow without duplicating protocol code.

Proposed design

New files

• src/gateway/voiceclaw-realtime/openai-realtime-protocol.ts — shared OpenAIRealtimeProtocolAdapter base class implementing VoiceClawRealtimeAdapter against the OpenAI-Realtime wire protocol. Parameterized by:
  • Default WebSocket base URL
  • Default model ID
  • Available voices
  • Event-name override table (xAI's response.text.delta vs OpenAI's response.output_text.delta; xAI's user-transcription event shape vs OpenAI's split events)
• src/gateway/voiceclaw-realtime/xai-realtime.ts — VoiceClawXaiRealtimeAdapter extends the shared base with xAI-specific defaults:
  • Base URL: wss://api.x.ai/v1/realtime?model=<model-id>
  • Default model: grok-voice-think-fast-1.0
  • Available voices: eve, ara, rex, sal, leo (all five supported; per-session override via session.config.voice)
  • Event-name overrides per the xAI docs

Modified files

• src/gateway/voiceclaw-realtime/types.ts — add "xai" to VoiceClawSessionConfigEvent.provider union (one-line change).
• src/gateway/voiceclaw-realtime/session.ts — extend the adapter factory so provider === "xai" instantiates VoiceClawXaiRealtimeAdapter.
• docs/gateway/index.md — document xAI as a /voiceclaw/realtime brain provider; required env var XAI_API_KEY.
• docs/providers/xai.md — note realtime full-duplex support; cross-link to gateway index.

Configuration shape

Operator config:

{
  "voiceclaw-realtime": {
    "default_provider": "xai",
    "xai": {
      "model": "grok-voice-think-fast-1.0",
      "voice": "ara",
      "audio_format": "audio/pcm",
      "sample_rate_hz": 24000,
      "vad": {
        "type": "server_vad",
        "threshold": 0.5,
        "silence_duration_ms": 500
      }
    }
  }
}

Plus env: XAI_API_KEY set in the gateway process environment. Optional follow-up: if maintainers prefer file-based secret loading, support an XAI_API_KEY_FILE pattern.

Client session.config first message (per-session voice override is supported — the example shows ara but any of the five xAI voices is valid):

{
  "type": "session.config",
  "provider": "xai",
  "model": "grok-voice-think-fast-1.0",
  "voice": "ara",
  "instructionsOverride": "...",
  "conversationHistory": [...],
  "watchdog": "enabled"
}

The adapter falls back to the operator-config default voice when session.config.voice is omitted, and falls back to a built-in adapter default (ara) when no operator config is present.

Endpoint and auth — unchanged

• Path stays /voiceclaw/realtime.
• Gateway auth (Bearer token in first session.config message) stays as-is.
• No new HTTP endpoint added.
• No change to gemini-live.ts, extensions/voice-call/, extensions/openai/realtime-voice-provider.ts, or extensions/xai/ batch/streaming-STT files.

Event mapping

OpenClaw → xAI (client → server)

xAI → OpenClaw (server → client)

Audio format

• Default in: audio/pcm Linear16 little-endian, 24 kHz mono (matches xAI default).
• Default out: same.
• Other formats supported by xAI (pcmu, pcma, lower sample rates): pass-through configurable; not used by default.

VAD / barge-in

Server-side VAD (turn_detection: {type: "server_vad", threshold, silence_duration_ms, prefix_padding_ms}) is the recommended default. xAI handles speech detection, end-of-turn, and barge-in. On barge-in mid-response, the adapter emits turn.ended (assistant role, with interrupted: true) before the next turn.started (user role).

Reconnect / resume

xAI does not document explicit resume tokens. Defensive implementation:

• On WS close (codes 1001/1006/1007/1011/1012/1013): auto-reconnect up to 2 attempts with 500 ms backoff, mirroring the gemini-live.ts pattern.
• On reconnect: open new WS, send fresh session.update, re-prime conversation history via conversation.item.create per historical message.
• On 120-min hard cap (xAI's documented session limit): emit session.rotating → close upstream → open new upstream → emit session.rotated. Operator UX continues seamlessly.

Open question for maintainers: does OpenClaw have a preferred reconnect/resume contract for /voiceclaw/realtime adapters, or should each adapter handle it as gemini-live.ts does?

Tools / function calling

OpenClaw tool catalog is exposed to xAI as function tools in session.update. On response.function_call_arguments.done:

1. Adapter emits tool.call to client with callId.
2. Adapter may send an immediate placeholder/tool-status response if supported by xAI / OpenAI-Realtime semantics; otherwise it should wait for the actual tool.result before sending function_call_output + response.create.
3. Client (the OpenClaw caller) executes the tool with policy/approval gates and returns via tool.result.
4. Adapter sends conversation.item.create with function_call_output to xAI.
5. Adapter sends response.create to resume generation.

Parallel function calls (xAI may emit multiple done events before audio) are handled by tracking all callIds and resolving all before response.create.

Files likely touched

• src/gateway/voiceclaw-realtime/types.ts — add "xai" to provider union
• src/gateway/voiceclaw-realtime/session.ts — extend adapter factory
• src/gateway/voiceclaw-realtime/openai-realtime-protocol.ts — NEW shared base
• src/gateway/voiceclaw-realtime/xai-realtime.ts — NEW xAI specialization
• src/gateway/voiceclaw-realtime/openai-realtime-protocol.test.ts — NEW shared protocol tests
• src/gateway/voiceclaw-realtime/xai-realtime.test.ts — NEW xAI-specific tests
• src/gateway/voiceclaw-realtime/session.test.ts — add "xai" provider-selection test
• docs/gateway/index.md — document xAI brain provider
• docs/providers/xai.md — note realtime support

Estimated code volume: shared base ~500-800 lines; xAI specialization ~150-250 lines; types/factory updates ~30-50 lines; tests ~400-600 lines; docs ~100-200 lines.

Hard non-changes: no modification to gemini-live.ts, extensions/voice-call/, extensions/openai/realtime-voice-provider.ts, or extensions/xai/ batch/streaming-STT files. No change to gateway auth, sandboxing, or the WS control protocol. No new HTTP endpoints.

Tests

• Mocked WebSocket only. No live xAI in CI. No XAI_API_KEY required to run tests.
• Audio passthrough — base64 PCM16 frames pass through unmodified.
• Transcript synthesis — verify adapter emits both transcript.delta AND transcript.done from xAI's single user-transcription completed event.
• Function-call flow — verify tool.call emission, tool.result acceptance, conversation.item.create + response.create forwarding; parallel calls handled correctly.
• Barge-in — inject input_audio_buffer.speech_started mid-response; verify turn.ended (interrupted) + turn.started (user).
• Reconnect — force WS close 1006; verify retry with conversation history re-priming.
• Session rotation — simulate 120-min cap; verify session.rotating / session.rotated.
• Error sanitization — xAI returns error reflecting key; verify sanitized error never contains key.
• Provider selection — config.provider === "xai" returns VoiceClawXaiRealtimeAdapter from session.ts factory.
• Voice selection — verify session.config.voice overrides operator-config default; verify all five xAI voices (eve, ara, rex, sal, leo) are accepted; verify invalid voice IDs are rejected with sanitized error.
• Secret never logged — inject mock logger; assert XAI_API_KEY value never appears in any log line.

Optional --live smoke against real xAI WSS would be operator-only / opt-in via explicit env flag, never run in CI.

Security

• XAI_API_KEY stays inside the OpenClaw gateway process environment.
• Never logged in any form (value, length, prefix, hash).
• Never reflected in errors returned to clients.
• No client-side xAI key — clients send the gateway Bearer token only, exactly as today.
• No new public endpoint added.
• Existing gateway auth (Bearer in first session.config message; per docs/gateway/index.md) remains unchanged.

Questions for maintainers

1. Surface choice — is /voiceclaw/realtime the right surface for xAI Realtime, or do maintainers prefer this go through a different path (e.g., the voice-call plugin's realtime providers, a new plugin-SDK extension)?
2. Adapter shape — would maintainers prefer a single shared OpenAIRealtimeProtocolAdapter base (parameterized for both xAI and OpenAI) or two provider-specific adapters with duplicated protocol logic?
3. Ship order — should xAI and OpenAI Realtime (issue #71195) ship in one combined PR, or xAI first as a smaller PR followed by OpenAI as a second PR? Either is workable; smaller PR is easier to review.
4. Code organization — should this be core gateway code (in src/gateway/voiceclaw-realtime/) or extracted via the plugin SDK? The gemini-live.ts precedent suggests core gateway code, but happy to follow maintainer preference.
5. Test/mocking style — any preferred WebSocket mocking pattern for voiceclaw-realtime adapters? gemini-live.test.ts is the natural reference; would happily follow the same style.
6. Roadmap overlap — does this overlap with planned work tied to #71195 or #12911? If maintainers are already in flight on #71195, we'd be glad to coordinate or contribute to that effort instead of opening a parallel PR.
7. Reconnect contract — does OpenClaw have a preferred reconnect/resume contract for /voiceclaw/realtime adapters, or is per-adapter best-effort (like gemini-live.ts) the expected pattern?

References

• xAI Voice Agent docs: https://docs.x.ai/developers/model-capabilities/audio/voice-agent
• xAI Voice Agent API model/pricing: https://docs.x.ai/developers/models/voice-agent-api
• xAI release notes: https://docs.x.ai/developers/release-notes
• xAI flagship voice model announcement: https://x.ai/news/grok-voice-think-fast-1
• xAI Voice Agent API launch: https://x.ai/news/grok-voice-agent-api
• xAI cookbook (voice examples): https://github.com/xai-org/xai-cookbook/tree/main/voice-examples
• OpenClaw /voiceclaw/realtime types: src/gateway/voiceclaw-realtime/types.ts
• OpenClaw /voiceclaw/realtime session: src/gateway/voiceclaw-realtime/session.ts
• OpenClaw Gemini adapter (reference shape): src/gateway/voiceclaw-realtime/gemini-live.ts
• OpenClaw gateway docs: https://docs.openclaw.ai/gateway
• OpenClaw xAI provider: https://docs.openclaw.ai/providers/xai
• OpenClaw issue #71195 — OpenAI Realtime in Talk Mode / /voiceclaw/realtime
• OpenClaw issue #12911 — Live Voice Mode in /chat UI (Grok Voice Loop)
• OpenClaw PR #70938 — origin of /voiceclaw/realtime

What we're requesting

Maintainer feedback on the design before opening a draft PR. Downstream adopters are prepared to drive the implementation but want to align on surface, adapter shape, and ship order before writing code so we don't waste review cycles.

Filed by an OpenClaw downstream adopter. Happy to attend a sync, drop in on a community call, or coordinate with anyone already working on #71195. Please flag any roadmap or design context we may be missing.