openclaw - 💡(How to fix) Fix Support Azure Foundry GPT Realtime Talk via gateway relay [1 pull requests]

Summary

Add first-class Talk support for Azure AI Foundry / Azure OpenAI GPT Realtime deployments.

Today OpenClaw can diagnose when OpenAI Realtime Talk is configured against api.openai.com with Azure/Foundry or OpenAI-compatible proxy credentials, but Azure Foundry users still need a supported path for voice Talk itself.

Microsoft documents Azure OpenAI GPT Realtime audio with Azure/Foundry endpoints, deployments, API-key or Microsoft Entra ID auth, and GA /openai/v1 endpoint behavior:

https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/realtime-audio?tabs=keyless%2Cwindows&pivots=ai-foundry-portal

Problem

Azure Foundry users can create API keys for model deployments, but those keys are scoped to Azure/Foundry endpoints. They are not direct OpenAI Platform credentials for https://api.openai.com.

For browser OpenAI Realtime WebRTC, OpenClaw currently needs an OpenAI Platform-compatible client-secret flow. Azure/Foundry credentials should not be sent there. Users need a path where OpenClaw connects to the Azure/Foundry Realtime deployment using Azure auth and keeps those credentials server-side.

Proposed direction

Support Azure Foundry / Azure OpenAI Talk through the Gateway Relay / server-side realtime bridge path, not by reusing the OpenAI Platform browser client-secret flow.

A likely configuration shape:

talk.realtime.provider = "openai"
talk.realtime.transport = "gateway-relay"
talk.realtime.providers.openai.azureEndpoint = "https://<resource>.openai.azure.com"
talk.realtime.providers.openai.azureDeployment = "<gpt-realtime-deployment>"
talk.realtime.providers.openai.azureApiVersion = "<supported-api-version>"

Possible follow-up alias:

talk.realtime.provider = "azure-openai"

that resolves to the existing OpenAI-compatible realtime bridge with Azure-specific auth/config validation.

Auth and compliance considerations

• Keep Azure API keys and Entra tokens server-side; never expose them to browser Talk clients.
• Prefer Microsoft Entra ID / managed identity where supported by the Azure Realtime endpoint.
• Support API-key auth as a practical initial path for local/dev and existing Azure OpenAI deployments.
• Document Key Vault / secret-reference usage and rotation expectations for API-key deployments.
• Do not silently fall back from Azure/Foundry auth to api.openai.com.
• Surface Azure-specific 401/403/deployment/region/model errors instead of direct OpenAI Platform credential diagnostics.

Implementation plan

1. Harden the existing Azure deployment bridge path for current Azure GPT Realtime GA endpoints.
2. Add config validation / doctor diagnostics for:
   • missing azureEndpoint
   • missing azureDeployment
   • unsupported transport for Azure Realtime
   • missing API key / Entra auth
   • Azure auth failures
   • unsupported region/model/deployment errors where detectable
3. Ensure browser-owned Talk chooses Gateway Relay when Azure Realtime config is present.
4. Add regression tests for:
   • Azure API-key relay config
   • missing deployment
   • missing auth
   • invalid Azure 401/403 preserving Azure-relevant errors
   • no accidental api.openai.com request with Azure credentials
   • custom endpoint/proxy compatibility
5. Update Talk and OpenAI provider docs with a complete Azure Foundry Realtime setup example.
6. Optionally add Entra ID / managed identity support after the API-key route is stable.

Acceptance criteria

• A user with Azure Foundry / Azure OpenAI GPT Realtime deployment can enable Talk without direct OpenAI Platform credentials.
• Azure credentials remain server-side and are not sent to browser clients.
• OpenClaw connects to the configured Azure/Foundry endpoint and deployment, not api.openai.com.
• Azure auth/deployment errors are actionable and Azure-specific.
• Existing direct OpenAI Realtime WebRTC setups and custom endpoint proxy setups continue to work.

Related context

• Related diagnostic PR: #87273
• Related prior realtime protocol issue: #80808