openclaw - 💡(How to fix) Fix Feature: Per-API-call latency metrics in diagnostics.otel [1 comments, 1 participants]

Summary

The built-in diagnostics.otel module tracks openclaw.run.duration_ms which measures the entire agent turn (potentially 10+ tool loop iterations). There is no per-API-call latency metric, making it impossible to distinguish between slow LLM responses vs slow tool execution vs framework overhead.

Problem

When investigating slow oncall agent responses, we found:

• Some sessions had 10-14s per tool roundtrip (vs normal ~2s)
• Root cause analysis required manually parsing session JSONL timestamps
• Gateway logs showed Slack WebSocket instability during the same period, but we couldn't determine how much delay was from Slack WS vs Claude API response time
• The existing openclaw.run.duration_ms histogram gave us only one data point (342s total), not the per-call breakdown

Current State

The timing code already exists in src/agents/pi-embedded-runner/run/attempt.ts:

// Line 1467
const promptStartedAt = Date.now();

// Line 1670  
log.debug(`embedded run prompt end: runId=... durationMs=${Date.now() - promptStartedAt}`);

But:

1. It only writes to log.debug() — filtered out at default log level
2. It is not emitted as a diagnostic event → diagnostics-otel cannot export it
3. The agent_end hook receives durationMs but that's per-attempt, not surfaced as a metric

Requested Metrics

1. Per-API-call latency (openclaw.prompt.duration_ms)

• Type: Histogram
• Attributes: openclaw.channel, openclaw.provider, openclaw.model, openclaw.sessionKey
• What it measures: Time from prompt assembly start to LLM response complete (the existing promptStartedAt → Date.now() in attempt.ts)
• Emit point: attempt.ts:1668 (the finally block after prompt execution)

2. Slack WebSocket health

• openclaw.slack.ws_pong_timeout_total (Counter) — pong wasn't received within timeout
• openclaw.slack.ws_reconnect_total (Counter) — WebSocket reconnection attempts
• Emit point: extensions/slack/src/monitor/provider.ts where reconnectAttempts is tracked

3. Handle tool.loop diagnostic event

• tool.loop is already defined in src/infra/diagnostic-events.ts (lines 137-148) but the OTel service in extensions/diagnostics-otel/src/service.ts does not handle it (silently dropped)
• This would give per-tool-loop iteration metrics

Suggested Implementation

Minimal change (~20 lines):

1. In attempt.ts:1668, add emitDiagnosticEvent({ type: "prompt.completed", durationMs, provider, model, sessionKey })
2. In diagnostics-otel/src/service.ts, handle "prompt.completed" → record to a new histogram
3. In extensions/slack/src/monitor/provider.ts, emit diagnostic events for pong timeout and reconnect

Environment

• OpenClaw 2026.4.1 (da64a97)
• Provider: Anthropic Claude Opus 4.6
• Channel: Slack (Socket Mode)