• Metrics: memory usage, tool latency, error counts, restart frequency
• Error rate per tool
• Error rate >5% (last 5min) → critical

Fix Plan

To integrate OpenTelemetry auto-instrumentation and a simple HTTP dashboard (Pulse) for proactive monitoring of OpenClaw agent health, follow these steps:

• Step 1: Add dependencies
  • Add opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation-requests to pyproject.toml:

  [tool.poetry.dependencies]
  opentelemetry-api = "^1.13.0"
  opentelemetry-sdk = "^1.13.0"
  opentelemetry-instrumentation-requests = "^0.23.0"

• Step 2: Initialize tracer and metrics
  • Create src/observability/otel_setup.py to initialize the tracer and metrics:

  from opentelemetry import trace
  from opentelemetry.sdk.trace import TracerProvider
  from opentelemetry.sdk.trace.export import (
      ConsoleSpanExporter,
      SimpleSpanProcessor,
  )
  
  provider = TracerProvider()
  processor = SimpleSpanProcessor(ConsoleSpanExporter())
  provider.add_span_processor(processor)
  trace.set_tracer_provider(provider)

• Step 3: Wrap agent entry
  • Wrap the agent entry with opentelemetry-instrument:

  opentelemetry-instrument python -m openclaw ...

• Step 4: Create Pulse app
  • Create a Flask app for the Pulse dashboard with a /metrics endpoint:

  from flask import Flask, jsonify
  from prometheus_client import generate_latest
  
  app = Flask(__name__)
  
  @app.route("/metrics")
  def metrics():
      return generate_latest()

• Step 5: Deploy Pulse
  • Deploy the Pulse app alongside the gateway as a systemd service openclaw-pulse.

Verification

To verify that the fix worked, check the following:

• The Pulse dashboard is accessible and displays the expected metrics (memory usage, tool latency, error counts, restart frequency).
• The /metrics endpoint returns Prometheus-formatted metrics.
• The /health endpoint returns a successful response.

Extra Tips

• Make sure to configure the observability.json file with the correct settings for metrics interval, trace sampling ratio, and alerts.
• Use a tool like curl to test the /metrics and /health endpoints.
• Consider adding additional metrics and alerts to the Pulse dashboard as needed.