Discussion: Pulse — Central Health Dashboard

Repository: New (pulse) or part of OpenClaw core Issue Type: Discussion Priority: P2 — Observability stack Related: OpenClaw #41924 (agent health monitoring), optimization-brainstorm-v30

Goal

Provide single-pane-of-glass view into health of all OpenClaw components:

• Gateway process (CPU, memory, uptime)
• Agent memory usage, token budget, active tasks
• Cron job status (last run, success/failure)
• Disk space, database sizes
• External service connectivity (GitHub, RSS feeds, APIs)

Architecture

Data Collection

Agent-side (OpenClaw core):

• Expose /metrics endpoint (Prometheus text format)
• Metrics:
  • openclaw_agent_memory_bytes{type="workspace"}
  • openclaw_agent_tokens_total, openclaw_agent_tokens_used
  • openclaw_tasks_active, openclaw_tasks_completed_total
  • openclaw_cron_jobs{failed="bool"}
  • openclaw_gateway_up{status="bool"}

System-side (node-exporter already gives CPU/mem/disk; we just scrape)

Push vs Pull: Use Prometheus pull from /metrics endpoint. Simpler.

Storage & Visualization

• Prometheus: scrape /metrics every 15s, store TSDB
• Grafana: pre-built dashboard:
  • Agent memory gauge, token usage sparkline
  • Task queue depth
  • Up/down status for gateway, cron
  • Alerts table

Alerting

• Rules (PrometheusAlertmanager):
  • openclaw_agent_memory_bytes > 0.9 * budget → warn
  • openclaw_tasks_active > 10 → info (busy)
  • openclaw_cron_jobs{failed="true"} == 1 → critical
  • up{job="openclaw_gateway"} == 0 → critical
• Notifications: Discord webhook (since we already have Discord integration)

Deployment

Quick Start (Docker Compose)

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"

  node-exporter:
    image: prom/node-exporter
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'

volumes:
  prom_data:
  grafana_data:

Add OpenClaw agent to Prometheus scrape config.

Dashboard Panels

• Top row: Status summary (Gateway up? Agent healthy? Cron failed?)
• Middle:
  • Memory usage (workspace size vs budget)
  • Token budget utilization (percentage)
  • Active tasks count
  • Queue depth
• Bottom:
  • Cron job history (last 24h) with success/fail markers
  • External API latency (GitHub, RSS feed responses)
  • Disk usage (memory/, price_data/, etc.)

Alternatives

• Netdata: easier setup, less flexible; good for personal use. Could use instead of Prom+Grafana.
• Simple HTML + cron: Too basic; no alerting.
• Sentry: For exceptions only, not system metrics.

Decision

Given we already have node-exporter likely running, Prometheus + Grafana is natural. If too heavy, switch to Netdata later.

Estimated setup: 2 hours (including adding /metrics to OpenClaw agent).

Open Questions

• Should we include QuantPipe build size, DB size? Yes, as separate panels.
• Should we expose internal agent errors as metrics? Yes, counter openclaw_agent_errors_total{type="..."}.
• How to secure /metrics endpoint? Restrict to localhost; Prometheus scrapes via localhost. If remote, add basic auth.

Next Steps

1. Add /metrics endpoint to OpenClaw agent (if not present)
2. Define initial metric set (as above)
3. Create docker-compose.yml in infra/pulse/
4. Build Grafana dashboard (export as JSON)
5. Configure Alertmanager → Discord webhook
6. Document deployment steps

Let's implement after ToolResultCompactor to manage memory budget automatically.