Workspace file management substrate — feedback wanted

Summary

Five substrates lifting workspace file management into pluggable contracts: universal oc:// addressing across md / jsonc / jsonl / yaml; a kind-agnostic lint framework; a fixer-spec adapter that extends the existing openclaw doctor; a generalized LKG store with FS + git backends and operator-pinned labels; a content-addressable PolicyIR. The train converts recurring per-feature ad-hoc parsing / walking / recovery — the surface behind a lot of ongoing pain (#54310, #76619, #14526, #40245, #70407, #76042 and friends) — into shared substrates plugin authors can build against. Looking for directional feedback before opening any PR.

Motivation — the E2E plugin-author story

Today, a plugin author who wants to ship a tool with policy lives across half a dozen ad-hoc surfaces. They write a TOOLS.md entry in their plugin, but there's no shared validator — the format they pick may or may not be what the runtime expects. They might author a JSON file alongside, but writes through JSON.stringify(_, null, 2) eat their comments and key ordering (#65623, #62438). Their config might validate today and crash after the next upgrade because the schema migrated under them (#57567, #70407, #65786). Their session logs might silently break (#13700 cluster). Different consumers of the same plugin's policy disagree on what it permits because there's no canonical hash. Audit envelopes can't say "decision X was made against THIS policy" because there's no notion of "the active policy bytes." After an upgrade goes bad, there's no atomic way to roll back a cohort of files (#14526). Each plugin re-invents discovery, addressing, validation, fix, recovery, and auditability for itself.

The substrate train fixes all of that with one mental model. A plugin author's E2E flow becomes:

1. Declare — author edits TOOLS.md (markdown — the easy editing surface). Address any field at any time via oc://TOOLS.md/Tools/[id=my-tool]/sensitivity.
2. Lint — openclaw pinch run flags shape problems via the plugin's registered LintRules. Diagnostics carry oc:// paths, so the editor jumps straight to the offending node.
3. Fix — openclaw doctor with the plugin's OcPathFixerSpec repairs the flag. Three tiers — additive (default), mutating (operator opts in), regenerative (explicit flag) — tell operators exactly what's about to change.
4. Generate — openclaw policy generate extracts a content-addressable PolicyIR from the MD sources. Same shape → same policyId. Fleet audits compare hashes.
5. Enforce — at runtime, api.registerTrustedToolPolicy evaluates against the PolicyIR. The decision context carries ctx.lkgFingerprint of the active policy bytes, so audit envelopes can prove WHICH policy was in force at decision time.
6. Recover — openclaw cage promote --label pre-upgrade-X pins the cohort before any risky operation. If the upgrade goes bad (#57567 / #70407 / #76042), openclaw cage rollback --label pre-upgrade-X restores every tracked file atomically.
7. Drift-detect — openclaw policy check catches "policy.jsonc moved on, sources moved further" between regens. No more silent governance drift after an upgrade auto-migrates schema (#74395).

Same five primitives (path, pinch, doctor, cage, policy) cover every step. A plugin author learns one mental model and applies it everywhere their plugin touches workspace state. Today's per-plugin reinvention (claws alone has parallel implementations totaling ~11k LoC across openclaw-policy/, claws-sdk/src/governance/, and claws-sdk/src/compliance/) collapses into a single shared substrate set.

1. oc-path — universal addressing (CLI: openclaw path)

One oc:// URI scheme + tagged-union AST + universal verbs (resolveOcPath, setOcPath, findOcPaths) work across md / jsonc / jsonl / yaml. Per-kind parsers and emitters round-trip byte-for-byte — no more comment-eating on JSON.stringify(_, null, 2) writes (#65623, #62438), no more silent frontmatter loss on parse (#54310, #54311, #57091, #69475). Substrate-level redaction-sentinel guard at every emit* boundary closes the __OPENCLAW_REDACTED__-literal-on-disk regression. Workspace manifest builder classifies every workspace file by canonical role.

oc://AGENTS.md/Tools/-1/risk             # last tool's risk field (positional)
oc://session.jsonl/L42/event             # event field on line 42 (jsonl line addressing)
oc://policy.jsonc/tools/[id=send-email]/sensitivity   # predicate addressing
oc://*.lobster/steps/*/{command,run}     # wildcard + segment union for find verb
oc://AGENTS.md/Tools/+/id                # INSERTION POINT — append a new tool entry
oc://policy.jsonc/denyRules/+0           # insert at index 0 (prepend)

The CLI is the eval/REPL probe — like jq for OpenClaw workspaces:

$ openclaw path resolve 'oc://AGENTS.md/Tools/-1/risk'
{ "kind": "leaf", "valueText": "R3", "line": 14 }

$ openclaw path set 'oc://AGENTS.md/Tools/+/id' 'my-new-tool' --dry-run
# prints would-be bytes; --no-dry-run writes through emitMd

$ openclaw path find 'oc://*.lobster/steps/*/command'
# enumerates every matching node across every *.lobster file

2. pinch — kind-agnostic lint (CLI: openclaw pinch)

One LintRule shape across all four kinds. Plugin authors register at module-init via api.registerLintRule(rule). Every diagnostic carries an oc:// path so editors / doctor / fixers deep-link to the offending node. 23 starter rules in v0; 21 use only findOcPaths / resolveOcPath / match.line — kind-narrow logic is the documented exception. Each starter rule is sourced from a real user-filed issue: openclaw/lobster #25, #26, #41 (shell-tool collision), #76, #77 (duplicate step ids); #54310, #54311, #57091, #69475 (md frontmatter cluster); #76619 (config dup keys), #74462 (config missing keys), #65623, #62438 (secrets-as-literals); #13700 cluster (session log issues).

import { findOcPaths, parseOcPath } from '@openclaw/oc-path';
import type { LintRule } from '@openclaw/oc-lint';

export const stepShellToolCollision: LintRule = {
  id: 'lobster-yaml-starter-v0/step/shell-tool-collision',
  severity: 'error',
  description: 'step `command:` references in-process tool name; will fail at shell-exec',
  appliesTo: '*.lobster',  // closes openclaw/lobster #25, #26, #41
  check(ctx) {
    return findOcPaths(ctx.ast, parseOcPath(
      `oc://${ctx.fileName}/steps/*/{command,run}`,
    ))
      .filter(({ match }) =>
        match.kind === 'leaf' &&
        IN_PROCESS_TOOLS.includes(match.valueText.trim().split(/\s+/)[0]),
      )
      .map(({ path, match }) => ({
        message: `step[${path.item}].${path.field} references in-process tool`,
        ocPath: `oc://${ctx.fileName}/steps/${path.item}/${path.field}`,
        line: match.line,
      }));
  },
};

$ openclaw pinch run                     # all registered rules over the workspace
$ openclaw pinch list-rules              # enumerate registered rules
$ openclaw pinch lint <file...>          # lint specific files

3. oc-doctor — fixer-spec adapter (extends existing openclaw doctor)

OcPathFixerSpec contract with three-tier classification: additive (default — only inserts; safe), mutating (operator opts in), regenerative (highest blast radius — --fix-regenerative flag). Plugins register via registerOcPathFixer. Each fixer pairs one-for-one with a pinch starter rule — same issue refs (#54310 cluster, #76619, #74462, openclaw/lobster #25/#41/#76/#77, #13700 cluster). What lint flags, doctor fixes. siblingFiles access supports cross-file fixers natively. The substrate slots into the existing openclaw doctor pipeline via src/plugins/doctor-contract-registry.ts — no new top-level CLI verb.

import { setOcPath, parseOcPath, emitYaml } from '@openclaw/oc-path';
import type { OcPathFixerSpec } from '@openclaw/oc-doctor';

export const stepSwapShellToPipeline: OcPathFixerSpec = {
  id: 'lobster-yaml-starter-v0/step/swap-shell-to-pipeline',
  severity: 'error',
  tier: 'mutating',
  appliesTo: '*.lobster',  // pairs with stepShellToolCollision lint rule above
  async detect(ctx) { /* same shape as the paired lint rule */ },
  async fix({ ast, fileName }) {
    // for each colliding step, rewrite field key from `command` → `pipeline`
    return emitYaml(/* mutated AST */);
  },
};

4. lkg-cage — LKG store + operator-pinned labels (CLI: openclaw cage)

Every LKGTracker declares a path + parser + validator. The store observes (read → parse → validate → react), promotes valid bytes to last-known-good, recovers from .lkg companions when bytes go bad, and lets operators pin labeled cohorts for atomic rollback. FS-backed and git-backed backends share the same LKGStore interface — git impl uses lkg/<name> git tags; FS uses <path>.lkg.label.<name> companion files. Same contract; native semantics per backend.

Coverage: closes #14526 (proposal: safer self-update with auto-rollback) — direct fit; closes #40245 (multi-agent shared workspace) via the git backend's three-way merge; generalizes #70528 (recoverConfigFromLastKnownGood); enables session-checkpointing cluster #13700 / #17211 / #58028 / #60864. Concrete real-world failure modes the labels feature addresses: #57567 (config migration failure during upgrade), #65358 (memory/session state broke across update), #74300 (version mismatch silent state incompatibility), #70407 (dreaming.storage schema migration breaks CLI on upgrade), #65786 (post-upgrade Feishu config-invalid), #76042 ("upgrade or clean install is VERY long or broken"), #65177 (openclaw doctor --fix doesn't migrate post-upgrade), #74395 (auto-migration silently switches default model).

# Before upgrading 2026.4.21 → 2026.4.22
$ openclaw cage promote --label pre-upgrade-2026.4.22
{ "ok": true, "promoted": 12, "label": "pre-upgrade-2026.4.22" }

$ openclaw update self
$ openclaw doctor   # new version flags drift — e.g. dreaming.storage schema (#70407)

# Roll back the cohort atomically — verifies every companion before any write
$ openclaw cage rollback --label pre-upgrade-2026.4.22
{ "ok": true, "label": "pre-upgrade-2026.4.22", "restoredCount": 12 }

# Once a future fixed upgrade sticks, free disk
$ openclaw cage delete-label pre-upgrade-2026.4.22

5. policy — content-addressable PolicyIR + LKG anchoring (CLI: openclaw policy)

PolicyIR is a content-addressable artifact (sha256 over RFC 8785 JCS canonical-shape bytes — two policies with the same SHAPE have the same policyId, useful for fleet audits). Generators are pluggable; the reference markdown generator extracts ToolSpecs from TOOLS.md ### name # R<n>, CAPS, sensitivity:level headings and DenyRules from SOUL.md. Policy decisions ride the existing api.registerTrustedToolPolicy({ id, evaluate }) slot with ctx.lkgFingerprint of the active policy bytes — decision audit envelopes can prove WHICH policy was active at decision time. Closes the "what policy was active when decision X happened?" forensics gap (relevant to #76042, #65786, #74395 post-upgrade scenarios where governance silently drifted).

A plugin that contributes a tool, plus its lint rule and doctor fixer:

<!-- In the plugin's TOOLS.md -->
### my-plugin/send-fax # R5, COMMUNICATE, IRREVERSIBLE_EXTERNAL, sensitivity:restricted
Send a fax. R5 because outbound communications are irreversible.

// Plugin's lint pack
api.registerLintRule({
  id: 'my-plugin-v0/tools/fax-needs-restricted',
  severity: 'error',
  appliesTo: 'TOOLS.md',
  check(ctx) {
    return findOcPaths(ctx.ast, parseOcPath(
      'oc://TOOLS.md/Tools/[id=my-plugin/send-fax]/sensitivity',
    ))
      .filter(m => m.match.kind === 'leaf' && m.match.valueText !== 'restricted')
      .map(m => ({
        message: 'must be sensitivity:restricted',
        ocPath: 'oc://TOOLS.md/Tools/[id=my-plugin/send-fax]/sensitivity',
        line: m.match.line,
      }));
  },
});

// Plugin's doctor pack — pairs one-for-one with the lint rule
api.registerOcPathFixer({
  id: 'my-plugin-v0/tools/snap-fax-sensitivity',
  severity: 'error',
  tier: 'mutating',
  appliesTo: 'TOOLS.md',
  async detect(ctx) { /* same shape */ },
  async fix({ ast }) {
    return emitMd(setOcPath(ast, parseOcPath(
      'oc://TOOLS.md/Tools/[id=my-plugin/send-fax]/sensitivity',
    ), 'restricted').ast);
  },
});

$ openclaw policy generate                         # produce policy.jsonc from MD sources
$ openclaw policy check                            # detect drift between policy.jsonc and current MD
$ openclaw policy diff --against <policyId>        # show shape delta vs a known policy
$ openclaw policy evaluate --tool send-email --args '...'   # dry-run a decision

The shape: every plugin contributes (a) tools/policy declarations in MD, (b) lint rules that flag misconfiguration, (c) doctor fixers paired one-for-one with the rules, (d) optionally LKG trackers if the plugin owns recoverable workspace state. One mental model across all five substrates.

Branches on the fork (giodl73-repo/openclaw)

5 stacked branches, each a self-contained substrate as a workspace package — kept as packages/<name>/ for review ease (one isolatable unit per PR):

• substrate/oc-paths — compare/main...substrate/oc-paths
• substrate/pinch — compare/substrate/oc-paths...substrate/pinch
• substrate/oc-doctor — compare/substrate/pinch...substrate/oc-doctor
• substrate/lkg-cage (FS + git backends + labels feature) — compare/substrate/oc-doctor...substrate/lkg-cage
• substrate/policy — compare/substrate/lkg-cage...substrate/policy

Showcase tip: tree/substrate/policy (all 5 working together)

Each branch carries: spec at docs/reference/<topic>-design.md, full source, register.<verb>.ts CLI integration. ~2,046 tests passing across the train.

In parallel I'm putting together a src/ + extensions/ branch (substrates as src/<name>/ directories, lkg-git as extensions/lkg-git/) to validate that placement against the same set; happy to take that as the canonical shape if you prefer.

Feedback wanted

Open to feedback at every layer — substrate names, CLI verb names (path / pinch / cage / policy), placement (packages/ vs src/ vs extensions/), what gets bundled together vs split apart, contract shapes, pack-id conventions, anything else. Direction-giving feedback works ("we want X" / "rename Y to Z" / "split A out of B") — we can iterate.

Specific questions where I'd most value a maintainer view:

• PR sequencing: as a stacked train (one PR per substrate, ~5 PRs strict-linear oc-path → others), or some other shape?
• Scope per PR: each substrate is ~3-12k LoC. Larger than your typical merged feat. Split each further, or ship as scoped?
• Labels feature with lkg-cage (closes #14526): in the same PR as the LKG contract, or separate follow-up?

AI-assisted disclosure

Drafted with Claude Opus 4.7 (1M context) by [email protected] (Microsoft).