WS-2868: harden host storage governance

scripts/storage-governance/README.md

@@ -0,0 +1,54 @@
+# Storage governance runbook
+
+This directory contains two deliberately small host-safety jobs:
+
+- `storage_guard.py` samples the host once per minute and applies configured
+  low-water admission controls. It never archives or deletes data.
+- `retention_worker.py` is the single owner of external-volume canaries,
+  workspace GC eligibility, and transactional archives.
+
+## Safe rollout
+
+Start the retention worker with both `archive_enabled` and `delete_source` set
+to `false`. In this mode every formal cron invocation checks the exact external
+volume UUID, verifies a nested canary tree by file count, byte count, entry
+metadata, and deterministic sample hashes, and writes only a GC dry-run report.
+
+GC eligibility is fail-closed. A workspace is listed as eligible only when its
+issue is `done` or `cancelled`, its matching run is terminal, the configured
+seven-day retention window has elapsed, children are terminal, no run/lease is
+active, no pin or open file exists, no recent write exists, and local identity
+agrees with the control plane. Filesystem traversal never follows symlinks;
+out-of-tree symlinks reject the candidate.
+
+After an operator approves a dry-run list, its one-time `approval_token` values
+must be put in `approved_candidates` before `archive_enabled` is enabled; keep
+`delete_source` false. A completed archive marker consumes the token so a later
+cron run cannot archive the same snapshot again. A transaction freezes the source manifest,
+copies to `.partial`, fsyncs and verifies it, atomically renames the archive,
+and writes `COMPLETE.json`, then re-hashes the committed payload. Automated
+source deletion is deliberately rejected until Multica exposes a producer-shared
+lease; filesystem isolation alone cannot close the open-file-descriptor race.
+
+The guard's minute path samples free space, swap, and daemon state before any
+recursive work. Directory/category scans run from a 15-minute cache after the
+breaker decision, so capacity attribution cannot delay low-water enforcement.
+
+## Formal cron lineage
+
+Use the same command for canary, GC audit, and archive. The environment marker
+prevents a normal manual invocation from being mistaken for a cron result:
+
+On macOS, `/usr/sbin/cron` normally lacks Full Disk Access to external media.
+The formal entry therefore runs a synchronous bridge. The bridge proves its
+own live `cron` ancestry, writes a fresh one-time token, starts the FDA-capable
+LaunchAgent, and waits for a token-matched receipt. The worker refuses a green
+result unless that bridge process and its cron parent are still alive:
+
+```cron
+*/15 * * * * /usr/bin/python3 /Users/example/.local/libexec/storage-governance/retention_cron_bridge.py --trigger /Users/example/.local/state/storage-governance/cron-trigger.json --receipt /Users/example/.local/state/storage-governance/cron-receipt.json --alert-log /Users/example/.local/state/storage-governance/retention-alerts.jsonl --config /Users/example/.local/libexec/storage-governance/retention-config.json
+```
+
+Keep the lock and report on the internal volume, and the archive root on the
+external volume. A lock collision or canary failure exits nonzero and records
+an alert; it never starts a second copy or removes a source.

scripts/storage-governance/com.multica.storage-guard.plist.example

@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>com.multica.storage-guard</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>/usr/bin/python3</string>
+    <string>-E</string>
+    <string>__INSTALL_ROOT__/storage_guard.py</string>
+    <string>--config</string>
+    <string>__INSTALL_ROOT__/config.json</string>
+  </array>
+  <key>EnvironmentVariables</key>
+  <dict>
+    <key>HOME</key>
+    <string>__HOME__</string>
+    <key>PATH</key>
+    <string>__HOME__/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+    <key>OPENAI_API_KEY</key>
+    <string></string>
+    <key>OPENAI_BASE_URL</key>
+    <string></string>
+  </dict>
+  <key>RunAtLoad</key>
+  <true/>
+  <key>StartInterval</key>
+  <integer>60</integer>
+  <key>ProcessType</key>
+  <string>Background</string>
+  <key>StandardOutPath</key>
+  <string>__HOME__/Library/Logs/storage-governance.guard.log</string>
+  <key>StandardErrorPath</key>
+  <string>__HOME__/Library/Logs/storage-governance.guard.err.log</string>
+</dict>
+</plist>

scripts/storage-governance/com.multica.storage-retention.plist.example

@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>com.multica.storage-retention</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>/usr/bin/osascript</string>
+    <string>__INSTALL_ROOT__/retention-cron.scpt</string>
+  </array>
+  <key>RunAtLoad</key>
+  <false/>
+  <key>ProcessType</key>
+  <string>Background</string>
+  <key>StandardOutPath</key>
+  <string>__HOME__/Library/Logs/multica-storage-retention.launchd.log</string>
+  <key>StandardErrorPath</key>
+  <string>__HOME__/Library/Logs/multica-storage-retention.launchd.err.log</string>
+</dict>
+</plist>

scripts/storage-governance/config.example.json

@@ -0,0 +1,54 @@
+{
+  "internal_path": "/",
+  "external_path": "/Volumes/MacMini-HotSSD",
+  "level1_free_gib": 25,
+  "level1_clear_gib": 28,
+  "level2_free_gib": 18,
+  "level2_clear_gib": 21,
+  "external_min_free_gib": 100,
+  "safety_floor_gib": 25,
+  "burst_reserve_gib": 10,
+  "minimum_observation_hours": 48,
+  "maximum_observation_hours": 72,
+  "expected_interval_seconds": 60,
+  "minimum_sample_coverage": 0.8,
+  "growth_scan_interval_seconds": 900,
+  "growth_scan_budget_seconds": 5,
+  "required_field_max_gap_seconds": 1800,
+  "retention_report_max_age_seconds": 1800,
+  "shadow_runs_path": "/Users/example/path/to/m0-shadow-observer/data/runs",
+  "workspace_roots": [
+    "/Users/example/multica_workspaces",
+    "/Users/example/multica_workspaces_desktop-api.multica.ai"
+  ],
+  "retention_report_path": "/Users/example/.local/state/storage-governance/gc-dry-run.json",
+  "retention_days": 7,
+  "cursor_path": "/Users/example/Library/Application Support/Cursor",
+  "logs_paths": [
+    "/Users/example/.multica",
+    "/Users/example/Library/Logs"
+  ],
+  "required_growth_fields": [
+    "shadow_runs_bytes",
+    "workspace_total_bytes",
+    "workspace_inflight_bytes",
+    "workspace_gc_eligible_bytes",
+    "workspace_gc_backlog_bytes",
+    "workspace_unclassified_bytes",
+    "cursor_bytes",
+    "logs_bytes",
+    "swap_used_bytes",
+    "external_free_bytes"
+  ],
+  "observer_labels": [
+    "ai.multica.ws2512.m0-shadow-observer"
+  ],
+  "nonproduction_launchagents": [],
+  "lark_open_id": "",
+  "alert_cooldown_seconds": 3600,
+  "legacy_sigstop_fallback": false,
+  "state_path": "/Users/example/.local/state/storage-governance/guard-state.json",
+  "growth_cache_path": "/Users/example/.local/state/storage-governance/growth-metrics-cache.json",
+  "metrics_path": "/Users/example/.local/state/storage-governance/host-samples.jsonl",
+  "capacity_report_path": "/Users/example/.local/state/storage-governance/capacity-report.json"
+}

scripts/storage-governance/retention-config.example.json

@@ -0,0 +1,23 @@
+{
+  "external_path": "/Volumes/MacMini-HotSSD",
+  "canary_root": "/Volumes/MacMini-HotSSD/archive/multica-archive",
+  "external_volume_uuid": "00000000-0000-0000-0000-000000000000",
+  "external_min_free_gib": 100,
+  "archive_root": "/Volumes/MacMini-HotSSD/MulticaArchive/workspaces",
+  "workspace_roots": [
+    "/Users/example/multica_workspaces",
+    "/Users/example/multica_workspaces_desktop-api.multica.ai"
+  ],
+  "retention_days": 7,
+  "recent_write_seconds": 86400,
+  "archive_enabled": false,
+  "approved_candidates": [],
+  "delete_source": false,
+  "require_cron_lineage": true,
+  "cron_bridge_trigger_path": "/Users/example/.local/state/storage-governance/cron-trigger.json",
+  "cron_bridge_receipt_path": "/Users/example/.local/state/storage-governance/cron-receipt.json",
+  "lock_path": "/Users/example/.local/state/storage-governance/retention.lock",
+  "report_path": "/Users/example/.local/state/storage-governance/gc-dry-run.json",
+  "alert_log_path": "/Users/example/.local/state/storage-governance/retention-alerts.jsonl",
+  "lark_open_id": ""
+}

scripts/storage-governance/retention-cron.applescript.example

@@ -0,0 +1,4 @@
+set commandText to "/usr/bin/env -i HOME=__HOME__ PATH=__HOME__/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin MULTICA_STORAGE_CRON_BRIDGE=1 OPENAI_API_KEY='' OPENAI_BASE_URL='' /usr/bin/python3 __INSTALL_ROOT__/retention_worker.py --config __INSTALL_ROOT__/retention-config.json >> __HOME__/Library/Logs/multica-storage-retention.log 2>&1"
+with timeout of 7200 seconds
+  do shell script commandText
+end timeout

scripts/storage-governance/retention_cron_bridge.py

@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""Synchronous cron-to-LaunchAgent bridge for macOS Full Disk Access."""
+
+from __future__ import annotations
+
+import argparse
+import fcntl
+import json
+import os
+import subprocess
+import tempfile
+import time
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+from retention_worker import atomic_write_failure_report, send_alert
+
+
+class SingleInstanceLock:
+    def __init__(self, path: Path):
+        self.path = path
+        self.handle = None
+
+    def __enter__(self) -> "SingleInstanceLock":
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        self.handle = self.path.open("a+")
+        fcntl.flock(self.handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+        return self
+
+    def __exit__(self, exc_type: object, exc: object, traceback: object) -> None:
+        if self.handle is not None:
+            fcntl.flock(self.handle.fileno(), fcntl.LOCK_UN)
+            self.handle.close()
+
+
+def launchctl_kickstart_command(label: str, *, uid: int) -> list[str]:
+    return ["/bin/launchctl", "kickstart", "gui/%d/%s" % (uid, label)]
+
+
+def receipt_exit_code(value: dict, token: str) -> Optional[int]:
+    if value.get("token") != token:
+        return None
+    if value.get("status") == "green":
+        return 0
+    if value.get("status") == "red":
+        return 1
+    return None
+
+
+def record_lock_collision(path: Path) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("a", encoding="utf-8") as handle:
+        handle.write(
+            json.dumps(
+                {
+                    "recorded_at": datetime.now(timezone.utc).isoformat(),
+                    "status": "locked",
+                    "message": "previous bridge is still running; skipped overlapping retention launch",
+                },
+                sort_keys=True,
+            )
+            + "\n"
+        )
+        handle.flush()
+        os.fsync(handle.fileno())
+
+
+def report_bridge_failure(config_path: Optional[str], message: str) -> None:
+    if not config_path:
+        return
+    try:
+        config = json.loads(Path(config_path).read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return
+    for action in (
+        lambda: atomic_write_failure_report(config, message),
+        lambda: send_alert(config, message),
+    ):
+        try:
+            action()
+        except Exception:
+            pass
+
+
+def atomic_write(path: Path, value: dict) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    descriptor, temporary = tempfile.mkstemp(prefix=".%s." % path.name, dir=str(path.parent))
+    try:
+        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+            json.dump(value, handle, sort_keys=True)
+            handle.write("\n")
+            handle.flush()
+            os.fsync(handle.fileno())
+        os.replace(temporary, path)
+    finally:
+        try:
+            os.unlink(temporary)
+        except FileNotFoundError:
+            pass
+
+
+def ancestry(pid: int, limit: int = 8) -> list[dict]:
+    values = []
+    for _ in range(limit):
+        parent_result = subprocess.run(
+            ["/bin/ps", "-p", str(pid), "-o", "ppid="], capture_output=True, text=True, check=False
+        )
+        command_result = subprocess.run(
+            ["/bin/ps", "-p", str(pid), "-o", "comm="], capture_output=True, text=True, check=False
+        )
+        if parent_result.returncode != 0 or command_result.returncode != 0:
+            break
+        parent = int(parent_result.stdout.strip())
+        command = command_result.stdout.strip()
+        values.append({"pid": pid, "parent_pid": parent, "command": command})
+        if parent <= 1 or parent == pid:
+            break
+        pid = parent
+    return values
+
+
+def run_bridge(args: argparse.Namespace) -> int:
+    lineage = ancestry(os.getpid())
+    if not any(Path(str(item["command"])).name == "cron" for item in lineage):
+        raise SystemExit("retention bridge refuses non-cron ancestry")
+    token = uuid.uuid4().hex
+    trigger = Path(args.trigger)
+    receipt = Path(args.receipt)
+    atomic_write(
+        trigger,
+        {
+            "schema": "multica.storage-cron-trigger.v1",
+            "token": token,
+            "created_at": datetime.now(timezone.utc).isoformat(),
+            "bridge_pid": os.getpid(),
+            "cron_ancestry": lineage,
+        },
+    )
+    process = subprocess.run(
+        launchctl_kickstart_command(args.label, uid=os.getuid()),
+        capture_output=True,
+        text=True,
+        check=False,
+    )
+    if process.returncode != 0:
+        raise SystemExit("launchctl kickstart failed: " + (process.stderr or process.stdout).strip())
+    deadline = time.monotonic() + args.timeout
+    while time.monotonic() < deadline:
+        try:
+            value = json.loads(receipt.read_text(encoding="utf-8"))
+        except (FileNotFoundError, OSError, json.JSONDecodeError):
+            time.sleep(2)
+            continue
+        result = receipt_exit_code(value, token)
+        if result is not None:
+            return result
+        time.sleep(2)
+    raise SystemExit("retention worker receipt timeout")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--trigger", required=True)
+    parser.add_argument("--receipt", required=True)
+    parser.add_argument("--label", default="com.multica.storage-retention")
+    parser.add_argument("--timeout", type=int, default=7200)
+    parser.add_argument("--lock")
+    parser.add_argument("--alert-log")
+    parser.add_argument("--config")
+    args = parser.parse_args()
+    lock_path = Path(args.lock) if args.lock else Path(args.trigger).with_name("retention-cron-bridge.lock")
+    try:
+        with SingleInstanceLock(lock_path):
+            try:
+                return run_bridge(args)
+            except SystemExit as error:
+                report_bridge_failure(args.config, "storage retention cron bridge failed: %s" % error)
+                raise
+    except BlockingIOError:
+        alert_path = Path(args.alert_log) if args.alert_log else Path(args.trigger).with_name("retention-alerts.jsonl")
+        record_lock_collision(alert_path)
+        report_bridge_failure(args.config, "storage retention cron bridge skipped: previous bridge is still running")
+        return 75
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())