diff --git a/docs/plans/2026-08-04-st1-electron-updater-audit-design.md b/docs/plans/2026-08-04-st1-electron-updater-audit-design.md
new file mode 100644
index 00000000000..acb4b8dcefc
--- /dev/null
+++ b/docs/plans/2026-08-04-st1-electron-updater-audit-design.md
@@ -0,0 +1,69 @@
+# ST-1 Electron Updater Audit Design
+
+## Context
+
+The host storage constitution requires a monthly audit of Electron updater
+residue after Tencent Meeting left more than 15 GiB of stale update packages.
+The existing storage-governance line already owns a cron-to-FDA bridge, a
+single-instance lock, atomic JSON evidence, and Lark alert delivery. The audit
+must extend that line instead of creating a second garbage-collection owner.
+
+## Considered approaches
+
+1. Extend `retention_worker.py` with a monthly-gated, read-only audit. This
+ reuses the existing scheduler, lock, evidence writer, and alert channel while
+ keeping updater scans out of the minute-level breaker hot path. This is the
+ selected approach.
+2. Add a dedicated monthly LaunchAgent and script. This is easy to isolate, but
+ duplicates scheduling, locking, logging, and alert ownership.
+3. Add updater scans to `storage_guard.py`. This gives fast visibility, but
+ recursive size scans would make the low-water breaker slower and less
+ predictable.
+
+## Architecture
+
+`retention_worker.py` gains a bounded scanner driven by an
+`electron_updater_audit` config object. Each configured glob must resolve below
+the configured home directory. Matches are de-duplicated, symlinks are rejected,
+and directory traversal never follows symlinks. The scanner records bytes, file
+count, newest and oldest mtimes, and age for each match; it never moves or
+deletes data.
+
+The formal retention run checks for the current Shanghai calendar month's
+evidence before touching the external archive volume. On or after the configured
+day of month, a missing report triggers the scan. This ordering lets ST-1 leave
+evidence even if the external archive is unavailable. A manual audit-only CLI
+mode forces the same scanner for commissioning and incident response without
+claiming cron lineage.
+
+The audit writes
+`~/.org/metrics/st1-electron-updater-audit-YYYY-MM.json` atomically. `green`
+means the scan completed and no configured size or stale-residue threshold was
+crossed; `attention` means operator review is warranted; `red` means the scan
+could not provide complete evidence. Formal `attention` and `red` results use
+the existing alert channel. A `red` result also fails the retention invocation
+closed, while `attention` does not block unrelated canary or GC reporting.
+
+## Initial scan surface
+
+The host config covers the known sharp edges without a full-disk search:
+
+- sandboxed `UpdatePackages` roots (Tencent Meeting pattern);
+- `update.noindex`, `update_downloading`, and `Software Update` directories;
+- Electron `*-updater` and scoped updater caches;
+- Squirrel `*.ShipIt` caches;
+- MiniMax Agent `hot-update` payloads;
+- pending/update ZIP files under Multica daemon state.
+
+Thresholds are configuration, initially 5 GiB total, 1 GiB per candidate, and
+45 days stale for candidates of at least 100 MiB. The audit is observational;
+crossing a threshold does not authorize cleanup.
+
+## Verification
+
+Unit tests use temporary directory trees to prove discovery, de-duplication,
+byte counts, no symlink following, threshold classification, month/day gating,
+and forced audit behavior. Commissioning runs the audit-only CLI against the
+real host configuration, validates the JSON schema and current-month mtime, and
+then verifies the deployed worker and config match the reviewed repository
+files.
diff --git a/docs/superpowers/plans/2026-08-04-st1-electron-updater-audit.md b/docs/superpowers/plans/2026-08-04-st1-electron-updater-audit.md
new file mode 100644
index 00000000000..61def65f49f
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-04-st1-electron-updater-audit.md
@@ -0,0 +1,180 @@
+# ST-1 Electron Updater Audit Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add a non-destructive, machine-verifiable monthly ST-1 audit of Electron updater residue to the existing storage-governance worker.
+
+**Architecture:** A bounded scanner in `retention_worker.py` reads explicit host-relative globs, emits one atomic report per Shanghai calendar month, and is invoked before external-volume checks by the existing formal cron worker. A manual audit-only mode commissions the same code path; no second scheduler or cleanup owner is introduced.
+
+**Tech Stack:** Python 3 standard library (`glob`, `os`, `pathlib`, `zoneinfo`, `unittest`), JSON configuration, existing cron-to-FDA storage-governance bridge.
+
+---
+
+### Task 1: Specify scanner behavior with failing tests
+
+**Files:**
+- Modify: `scripts/storage-governance/test_retention_worker.py`
+- Test: `scripts/storage-governance/test_retention_worker.py`
+
+- [ ] **Step 1: Add imports and fixtures**
+
+Import `audit_electron_updaters` and `maybe_run_electron_updater_audit`. Build a
+temporary home with an updater directory, a pending ZIP, and a symlink to an
+out-of-root payload.
+
+- [ ] **Step 2: Assert discovery and safety behavior**
+
+Assert that the report de-duplicates overlapping globs, sums regular-file
+payload, reports the real candidates, records the symlink as an error without
+following it, and never changes source paths.
+
+- [ ] **Step 3: Assert threshold and monthly-gate behavior**
+
+Use an injected UTC time whose Shanghai date is before and after the configured
+day. Assert `skipped` before the day, a report on/after the day, `skipped` when
+valid current-month evidence exists, and a rerun with `force=True`.
+
+- [ ] **Step 4: Run the focused tests and verify RED**
+
+Run: `python3 -m unittest scripts.storage-governance.test_retention_worker.ElectronUpdaterAuditTest -v`
+
+Expected: import failure because the audit functions do not exist.
+
+### Task 2: Implement the scanner and monthly evidence gate
+
+**Files:**
+- Modify: `scripts/storage-governance/retention_worker.py`
+- Test: `scripts/storage-governance/test_retention_worker.py`
+
+- [ ] **Step 1: Add the audit functions**
+
+Implement:
+
+```python
+def audit_electron_updaters(
+ config: Dict[str, Any],
+ *,
+ now: Callable[[], datetime] = utc_now,
+ invocation_source: str = "manual",
+) -> Dict[str, Any]: ...
+
+def maybe_run_electron_updater_audit(
+ config: Dict[str, Any],
+ *,
+ now: Callable[[], datetime] = utc_now,
+ force: bool = False,
+ invocation_source: str = "manual",
+) -> Dict[str, Any]: ...
+```
+
+Resolve the configured timezone with `ZoneInfo`, validate every match beneath
+`home_path`, scan without following symlinks, atomically write the month-named
+JSON report, and classify `green`, `attention`, or `red` from scan errors and
+configured thresholds.
+
+- [ ] **Step 2: Run the focused tests and verify GREEN**
+
+Run: `python3 -m unittest scripts.storage-governance.test_retention_worker.ElectronUpdaterAuditTest -v`
+
+Expected: all `ElectronUpdaterAuditTest` cases pass.
+
+- [ ] **Step 3: Run the complete storage-governance tests**
+
+Run: `python3 -m unittest discover -s scripts/storage-governance -p 'test_*.py' -v`
+
+Expected: all tests pass with zero failures.
+
+### Task 3: Integrate scheduling and audit-only CLI
+
+**Files:**
+- Modify: `scripts/storage-governance/retention_worker.py`
+- Modify: `scripts/storage-governance/test_retention_worker.py`
+
+- [ ] **Step 1: Add failing integration tests**
+
+Patch external-volume construction and assert `run_worker` invokes the monthly
+audit before external-volume checks. Add an argument-parser test or subprocess
+test proving `--electron-audit-only` does not require cron lineage or external
+storage.
+
+- [ ] **Step 2: Verify RED**
+
+Run the two new test names directly and confirm they fail because the formal and
+CLI integration is absent.
+
+- [ ] **Step 3: Implement the integration**
+
+Call `maybe_run_electron_updater_audit` immediately after formal cron lineage
+verification, attach its summary to the retention report, alert once when a
+formal scan returns `attention`, and fail closed on `red`. Add
+`--electron-audit-only` to `main`; this mode takes the existing lock, forces the
+audit, prints a compact JSON result, and exits nonzero only for `red`.
+
+- [ ] **Step 4: Verify GREEN**
+
+Run the focused tests, then the complete storage-governance suite.
+
+### Task 4: Configure, document, deploy, and commission
+
+**Files:**
+- Modify: `scripts/storage-governance/retention-config.example.json`
+- Modify: `scripts/storage-governance/README.md`
+- Modify: `/Users/tangyuanjc/.local/libexec/storage-governance/retention_worker.py` (deployed copy)
+- Modify: `/Users/tangyuanjc/.local/libexec/storage-governance/retention-config.json` (deployed config)
+- Create: `/Users/tangyuanjc/.org/metrics/st1-electron-updater-audit-2026-08.json` (runtime evidence)
+
+- [ ] **Step 1: Add the host-neutral example config**
+
+Document `enabled`, `timezone`, `day_of_month`, `home_path`, `report_dir`,
+bounded `patterns`, `warn_total_gib`, `warn_candidate_gib`, `stale_days`, and
+`stale_min_mib`.
+
+- [ ] **Step 2: Update the runbook**
+
+State that ST-1 is monthly-gated inside the formal retention owner, is strictly
+read-only, produces month-named evidence, and can be commissioned with
+`--electron-audit-only`.
+
+- [ ] **Step 3: Deploy reviewed files and config**
+
+Copy the reviewed worker to the existing libexec location, update only the
+`electron_updater_audit` config object, preserve executable mode, and validate
+both JSON files.
+
+- [ ] **Step 4: Run the real audit**
+
+Run:
+
+```bash
+/usr/bin/python3 /Users/tangyuanjc/.local/libexec/storage-governance/retention_worker.py \
+ --config /Users/tangyuanjc/.local/libexec/storage-governance/retention-config.json \
+ --electron-audit-only
+```
+
+Expected: exit 0 with `green` or `attention`, plus a current-month evidence
+path. Validate the report with `python3 -m json.tool`, confirm its mtime is in
+the current Shanghai month, and confirm no candidate has `action: deleted` or
+`action: moved` because the schema exposes observations only.
+
+### Task 5: Final verification and delivery
+
+**Files:**
+- Verify all modified files
+
+- [ ] **Step 1: Run fresh verification**
+
+Run the full Python suite, `python3 -m py_compile` on all storage-governance
+Python files, `python3 -m json.tool` on example and deployed configs and the
+monthly report, `cmp` on repository/deployed worker, and `git diff --check`.
+
+- [ ] **Step 2: Commit and update the open storage-governance PR**
+
+Create atomic conventional commits, push them to the existing PR #6281 head,
+and add `Closes WS-3010` to that PR body so this follow-up is linked and closes
+on merge.
+
+- [ ] **Step 3: Deliver Multica evidence**
+
+Create the required Allen closing sub-issue, post exactly one concise WS-3010
+result comment with the audit command, exit code, raw compact output, report
+summary, test count, and PR URL, then move WS-3010 to `in_review`.
diff --git a/scripts/storage-governance/README.md b/scripts/storage-governance/README.md
new file mode 100644
index 00000000000..a9bcac424c9
--- /dev/null
+++ b/scripts/storage-governance/README.md
@@ -0,0 +1,83 @@
+# Storage governance runbook
+
+This directory contains two deliberately small host-safety jobs:
+
+- `storage_guard.py` samples the host once per minute and applies configured
+ low-water admission controls. It never archives or deletes data.
+- `retention_worker.py` is the single owner of external-volume canaries,
+ workspace GC eligibility, transactional archives, and the monthly ST-1
+ Electron updater residue audit.
+
+## Safe rollout
+
+Start the retention worker with both `archive_enabled` and `delete_source` set
+to `false`. In this mode every formal cron invocation checks the exact external
+volume UUID, verifies a nested canary tree by file count, byte count, entry
+metadata, and deterministic sample hashes, and writes only a GC dry-run report.
+
+GC eligibility is fail-closed. A workspace is listed as eligible only when its
+issue is `done` or `cancelled`, its matching run is terminal, the configured
+seven-day retention window has elapsed, children are terminal, no run/lease is
+active, no pin or open file exists, no recent write exists, and local identity
+agrees with the control plane. Filesystem traversal never follows symlinks;
+out-of-tree symlinks reject the candidate.
+
+After an operator approves a dry-run list, its one-time `approval_token` values
+must be put in `approved_candidates` before `archive_enabled` is enabled; keep
+`delete_source` false. A completed archive marker consumes the token so a later
+cron run cannot archive the same snapshot again. A transaction freezes the source manifest,
+copies to `.partial`, fsyncs and verifies it, atomically renames the archive,
+and writes `COMPLETE.json`, then re-hashes the committed payload. Automated
+source deletion is deliberately rejected until Multica exposes a producer-shared
+lease; filesystem isolation alone cannot close the open-file-descriptor race.
+
+The guard's minute path samples free space, swap, and daemon state before any
+recursive work. Directory/category scans run from a 15-minute cache after the
+breaker decision, so capacity attribution cannot delay low-water enforcement.
+
+## ST-1 Electron updater audit
+
+The retention worker also owns the read-only monthly audit for Electron updater
+residue. On or after `electron_updater_audit.day_of_month`, it scans only the
+configured home-relative globs and atomically writes
+`st1-electron-updater-audit-YYYY-MM.json` below the configured report directory.
+It does this before external-volume checks, so an unavailable archive disk does
+not erase the month's ST-1 evidence.
+
+Matches are de-duplicated and recursively sized without following symlinks.
+The report records file counts, bytes, mtimes, stale candidates, and threshold
+reasons. It never moves or deletes a match. `attention` sends the existing Lark
+alert once for that month's report; an incomplete `red` scan fails the formal
+worker closed. A valid `green` or `attention` report suppresses later scans in
+the same Shanghai calendar month.
+
+Commission the exact same scanner without cron lineage or the external archive:
+
+```bash
+/usr/bin/python3 /Users/example/.local/libexec/storage-governance/retention_worker.py \
+ --config /Users/example/.local/libexec/storage-governance/retention-config.json \
+ --electron-audit-only
+```
+
+Audit-only mode still takes the retention worker's single-instance lock and
+forces a fresh report. It is for commissioning and incident response; scheduled
+evidence continues to come from the existing formal cron owner.
+
+## Formal cron lineage
+
+Use the same command for canary, GC audit, and archive. The environment marker
+prevents a normal manual invocation from being mistaken for a cron result:
+
+On macOS, `/usr/sbin/cron` normally lacks Full Disk Access to external media.
+The formal entry therefore runs a synchronous bridge. The bridge proves its
+own live `cron` ancestry, writes a fresh one-time token, starts the FDA-capable
+LaunchAgent, and waits for a token-matched receipt. The worker refuses a green
+result unless that bridge process and its cron parent are still alive:
+
+```cron
+*/15 * * * * /usr/bin/python3 /Users/example/.local/libexec/storage-governance/retention_cron_bridge.py --trigger /Users/example/.local/state/storage-governance/cron-trigger.json --receipt /Users/example/.local/state/storage-governance/cron-receipt.json --alert-log /Users/example/.local/state/storage-governance/retention-alerts.jsonl --config /Users/example/.local/libexec/storage-governance/retention-config.json
+```
+
+Keep the lock and report on the internal volume, and the archive root on the
+external volume. A lock collision or canary failure exits nonzero and records
+an alert; it never starts a second copy or removes a source.
diff --git a/scripts/storage-governance/com.multica.storage-guard.plist.example b/scripts/storage-governance/com.multica.storage-guard.plist.example
new file mode 100644
index 00000000000..987faa730fc
--- /dev/null
+++ b/scripts/storage-governance/com.multica.storage-guard.plist.example
@@ -0,0 +1,37 @@
+
+
+
+
+ Label
+ com.multica.storage-guard
+ ProgramArguments
+
+ /usr/bin/python3
+ -E
+ __INSTALL_ROOT__/storage_guard.py
+ --config
+ __INSTALL_ROOT__/config.json
+
+ EnvironmentVariables
+
+ HOME
+ __HOME__
+ PATH
+ __HOME__/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
+ OPENAI_API_KEY
+
+ OPENAI_BASE_URL
+
+
+ RunAtLoad
+
+ StartInterval
+ 60
+ ProcessType
+ Background
+ StandardOutPath
+ __HOME__/Library/Logs/storage-governance.guard.log
+ StandardErrorPath
+ __HOME__/Library/Logs/storage-governance.guard.err.log
+
+
diff --git a/scripts/storage-governance/com.multica.storage-retention.plist.example b/scripts/storage-governance/com.multica.storage-retention.plist.example
new file mode 100644
index 00000000000..59b3a86cc31
--- /dev/null
+++ b/scripts/storage-governance/com.multica.storage-retention.plist.example
@@ -0,0 +1,21 @@
+
+
+
+
+ Label
+ com.multica.storage-retention
+ ProgramArguments
+
+ /usr/bin/osascript
+ __INSTALL_ROOT__/retention-cron.scpt
+
+ RunAtLoad
+
+ ProcessType
+ Background
+ StandardOutPath
+ __HOME__/Library/Logs/multica-storage-retention.launchd.log
+ StandardErrorPath
+ __HOME__/Library/Logs/multica-storage-retention.launchd.err.log
+
+
diff --git a/scripts/storage-governance/config.example.json b/scripts/storage-governance/config.example.json
new file mode 100644
index 00000000000..e27ce7c1608
--- /dev/null
+++ b/scripts/storage-governance/config.example.json
@@ -0,0 +1,54 @@
+{
+ "internal_path": "/",
+ "external_path": "/Volumes/MacMini-HotSSD",
+ "level1_free_gib": 25,
+ "level1_clear_gib": 28,
+ "level2_free_gib": 18,
+ "level2_clear_gib": 21,
+ "external_min_free_gib": 100,
+ "safety_floor_gib": 25,
+ "burst_reserve_gib": 10,
+ "minimum_observation_hours": 48,
+ "maximum_observation_hours": 72,
+ "expected_interval_seconds": 60,
+ "minimum_sample_coverage": 0.8,
+ "growth_scan_interval_seconds": 900,
+ "growth_scan_budget_seconds": 5,
+ "required_field_max_gap_seconds": 1800,
+ "retention_report_max_age_seconds": 1800,
+ "shadow_runs_path": "/Users/example/path/to/m0-shadow-observer/data/runs",
+ "workspace_roots": [
+ "/Users/example/multica_workspaces",
+ "/Users/example/multica_workspaces_desktop-api.multica.ai"
+ ],
+ "retention_report_path": "/Users/example/.local/state/storage-governance/gc-dry-run.json",
+ "retention_days": 7,
+ "cursor_path": "/Users/example/Library/Application Support/Cursor",
+ "logs_paths": [
+ "/Users/example/.multica",
+ "/Users/example/Library/Logs"
+ ],
+ "required_growth_fields": [
+ "shadow_runs_bytes",
+ "workspace_total_bytes",
+ "workspace_inflight_bytes",
+ "workspace_gc_eligible_bytes",
+ "workspace_gc_backlog_bytes",
+ "workspace_unclassified_bytes",
+ "cursor_bytes",
+ "logs_bytes",
+ "swap_used_bytes",
+ "external_free_bytes"
+ ],
+ "observer_labels": [
+ "ai.multica.ws2512.m0-shadow-observer"
+ ],
+ "nonproduction_launchagents": [],
+ "lark_open_id": "",
+ "alert_cooldown_seconds": 3600,
+ "legacy_sigstop_fallback": false,
+ "state_path": "/Users/example/.local/state/storage-governance/guard-state.json",
+ "growth_cache_path": "/Users/example/.local/state/storage-governance/growth-metrics-cache.json",
+ "metrics_path": "/Users/example/.local/state/storage-governance/host-samples.jsonl",
+ "capacity_report_path": "/Users/example/.local/state/storage-governance/capacity-report.json"
+}
diff --git a/scripts/storage-governance/retention-config.example.json b/scripts/storage-governance/retention-config.example.json
new file mode 100644
index 00000000000..84e1fb5a3fa
--- /dev/null
+++ b/scripts/storage-governance/retention-config.example.json
@@ -0,0 +1,88 @@
+{
+ "external_path": "/Volumes/MacMini-HotSSD",
+ "canary_root": "/Volumes/MacMini-HotSSD/archive/multica-archive",
+ "external_volume_uuid": "00000000-0000-0000-0000-000000000000",
+ "external_min_free_gib": 100,
+ "archive_root": "/Volumes/MacMini-HotSSD/MulticaArchive/workspaces",
+ "workspace_roots": [
+ "/Users/example/multica_workspaces",
+ "/Users/example/multica_workspaces_desktop-api.multica.ai"
+ ],
+ "retention_days": 7,
+ "recent_write_seconds": 86400,
+ "archive_enabled": false,
+ "approved_candidates": [],
+ "delete_source": false,
+ "require_cron_lineage": true,
+ "cron_bridge_trigger_path": "/Users/example/.local/state/storage-governance/cron-trigger.json",
+ "cron_bridge_receipt_path": "/Users/example/.local/state/storage-governance/cron-receipt.json",
+ "lock_path": "/Users/example/.local/state/storage-governance/retention.lock",
+ "report_path": "/Users/example/.local/state/storage-governance/gc-dry-run.json",
+ "alert_log_path": "/Users/example/.local/state/storage-governance/retention-alerts.jsonl",
+ "electron_updater_audit": {
+ "enabled": true,
+ "timezone": "Asia/Shanghai",
+ "day_of_month": 15,
+ "home_path": "/Users/example",
+ "report_dir": "/Users/example/.org/metrics",
+ "patterns": [
+ {
+ "label": "sandbox_update_packages",
+ "glob": "Library/Containers/*/Data/Library/Global/UpdatePackages"
+ },
+ {
+ "label": "application_support_update_noindex",
+ "glob": "Library/Application Support/*/update.noindex"
+ },
+ {
+ "label": "application_support_update_downloading",
+ "glob": "Library/Application Support/*/update_downloading"
+ },
+ {
+ "label": "application_support_software_update",
+ "glob": "Library/Application Support/*/Software Update"
+ },
+ {
+ "label": "sandbox_application_support_update_noindex",
+ "glob": "Library/Containers/*/Data/Library/Application Support/*/update.noindex"
+ },
+ {
+ "label": "sandbox_application_support_update_downloading",
+ "glob": "Library/Containers/*/Data/Library/Application Support/*/update_downloading"
+ },
+ {
+ "label": "electron_software_update_cache",
+ "glob": "Library/Caches/*/Software Update"
+ },
+ {
+ "label": "electron_updater_cache",
+ "glob": "Library/Caches/*-updater"
+ },
+ {
+ "label": "scoped_electron_updater_cache",
+ "glob": "Library/Caches/@*-updater"
+ },
+ {
+ "label": "squirrel_shipit_cache",
+ "glob": "Library/Caches/*.ShipIt"
+ },
+ {
+ "label": "minimax_hot_update",
+ "glob": "Library/Application Support/MiniMax Agent/hot-update"
+ },
+ {
+ "label": "multica_pending_update_zip",
+ "glob": ".multica/*update*.zip"
+ },
+ {
+ "label": "multica_profile_pending_update_zip",
+ "glob": ".multica/profiles/*/*update*.zip"
+ }
+ ],
+ "warn_total_gib": 5,
+ "warn_candidate_gib": 1,
+ "stale_days": 45,
+ "stale_min_mib": 100
+ },
+ "lark_open_id": ""
+}
diff --git a/scripts/storage-governance/retention-cron.applescript.example b/scripts/storage-governance/retention-cron.applescript.example
new file mode 100644
index 00000000000..563d895fe64
--- /dev/null
+++ b/scripts/storage-governance/retention-cron.applescript.example
@@ -0,0 +1,4 @@
+set commandText to "/usr/bin/env -i HOME=__HOME__ PATH=__HOME__/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin MULTICA_STORAGE_CRON_BRIDGE=1 OPENAI_API_KEY='' OPENAI_BASE_URL='' /usr/bin/python3 __INSTALL_ROOT__/retention_worker.py --config __INSTALL_ROOT__/retention-config.json >> __HOME__/Library/Logs/multica-storage-retention.log 2>&1"
+with timeout of 7200 seconds
+ do shell script commandText
+end timeout
diff --git a/scripts/storage-governance/retention_cron_bridge.py b/scripts/storage-governance/retention_cron_bridge.py
new file mode 100644
index 00000000000..a60362377ac
--- /dev/null
+++ b/scripts/storage-governance/retention_cron_bridge.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""Synchronous cron-to-LaunchAgent bridge for macOS Full Disk Access."""
+
+from __future__ import annotations
+
+import argparse
+import fcntl
+import json
+import os
+import subprocess
+import tempfile
+import time
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+from retention_worker import atomic_write_failure_report, send_alert
+
+
+class SingleInstanceLock:
+ def __init__(self, path: Path):
+ self.path = path
+ self.handle = None
+
+ def __enter__(self) -> "SingleInstanceLock":
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self.handle = self.path.open("a+")
+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ return self
+
+ def __exit__(self, exc_type: object, exc: object, traceback: object) -> None:
+ if self.handle is not None:
+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_UN)
+ self.handle.close()
+
+
+def launchctl_kickstart_command(label: str, *, uid: int) -> list[str]:
+ return ["/bin/launchctl", "kickstart", "gui/%d/%s" % (uid, label)]
+
+
+def receipt_exit_code(value: dict, token: str) -> Optional[int]:
+ if value.get("token") != token:
+ return None
+ if value.get("status") == "green":
+ return 0
+ if value.get("status") == "red":
+ return 1
+ return None
+
+
+def record_lock_collision(path: Path) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(
+ json.dumps(
+ {
+ "recorded_at": datetime.now(timezone.utc).isoformat(),
+ "status": "locked",
+ "message": "previous bridge is still running; skipped overlapping retention launch",
+ },
+ sort_keys=True,
+ )
+ + "\n"
+ )
+ handle.flush()
+ os.fsync(handle.fileno())
+
+
+def report_bridge_failure(config_path: Optional[str], message: str) -> None:
+ if not config_path:
+ return
+ try:
+ config = json.loads(Path(config_path).read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ return
+ for action in (
+ lambda: atomic_write_failure_report(config, message),
+ lambda: send_alert(config, message),
+ ):
+ try:
+ action()
+ except Exception:
+ pass
+
+
+def atomic_write(path: Path, value: dict) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ descriptor, temporary = tempfile.mkstemp(prefix=".%s." % path.name, dir=str(path.parent))
+ try:
+ with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+ json.dump(value, handle, sort_keys=True)
+ handle.write("\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temporary, path)
+ finally:
+ try:
+ os.unlink(temporary)
+ except FileNotFoundError:
+ pass
+
+
+def ancestry(pid: int, limit: int = 8) -> list[dict]:
+ values = []
+ for _ in range(limit):
+ parent_result = subprocess.run(
+ ["/bin/ps", "-p", str(pid), "-o", "ppid="], capture_output=True, text=True, check=False
+ )
+ command_result = subprocess.run(
+ ["/bin/ps", "-p", str(pid), "-o", "comm="], capture_output=True, text=True, check=False
+ )
+ if parent_result.returncode != 0 or command_result.returncode != 0:
+ break
+ parent = int(parent_result.stdout.strip())
+ command = command_result.stdout.strip()
+ values.append({"pid": pid, "parent_pid": parent, "command": command})
+ if parent <= 1 or parent == pid:
+ break
+ pid = parent
+ return values
+
+
+def run_bridge(args: argparse.Namespace) -> int:
+ lineage = ancestry(os.getpid())
+ if not any(Path(str(item["command"])).name == "cron" for item in lineage):
+ raise SystemExit("retention bridge refuses non-cron ancestry")
+ token = uuid.uuid4().hex
+ trigger = Path(args.trigger)
+ receipt = Path(args.receipt)
+ atomic_write(
+ trigger,
+ {
+ "schema": "multica.storage-cron-trigger.v1",
+ "token": token,
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "bridge_pid": os.getpid(),
+ "cron_ancestry": lineage,
+ },
+ )
+ process = subprocess.run(
+ launchctl_kickstart_command(args.label, uid=os.getuid()),
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if process.returncode != 0:
+ raise SystemExit("launchctl kickstart failed: " + (process.stderr or process.stdout).strip())
+ deadline = time.monotonic() + args.timeout
+ while time.monotonic() < deadline:
+ try:
+ value = json.loads(receipt.read_text(encoding="utf-8"))
+ except (FileNotFoundError, OSError, json.JSONDecodeError):
+ time.sleep(2)
+ continue
+ result = receipt_exit_code(value, token)
+ if result is not None:
+ return result
+ time.sleep(2)
+ raise SystemExit("retention worker receipt timeout")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--trigger", required=True)
+ parser.add_argument("--receipt", required=True)
+ parser.add_argument("--label", default="com.multica.storage-retention")
+ parser.add_argument("--timeout", type=int, default=7200)
+ parser.add_argument("--lock")
+ parser.add_argument("--alert-log")
+ parser.add_argument("--config")
+ args = parser.parse_args()
+ lock_path = Path(args.lock) if args.lock else Path(args.trigger).with_name("retention-cron-bridge.lock")
+ try:
+ with SingleInstanceLock(lock_path):
+ try:
+ return run_bridge(args)
+ except SystemExit as error:
+ report_bridge_failure(args.config, "storage retention cron bridge failed: %s" % error)
+ raise
+ except BlockingIOError:
+ alert_path = Path(args.alert_log) if args.alert_log else Path(args.trigger).with_name("retention-alerts.jsonl")
+ record_lock_collision(alert_path)
+ report_bridge_failure(args.config, "storage retention cron bridge skipped: previous bridge is still running")
+ return 75
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/storage-governance/retention_worker.py b/scripts/storage-governance/retention_worker.py
new file mode 100755
index 00000000000..cad688a604a
--- /dev/null
+++ b/scripts/storage-governance/retention_worker.py
@@ -0,0 +1,1343 @@
+#!/usr/bin/env python3
+"""Fail-closed canary, workspace GC audit, and transactional archiver.
+
+The same process owns all three operations. A production configuration starts
+with ``archive_enabled=false`` and ``delete_source=false`` so the first run can
+only prove the external-volume path and emit a GC dry-run report.
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import fcntl
+import glob
+import hashlib
+import json
+import os
+import plistlib
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple
+from zoneinfo import ZoneInfo
+
+
+GIB = 1024**3
+TERMINAL_ISSUE_STATUSES = {"done", "cancelled"}
+TERMINAL_RUN_STATUSES = {"completed", "failed", "cancelled"}
+
+
+class ArchiveError(RuntimeError):
+ pass
+
+
+def utc_now() -> datetime:
+ return datetime.now(timezone.utc)
+
+
+def parse_timestamp(value: str) -> datetime:
+ parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ return parsed.astimezone(timezone.utc)
+
+
+def atomic_write_json(path: Path, value: Dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ descriptor, temporary = tempfile.mkstemp(prefix=".%s." % path.name, dir=str(path.parent))
+ try:
+ with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+ json.dump(value, handle, ensure_ascii=False, indent=2, sort_keys=True)
+ handle.write("\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temporary, path)
+ fsync_directory(path.parent)
+ finally:
+ try:
+ os.unlink(temporary)
+ except FileNotFoundError:
+ pass
+
+
+def append_jsonl(path: Path, value: Dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(value, ensure_ascii=False, sort_keys=True) + "\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+
+
+def fsync_directory(path: Path) -> None:
+ descriptor = os.open(str(path), os.O_RDONLY)
+ try:
+ os.fsync(descriptor)
+ finally:
+ os.close(descriptor)
+
+
+def fsync_tree(root: Path) -> None:
+ """Make copied regular-file contents durable before the commit rename."""
+
+ directories: List[Path] = []
+ for current, dirnames, filenames in os.walk(str(root), topdown=True, followlinks=False):
+ current_path = Path(current)
+ directories.append(current_path)
+ dirnames[:] = [name for name in dirnames if not (current_path / name).is_symlink()]
+ for name in filenames:
+ path = current_path / name
+ if path.is_symlink():
+ continue
+ descriptor = os.open(str(path), os.O_RDONLY)
+ try:
+ os.fsync(descriptor)
+ finally:
+ os.close(descriptor)
+ for directory in reversed(directories):
+ fsync_directory(directory)
+
+
+def hash_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ while True:
+ block = handle.read(1024 * 1024)
+ if not block:
+ break
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def _select_samples(paths: List[str], limit: int) -> List[str]:
+ if len(paths) <= limit:
+ return paths
+ if limit <= 1:
+ return [paths[0]]
+ indexes = {round(index * (len(paths) - 1) / (limit - 1)) for index in range(limit)}
+ return [paths[index] for index in sorted(indexes)]
+
+
+def tree_manifest(
+ root: Path,
+ *,
+ sample_limit: int = 16,
+ excluded_relative_paths: Iterable[str] = (),
+) -> Dict[str, Any]:
+ """Describe a tree without following symlinks.
+
+ The entry list freezes every file's size and mtime while hashes provide a
+ deterministic content sample. Both are compared before source deletion.
+ """
+
+ if not root.is_dir() or root.is_symlink():
+ raise ArchiveError("archive source must be a real directory: %s" % root)
+ excluded = set(excluded_relative_paths)
+ directories: List[str] = []
+ files: List[Dict[str, Any]] = []
+ symlinks: List[Dict[str, str]] = []
+ regular_paths: List[str] = []
+ for current, dirnames, filenames in os.walk(str(root), topdown=True, followlinks=False):
+ current_path = Path(current)
+ kept_dirs: List[str] = []
+ for name in sorted(dirnames):
+ path = current_path / name
+ relative = path.relative_to(root).as_posix()
+ if relative in excluded:
+ continue
+ if path.is_symlink():
+ symlinks.append({"path": relative, "target": os.readlink(str(path))})
+ else:
+ directories.append(relative)
+ kept_dirs.append(name)
+ dirnames[:] = kept_dirs
+ for name in sorted(filenames):
+ path = current_path / name
+ relative = path.relative_to(root).as_posix()
+ if relative in excluded:
+ continue
+ if path.is_symlink():
+ symlinks.append({"path": relative, "target": os.readlink(str(path))})
+ continue
+ stat = path.lstat()
+ if not path.is_file():
+ raise ArchiveError("unsupported non-regular entry: %s" % path)
+ files.append({"path": relative, "size": stat.st_size, "mtime_ns": stat.st_mtime_ns})
+ regular_paths.append(relative)
+ files.sort(key=lambda item: str(item["path"]))
+ directories.sort()
+ symlinks.sort(key=lambda item: item["path"])
+ samples = _select_samples(sorted(regular_paths), sample_limit)
+ content_hashes = {relative: hash_file(root / relative) for relative in sorted(regular_paths)}
+ return {
+ "file_count": len(files),
+ "directory_count": len(directories),
+ "symlink_count": len(symlinks),
+ "total_bytes": sum(int(item["size"]) for item in files),
+ "directories": directories,
+ "files": files,
+ "symlinks": symlinks,
+ "content_hashes": content_hashes,
+ "sample_hashes": {relative: content_hashes[relative] for relative in samples},
+ }
+
+
+def read_volume_uuid(path: Path) -> str:
+ process = subprocess.run(
+ ["/usr/sbin/diskutil", "info", "-plist", str(path)],
+ capture_output=True,
+ check=False,
+ )
+ if process.returncode != 0:
+ raise ArchiveError("diskutil could not inspect external volume")
+ try:
+ value = plistlib.loads(process.stdout)
+ except (plistlib.InvalidFileException, ValueError) as error:
+ raise ArchiveError("diskutil returned invalid volume metadata") from error
+ volume_uuid = value.get("VolumeUUID")
+ if not volume_uuid:
+ raise ArchiveError("external volume has no VolumeUUID")
+ return str(volume_uuid).upper()
+
+
+def available_bytes(path: Path) -> int:
+ stat = os.statvfs(str(path))
+ return stat.f_bavail * stat.f_frsize
+
+
+def _audit_clock(now: Callable[[], datetime], timezone_name: str) -> Tuple[datetime, datetime]:
+ checked_at = now()
+ if checked_at.tzinfo is None:
+ checked_at = checked_at.replace(tzinfo=timezone.utc)
+ checked_at = checked_at.astimezone(timezone.utc)
+ try:
+ local_time = checked_at.astimezone(ZoneInfo(timezone_name))
+ except Exception as error:
+ raise ArchiveError("electron updater audit timezone is invalid: %s" % timezone_name) from error
+ return checked_at, local_time
+
+
+def _audit_report_path(audit_config: Dict[str, Any], local_time: datetime) -> Path:
+ report_dir = Path(str(audit_config["report_dir"]))
+ return report_dir / ("st1-electron-updater-audit-%04d-%02d.json" % (local_time.year, local_time.month))
+
+
+def _path_is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _updater_candidate_metrics(path: Path) -> Tuple[Dict[str, Any], List[str]]:
+ errors: List[str] = []
+ file_count = 0
+ total_bytes = 0
+ skipped_symlink_count = 0
+ mtimes: List[float] = []
+ try:
+ root_stat = path.lstat()
+ mtimes.append(root_stat.st_mtime)
+ if path.is_file():
+ if stat.S_ISREG(root_stat.st_mode):
+ file_count = 1
+ total_bytes = root_stat.st_size
+ elif path.is_dir():
+ def record_walk_error(error: OSError) -> None:
+ errors.append("filesystem scan failed for %s: %s" % (path, error))
+
+ for current, dirnames, filenames in os.walk(
+ str(path),
+ topdown=True,
+ followlinks=False,
+ onerror=record_walk_error,
+ ):
+ current_path = Path(current)
+ kept_directories: List[str] = []
+ for name in sorted(dirnames):
+ child = current_path / name
+ if child.is_symlink():
+ skipped_symlink_count += 1
+ continue
+ try:
+ mtimes.append(child.lstat().st_mtime)
+ except OSError as error:
+ errors.append("filesystem metadata failed for %s: %s" % (child, error))
+ continue
+ kept_directories.append(name)
+ dirnames[:] = kept_directories
+ for name in sorted(filenames):
+ child = current_path / name
+ try:
+ child_stat = child.lstat()
+ except OSError as error:
+ errors.append("filesystem metadata failed for %s: %s" % (child, error))
+ continue
+ if stat.S_ISLNK(child_stat.st_mode):
+ skipped_symlink_count += 1
+ continue
+ mtimes.append(child_stat.st_mtime)
+ if stat.S_ISREG(child_stat.st_mode):
+ file_count += 1
+ total_bytes += child_stat.st_size
+ else:
+ errors.append("unsupported updater residue type: %s" % path)
+ except OSError as error:
+ errors.append("filesystem scan failed for %s: %s" % (path, error))
+ metrics = {
+ "file_count": file_count,
+ "total_bytes": total_bytes,
+ "skipped_symlink_count": skipped_symlink_count,
+ "oldest_mtime": datetime.fromtimestamp(min(mtimes), timezone.utc).isoformat() if mtimes else None,
+ "newest_mtime": datetime.fromtimestamp(max(mtimes), timezone.utc).isoformat() if mtimes else None,
+ "newest_mtime_epoch": max(mtimes) if mtimes else None,
+ }
+ return metrics, errors
+
+
+def audit_electron_updaters(
+ config: Dict[str, Any],
+ *,
+ now: Callable[[], datetime] = utc_now,
+ invocation_source: str = "manual",
+) -> Dict[str, Any]:
+ audit_config = config.get("electron_updater_audit")
+ if not isinstance(audit_config, dict):
+ raise ArchiveError("electron_updater_audit config is missing")
+ checked_at, local_time = _audit_clock(now, str(audit_config.get("timezone") or "Asia/Shanghai"))
+ home_path = Path(str(audit_config["home_path"]))
+ if home_path.is_symlink() or not home_path.is_dir():
+ raise ArchiveError("electron updater audit home must be a real directory: %s" % home_path)
+ home = home_path.resolve(strict=True)
+ raw_matches: Dict[Path, set[str]] = {}
+ excluded_symlink_matches: Dict[str, Dict[str, str]] = {}
+ errors: List[str] = []
+ patterns = audit_config.get("patterns")
+ if not isinstance(patterns, list) or not patterns:
+ raise ArchiveError("electron updater audit patterns must be a non-empty list")
+ for value in patterns:
+ if not isinstance(value, dict) or not value.get("label") or not value.get("glob"):
+ raise ArchiveError("electron updater audit pattern entries require label and glob")
+ label = str(value["label"])
+ relative_pattern = Path(str(value["glob"]))
+ if relative_pattern.is_absolute() or ".." in relative_pattern.parts:
+ raise ArchiveError("electron updater audit glob must stay relative to home: %s" % relative_pattern)
+ if "**" in relative_pattern.parts:
+ raise ArchiveError("electron updater audit recursive globs are not allowed: %s" % relative_pattern)
+ absolute_pattern = str(home / relative_pattern)
+ for matched in glob.glob(absolute_pattern):
+ path = Path(matched)
+ if path.is_symlink():
+ try:
+ target = os.readlink(str(path))
+ except OSError as error:
+ errors.append("updater residue symlink is unreadable: %s: %s" % (path, error))
+ continue
+ excluded_symlink_matches[str(path)] = {
+ "path": str(path),
+ "target": target,
+ "reason": "symlink match was not traversed or counted",
+ }
+ continue
+ try:
+ resolved = path.resolve(strict=True)
+ except OSError as error:
+ errors.append("updater residue match is unreadable: %s: %s" % (path, error))
+ continue
+ if not _path_is_within(resolved, home):
+ errors.append("updater residue match escaped home: %s" % path)
+ continue
+ raw_matches.setdefault(resolved, set()).add(label)
+
+ roots: List[Dict[str, Any]] = []
+ for path, labels in sorted(raw_matches.items(), key=lambda item: (len(item[0].parts), str(item[0]))):
+ parent = next((item for item in roots if _path_is_within(path, Path(str(item["path"])))), None)
+ if parent is not None:
+ parent["matched_labels"] = sorted(set(parent["matched_labels"]) | labels)
+ continue
+ roots.append({"path": str(path), "matched_labels": sorted(labels)})
+
+ candidates: List[Dict[str, Any]] = []
+ attention_reasons: List[str] = []
+ warn_candidate_bytes = int(float(audit_config.get("warn_candidate_gib", 1)) * GIB)
+ stale_seconds = int(float(audit_config.get("stale_days", 45)) * 86400)
+ stale_min_bytes = int(float(audit_config.get("stale_min_mib", 100)) * 1024**2)
+ for root in roots:
+ path = Path(str(root["path"]))
+ metrics, scan_errors = _updater_candidate_metrics(path)
+ errors.extend(scan_errors)
+ newest_epoch = metrics.pop("newest_mtime_epoch")
+ age_seconds = max(0.0, checked_at.timestamp() - newest_epoch) if newest_epoch is not None else None
+ candidate = {
+ **root,
+ **metrics,
+ "age_seconds": age_seconds,
+ "stale": bool(
+ age_seconds is not None
+ and age_seconds >= stale_seconds
+ and int(metrics["total_bytes"]) >= stale_min_bytes
+ ),
+ }
+ candidates.append(candidate)
+ if warn_candidate_bytes > 0 and int(candidate["total_bytes"]) >= warn_candidate_bytes:
+ attention_reasons.append(
+ "candidate exceeds %.3f GiB threshold: %s"
+ % (float(audit_config.get("warn_candidate_gib", 1)), candidate["path"])
+ )
+ if candidate["stale"]:
+ attention_reasons.append(
+ "stale candidate exceeds %.3f MiB floor: %s"
+ % (float(audit_config.get("stale_min_mib", 100)), candidate["path"])
+ )
+
+ total_bytes = sum(int(candidate["total_bytes"]) for candidate in candidates)
+ warn_total_bytes = int(float(audit_config.get("warn_total_gib", 5)) * GIB)
+ if warn_total_bytes > 0 and total_bytes >= warn_total_bytes:
+ attention_reasons.insert(
+ 0,
+ "total updater residue exceeds %.3f GiB threshold"
+ % float(audit_config.get("warn_total_gib", 5)),
+ )
+ status = "red" if errors else ("attention" if attention_reasons else "green")
+ report_path = _audit_report_path(audit_config, local_time)
+ report: Dict[str, Any] = {
+ "schema": "multica.st1-electron-updater-audit.v1",
+ "duty": "ST-1",
+ "status": status,
+ "audit_month": "%04d-%02d" % (local_time.year, local_time.month),
+ "recorded_at": checked_at.isoformat(),
+ "recorded_at_local": local_time.isoformat(),
+ "timezone": str(audit_config.get("timezone") or "Asia/Shanghai"),
+ "invocation_source": invocation_source,
+ "report_path": str(report_path),
+ "candidate_count": len(candidates),
+ "total_bytes": total_bytes,
+ "stale_candidate_count": sum(1 for candidate in candidates if candidate["stale"]),
+ "attention_reasons": attention_reasons,
+ "errors": errors,
+ "excluded_symlink_matches": [
+ excluded_symlink_matches[path] for path in sorted(excluded_symlink_matches)
+ ],
+ "candidates": candidates,
+ "destructive_actions": [],
+ }
+ atomic_write_json(report_path, report)
+ return report
+
+
+def maybe_run_electron_updater_audit(
+ config: Dict[str, Any],
+ *,
+ now: Callable[[], datetime] = utc_now,
+ force: bool = False,
+ invocation_source: str = "manual",
+) -> Dict[str, Any]:
+ audit_config = config.get("electron_updater_audit")
+ if not isinstance(audit_config, dict) or not audit_config.get("enabled", False):
+ return {"status": "skipped", "reason": "electron updater audit is disabled"}
+ _, local_time = _audit_clock(now, str(audit_config.get("timezone") or "Asia/Shanghai"))
+ due_day = int(audit_config.get("day_of_month", 15))
+ if due_day < 1 or due_day > 28:
+ raise ArchiveError("electron updater audit day_of_month must be between 1 and 28")
+ if not force and local_time.day < due_day:
+ return {"status": "skipped", "reason": "current month is before day %d" % due_day}
+ report_path = _audit_report_path(audit_config, local_time)
+ if not force and report_path.is_file():
+ try:
+ existing = json.loads(report_path.read_text(encoding="utf-8"))
+ modified = datetime.fromtimestamp(report_path.stat().st_mtime, ZoneInfo(str(audit_config.get("timezone") or "Asia/Shanghai")))
+ except (OSError, ValueError, json.JSONDecodeError):
+ existing = {}
+ modified = None
+ if (
+ isinstance(existing, dict)
+ and existing.get("schema") == "multica.st1-electron-updater-audit.v1"
+ and existing.get("audit_month") == "%04d-%02d" % (local_time.year, local_time.month)
+ and existing.get("status") in {"green", "attention"}
+ and modified is not None
+ and (modified.year, modified.month) == (local_time.year, local_time.month)
+ ):
+ return {
+ "status": "skipped",
+ "reason": "valid current-month evidence already exists",
+ "report_path": str(report_path),
+ "evidence_status": existing["status"],
+ }
+ return audit_electron_updaters(config, now=now, invocation_source=invocation_source)
+
+
+class Canary:
+ def __init__(
+ self,
+ destination: Path,
+ *,
+ expected_uuid: str,
+ min_free_bytes: int,
+ volume_path: Optional[Path] = None,
+ uuid_reader: Callable[[Path], str] = read_volume_uuid,
+ free_bytes_reader: Callable[[Path], int] = available_bytes,
+ postflight: Optional[Callable[[], Dict[str, Any]]] = None,
+ ):
+ self.destination = destination
+ self.expected_uuid = expected_uuid.upper()
+ self.min_free_bytes = min_free_bytes
+ self.volume_path = volume_path or destination
+ self.uuid_reader = uuid_reader
+ self.free_bytes_reader = free_bytes_reader
+ self.postflight = postflight
+
+ def run(self) -> Dict[str, Any]:
+ actual_uuid = self.uuid_reader(self.volume_path).upper()
+ if actual_uuid != self.expected_uuid:
+ raise ArchiveError(
+ "external volume UUID mismatch: expected %s, got %s"
+ % (self.expected_uuid, actual_uuid)
+ )
+ free_bytes = self.free_bytes_reader(self.volume_path)
+ if free_bytes < self.min_free_bytes:
+ raise ArchiveError(
+ "external volume below low-water mark: %d < %d bytes"
+ % (free_bytes, self.min_free_bytes)
+ )
+
+ canary_root = self.destination / ".multica-storage-canary"
+ canary_root.mkdir(parents=True, exist_ok=True)
+ run_name = "%s-%s" % (utc_now().strftime("%Y%m%dT%H%M%S.%fZ"), uuid.uuid4().hex[:8])
+ partial = canary_root / (run_name + ".partial")
+ final = canary_root / run_name
+ (partial / "nested" / "deeper").mkdir(parents=True)
+ (partial / "root.txt").write_text("multica storage canary\n", encoding="utf-8")
+ (partial / "nested" / "payload.bin").write_bytes(bytes(range(128)))
+ (partial / "nested" / "deeper" / "empty").write_bytes(b"")
+ fsync_tree(partial)
+ expected = tree_manifest(partial)
+ os.replace(str(partial), str(final))
+ fsync_directory(canary_root)
+ actual = tree_manifest(final)
+ if expected != actual:
+ raise ArchiveError("canary verification failed after atomic rename")
+ if self.postflight is not None:
+ verified_volume = self.postflight()
+ actual_uuid = str(verified_volume["volume_uuid"])
+ free_bytes = int(verified_volume["free_bytes"])
+ result = {
+ "schema": "multica.external-volume-canary.v1",
+ "status": "green",
+ "checked_at": utc_now().isoformat(),
+ "volume_uuid": actual_uuid,
+ "free_bytes": free_bytes,
+ "minimum_free_bytes": self.min_free_bytes,
+ "path": str(final),
+ "manifest": actual,
+ }
+ atomic_write_json(final / "CANARY.json", result)
+ completed = sorted(
+ (path for path in canary_root.iterdir() if path.is_dir() and not path.name.endswith(".partial")),
+ key=lambda path: path.name,
+ reverse=True,
+ )
+ for stale in completed[96:]:
+ shutil.rmtree(str(stale))
+ return result
+
+
+class ExternalVolumeGuard:
+ """Bind archive writes to one physical external volume at every phase."""
+
+ def __init__(
+ self,
+ external_root: Path,
+ archive_root: Path,
+ *,
+ expected_uuid: str,
+ min_free_bytes: int,
+ uuid_reader: Callable[[Path], str] = read_volume_uuid,
+ free_bytes_reader: Callable[[Path], int] = available_bytes,
+ ):
+ self.external_root = external_root
+ self.archive_root = archive_root
+ self.expected_uuid = expected_uuid.upper()
+ self.min_free_bytes = min_free_bytes
+ self.uuid_reader = uuid_reader
+ self.free_bytes_reader = free_bytes_reader
+
+ def check(self, required_bytes: int = 0) -> Dict[str, Any]:
+ if self.external_root.is_symlink() or self.archive_root.is_symlink():
+ raise ArchiveError("external or archive root must not be a symlink")
+ external = self.external_root.resolve(strict=True)
+ archive = self.archive_root.resolve(strict=True)
+ try:
+ archive.relative_to(external)
+ except ValueError as error:
+ raise ArchiveError("archive root is outside the verified external root") from error
+ if external.stat().st_dev != archive.stat().st_dev:
+ raise ArchiveError("archive root is not on the verified external device")
+ actual_uuid = self.uuid_reader(external).upper()
+ if actual_uuid != self.expected_uuid:
+ raise ArchiveError("archive volume UUID changed before commit")
+ free_bytes = self.free_bytes_reader(archive)
+ reserve = self.min_free_bytes + int(required_bytes * 1.10)
+ if free_bytes < reserve:
+ raise ArchiveError(
+ "archive volume lacks candidate budget: %d < %d bytes" % (free_bytes, reserve)
+ )
+ return {"volume_uuid": actual_uuid, "free_bytes": free_bytes, "required_reserve_bytes": reserve}
+
+
+class SingleInstanceLock:
+ def __init__(self, path: Path):
+ self.path = path
+ self.handle: Optional[Any] = None
+
+ def __enter__(self) -> "SingleInstanceLock":
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self.handle = self.path.open("a+")
+ try:
+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ self.handle.close()
+ self.handle = None
+ raise
+ self.handle.seek(0)
+ self.handle.truncate()
+ self.handle.write("%d\n" % os.getpid())
+ self.handle.flush()
+ return self
+
+ def __exit__(self, exc_type: Any, exc: Any, traceback: Any) -> None:
+ if self.handle is not None:
+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_UN)
+ self.handle.close()
+ self.handle = None
+
+
+class ArchiveManager:
+ def __init__(
+ self,
+ destination: Path,
+ *,
+ fail_at: Optional[str] = None,
+ preflight: Optional[Callable[[int], Any]] = None,
+ ):
+ self.destination = destination
+ self.fail_at = fail_at
+ self.preflight = preflight
+
+ def _fail(self, phase: str) -> None:
+ if self.fail_at == phase:
+ raise ArchiveError("injected failure at %s" % phase)
+
+ def archive(
+ self,
+ source: Path,
+ candidate_id: str,
+ *,
+ delete_source: bool = False,
+ delete_gate: Optional[Callable[[], bool]] = None,
+ approval_token: Optional[str] = None,
+ post_commit_hook: Optional[Callable[[Path], None]] = None,
+ ) -> Dict[str, Any]:
+ if delete_source:
+ raise ArchiveError("source deletion requires a producer lease and is disabled")
+ source = source.absolute()
+ if not source.is_dir() or source.is_symlink():
+ raise ArchiveError("refusing unsafe source: %s" % source)
+ destination = self.destination.absolute()
+ if source == destination or source in destination.parents or destination in source.parents:
+ raise ArchiveError("source and archive destination must not contain one another")
+ self.destination.mkdir(parents=True, exist_ok=True)
+
+ frozen = tree_manifest(source)
+ if self.preflight is not None:
+ self.preflight(int(frozen["total_bytes"]))
+ suffix = utc_now().strftime("%Y%m%dT%H%M%S.%fZ")
+ final = self.destination / (candidate_id + "-" + suffix)
+ partial = self.destination / (final.name + ".partial")
+ if partial.exists() or final.exists():
+ raise ArchiveError("archive destination already exists")
+
+ try:
+ shutil.copytree(str(source), str(partial), symlinks=True, copy_function=shutil.copy2)
+ fsync_tree(partial)
+ fsync_directory(self.destination)
+ self._fail("after_copy")
+
+ copied = tree_manifest(partial)
+ source_after_copy = tree_manifest(source)
+ if frozen != source_after_copy:
+ raise ArchiveError("source changed while archive copy was in progress")
+ if frozen != copied:
+ raise ArchiveError("archive count/bytes/entry/sample verification failed")
+ self._fail("after_verify")
+
+ if self.preflight is not None:
+ self.preflight(0)
+ os.replace(str(partial), str(final))
+ fsync_directory(self.destination)
+ self._fail("after_rename")
+
+ if post_commit_hook is not None:
+ post_commit_hook(final)
+ committed = tree_manifest(final)
+ if committed != frozen:
+ raise ArchiveError("committed archive changed before COMPLETE marker")
+
+ marker = {
+ "schema": "multica.transactional-archive.v1",
+ "completed_at": utc_now().isoformat(),
+ "candidate_id": candidate_id,
+ "source_path": str(source),
+ "archive_path": str(final),
+ "source_manifest": frozen,
+ "archive_manifest": committed,
+ "approval_token": approval_token,
+ "source_delete_enabled": False,
+ }
+ atomic_write_json(final / "COMPLETE.json", marker)
+ fsync_directory(final)
+ self._fail("after_complete")
+
+ source_before_delete = tree_manifest(source)
+ persisted = json.loads((final / "COMPLETE.json").read_text(encoding="utf-8"))
+ if source_before_delete != frozen or persisted != marker:
+ raise ArchiveError("source or COMPLETE marker changed before delete gate")
+ return {
+ "status": "complete",
+ "source_path": str(source),
+ "archive_path": str(final),
+ "source_deleted": False,
+ "manifest": frozen,
+ }
+ except ArchiveError:
+ raise
+ except Exception as error:
+ raise ArchiveError("transactional archive failed: %s" % error) from error
+
+
+class MulticaIssueClient:
+ def _json(self, argv: List[str]) -> Any:
+ try:
+ process = subprocess.run(argv, capture_output=True, text=True, check=False, timeout=15)
+ except subprocess.TimeoutExpired as error:
+ raise ArchiveError("multica query timed out") from error
+ if process.returncode != 0:
+ detail = (process.stderr or process.stdout).strip()
+ raise ArchiveError("multica query failed: %s" % detail)
+ try:
+ return json.loads(process.stdout)
+ except json.JSONDecodeError as error:
+ raise ArchiveError("multica query returned invalid JSON") from error
+
+ def get_issue(self, issue_id: str) -> Dict[str, Any]:
+ value = self._json(["multica", "issue", "get", issue_id, "--output", "json"])
+ if not isinstance(value, dict):
+ raise ArchiveError("issue response is not an object")
+ return value
+
+ def get_runs(self, issue_id: str) -> List[Dict[str, Any]]:
+ value = self._json(["multica", "issue", "runs", issue_id, "--output", "json"])
+ return [item for item in value if isinstance(item, dict)] if isinstance(value, list) else []
+
+ def get_children(self, issue_id: str) -> List[Dict[str, Any]]:
+ value = self._json(["multica", "issue", "children", issue_id, "--output", "json"])
+ if not isinstance(value, dict) or "stages" not in value or "unstaged" not in value or "total" not in value:
+ raise ArchiveError("children response schema is unknown")
+ children: List[Dict[str, Any]] = []
+ for item in value.get("unstaged") or []:
+ if isinstance(item, dict):
+ children.append(item)
+ for stage in value.get("stages") or []:
+ if not isinstance(stage, dict):
+ continue
+ for item in stage.get("issues") or stage.get("children") or []:
+ if isinstance(item, dict):
+ children.append(item)
+ if int(value["total"]) != len(children):
+ raise ArchiveError("children response count does not match parsed entries")
+ return children
+
+
+def default_open_file_checker(path: Path) -> bool:
+ process = subprocess.run(
+ ["/usr/sbin/lsof", "+D", str(path)],
+ capture_output=True,
+ text=True,
+ check=False,
+ timeout=30,
+ )
+ if process.returncode not in (0, 1):
+ raise ArchiveError("lsof failed while checking %s" % path)
+ lines = [line for line in process.stdout.splitlines() if line.strip()]
+ return len(lines) > 1
+
+
+def _latest_mtime_size_and_unsafe_symlinks(root: Path) -> Tuple[float, int, List[str]]:
+ latest = root.lstat().st_mtime
+ total_bytes = 0
+ unsafe: List[str] = []
+ resolved_root = root.resolve()
+ for current, dirnames, filenames in os.walk(str(root), topdown=True, followlinks=False):
+ current_path = Path(current)
+ kept_dirs: List[str] = []
+ for name in dirnames:
+ path = current_path / name
+ stat = path.lstat()
+ latest = max(latest, stat.st_mtime)
+ if path.is_symlink():
+ try:
+ path.resolve(strict=False).relative_to(resolved_root)
+ except ValueError:
+ unsafe.append(path.relative_to(root).as_posix())
+ else:
+ kept_dirs.append(name)
+ dirnames[:] = kept_dirs
+ for name in filenames:
+ path = current_path / name
+ stat = path.lstat()
+ latest = max(latest, stat.st_mtime)
+ if not path.is_symlink():
+ total_bytes += stat.st_size
+ if path.is_symlink():
+ try:
+ path.resolve(strict=False).relative_to(resolved_root)
+ except ValueError:
+ unsafe.append(path.relative_to(root).as_posix())
+ return latest, total_bytes, sorted(unsafe)
+
+
+class GCEvaluator:
+ def __init__(
+ self,
+ client: Any,
+ *,
+ now: Callable[[], datetime] = utc_now,
+ open_file_checker: Callable[[Path], bool] = default_open_file_checker,
+ retention_seconds: int = 7 * 86400,
+ recent_write_seconds: int = 24 * 3600,
+ ):
+ self.client = client
+ self.now = now
+ self.open_file_checker = open_file_checker
+ self.retention_seconds = retention_seconds
+ self.recent_write_seconds = recent_write_seconds
+
+ def evaluate(self, candidate: Path) -> Dict[str, Any]:
+ reasons: List[str] = []
+ details: Dict[str, Any] = {}
+ meta_path = candidate / ".gc_meta.json"
+ context_path = candidate / "workdir" / ".multica" / "daemon_task_context.json"
+ try:
+ meta = json.loads(meta_path.read_text(encoding="utf-8"))
+ context = json.loads(context_path.read_text(encoding="utf-8"))
+ except (FileNotFoundError, OSError, json.JSONDecodeError) as error:
+ return {
+ "path": str(candidate),
+ "eligible": False,
+ "reasons": ["identity metadata unreadable: %s" % error],
+ }
+ if not isinstance(meta, dict) or not isinstance(context, dict):
+ return {"path": str(candidate), "eligible": False, "reasons": ["identity metadata invalid"]}
+
+ issue_id = str(meta.get("issue_id") or "")
+ workspace_id = str(meta.get("workspace_id") or "")
+ if (
+ meta.get("kind") != "issue"
+ or not issue_id
+ or not workspace_id
+ or candidate.parent.name != workspace_id
+ or context.get("managed_by") != "multica-daemon-task"
+ or context.get("issue_id") != issue_id
+ ):
+ reasons.append("identity metadata does not match directory")
+
+ try:
+ completed_at = parse_timestamp(str(meta.get("completed_at") or ""))
+ age = (self.now() - completed_at).total_seconds()
+ details["completed_at"] = completed_at.isoformat()
+ details["age_seconds"] = age
+ if age < self.retention_seconds:
+ reasons.append("retention window has not elapsed")
+ except (ValueError, TypeError):
+ completed_at = None
+ reasons.append("completed_at is invalid")
+
+ try:
+ issue = self.client.get_issue(issue_id)
+ runs = self.client.get_runs(issue_id)
+ children = self.client.get_children(issue_id)
+ except Exception as error:
+ reasons.append("control-plane lookup failed closed: %s" % error)
+ issue, runs, children = {}, [], []
+
+ if issue.get("status") not in TERMINAL_ISSUE_STATUSES:
+ reasons.append("issue is not in an irreversible terminal status")
+ if issue.get("id") != issue_id or issue.get("workspace_id") != workspace_id:
+ reasons.append("identity differs from control plane")
+ metadata = issue.get("metadata") if isinstance(issue.get("metadata"), dict) else {}
+ if any(metadata.get(key) for key in ("gc_pin", "pinned", "retention_pin")) or (candidate / ".gc-pin").exists():
+ reasons.append("candidate has a retention pin")
+ if any(child.get("status") not in TERMINAL_ISSUE_STATUSES for child in children):
+ reasons.append("nonterminal child or supplement verification exists")
+ if any(run.get("status") not in TERMINAL_RUN_STATUSES for run in runs):
+ reasons.append("active run or lease exists")
+
+ matching_runs: List[Dict[str, Any]] = []
+ for run in runs:
+ run_id = str(run.get("id") or "")
+ work_dir = Path(str(run.get("work_dir") or ""))
+ if run_id.startswith(candidate.name) and work_dir == candidate / "workdir":
+ matching_runs.append(run)
+ if len(matching_runs) != 1:
+ reasons.append("identity has no unique matching task run")
+ else:
+ run = matching_runs[0]
+ if (
+ run.get("issue_id") not in (None, issue_id)
+ or run.get("workspace_id") not in (None, workspace_id)
+ or run.get("status") not in TERMINAL_RUN_STATUSES
+ ):
+ reasons.append("identity or terminal state differs from task run")
+ if completed_at is not None:
+ try:
+ run_completed = parse_timestamp(str(run.get("completed_at") or ""))
+ if abs((run_completed - completed_at).total_seconds()) > 1:
+ reasons.append("identity completed_at differs from task run")
+ except ValueError:
+ reasons.append("task run completed_at is invalid")
+
+ try:
+ if self.open_file_checker(candidate):
+ reasons.append("open file exists under candidate")
+ except Exception as error:
+ reasons.append("open file check failed closed: %s" % error)
+ try:
+ latest_mtime, total_bytes, unsafe_symlinks = _latest_mtime_size_and_unsafe_symlinks(candidate)
+ details["latest_mtime"] = datetime.fromtimestamp(latest_mtime, timezone.utc).isoformat()
+ details["size_bytes"] = total_bytes
+ if self.now().timestamp() - latest_mtime < self.recent_write_seconds:
+ reasons.append("recent write exists under candidate")
+ if unsafe_symlinks:
+ reasons.append("out-of-bound symlink present (not followed): %s" % ", ".join(unsafe_symlinks))
+ except OSError as error:
+ reasons.append("filesystem scan failed closed: %s" % error)
+
+ result = {
+ "path": str(candidate),
+ "issue_id": issue_id,
+ "workspace_id": workspace_id,
+ "eligible": not reasons,
+ "reasons": reasons,
+ "details": details,
+ }
+ if not reasons:
+ try:
+ manifest = tree_manifest(candidate)
+ digest = hashlib.sha256(
+ json.dumps(manifest, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")
+ ).hexdigest()
+ matching = matching_runs[0]
+ approval_identity = {
+ "source_path": str(candidate.resolve()),
+ "workspace_id": workspace_id,
+ "issue_id": issue_id,
+ "run_id": str(matching.get("id")),
+ "completed_at": str(meta.get("completed_at")),
+ "manifest_sha256": digest,
+ }
+ result["run_id"] = approval_identity["run_id"]
+ result["manifest_sha256"] = digest
+ result["approval_token"] = hashlib.sha256(
+ json.dumps(approval_identity, sort_keys=True, separators=(",", ":")).encode("utf-8")
+ ).hexdigest()
+ result["details"]["size_bytes"] = int(manifest["total_bytes"])
+ except Exception as error:
+ reasons.append("approval manifest failed closed: %s" % error)
+ result["eligible"] = False
+ return result
+
+
+def discover_candidates(roots: Iterable[Path]) -> List[Path]:
+ candidates: List[Path] = []
+ for root in roots:
+ if not root.is_dir() or root.is_symlink():
+ continue
+ for workspace in sorted(root.iterdir()):
+ if not workspace.is_dir() or workspace.is_symlink() or workspace.name.startswith("."):
+ continue
+ for candidate in sorted(workspace.iterdir()):
+ if candidate.is_dir() and not candidate.is_symlink() and (candidate / ".gc_meta.json").is_file():
+ candidates.append(candidate)
+ return candidates
+
+
+def consumed_approval_tokens(archive_root: Path) -> set[str]:
+ tokens: set[str] = set()
+ if not archive_root.is_dir():
+ return tokens
+ for marker_path in archive_root.glob("*/COMPLETE.json"):
+ try:
+ marker = json.loads(marker_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ continue
+ token = marker.get("approval_token")
+ if token:
+ tokens.add(str(token))
+ return tokens
+
+
+def has_message_id(value: Any) -> bool:
+ if isinstance(value, dict):
+ return bool(value.get("message_id")) or any(has_message_id(child) for child in value.values())
+ if isinstance(value, list):
+ return any(has_message_id(child) for child in value)
+ return False
+
+
+def send_alert(config: Dict[str, Any], message: str) -> None:
+ alert_path = Path(str(config["alert_log_path"]))
+ try:
+ append_jsonl(alert_path, {"recorded_at": utc_now().isoformat(), "message": message})
+ except OSError:
+ pass
+ open_id = str(config.get("lark_open_id") or "")
+ if not open_id:
+ return
+ try:
+ process = subprocess.run(
+ [
+ "lark-cli",
+ "im",
+ "+messages-send",
+ "--as",
+ "bot",
+ "--user-id",
+ open_id,
+ "--text",
+ message,
+ "--format",
+ "json",
+ ],
+ capture_output=True,
+ text=True,
+ check=False,
+ timeout=15,
+ )
+ response = json.loads(process.stdout) if process.stdout else {}
+ delivered = process.returncode == 0 and has_message_id(response)
+ except (OSError, subprocess.TimeoutExpired, json.JSONDecodeError):
+ delivered = False
+ process = None
+ if not delivered:
+ exit_code = process.returncode if process is not None else None
+ try:
+ append_jsonl(
+ alert_path,
+ {"recorded_at": utc_now().isoformat(), "message": "lark alert delivery failed", "exit_code": exit_code},
+ )
+ except OSError:
+ pass
+
+
+def process_ancestry(start_pid: Optional[int] = None, limit: int = 8) -> List[Dict[str, Any]]:
+ pid = os.getpid() if start_pid is None else start_pid
+ values: List[Dict[str, Any]] = []
+ for _ in range(limit):
+ process = subprocess.run(
+ ["/bin/ps", "-p", str(pid), "-o", "ppid=", "-o", "comm="],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ fields = process.stdout.strip().split(None, 1)
+ if process.returncode != 0 or len(fields) != 2:
+ break
+ parent = int(fields[0])
+ command = fields[1]
+ values.append({"pid": pid, "parent_pid": parent, "command": command})
+ if parent <= 1 or parent == pid:
+ break
+ pid = parent
+ return values
+
+
+def verify_cron_bridge(config: Dict[str, Any]) -> Tuple[Dict[str, Any], List[Dict[str, Any]]]:
+ if os.environ.get("MULTICA_STORAGE_CRON_BRIDGE") != "1":
+ raise ArchiveError("formal cron bridge marker is missing")
+ trigger_path = Path(str(config["cron_bridge_trigger_path"]))
+ try:
+ trigger = json.loads(trigger_path.read_text(encoding="utf-8"))
+ bridge_pid = int(trigger["bridge_pid"])
+ created_at = parse_timestamp(str(trigger["created_at"]))
+ except (FileNotFoundError, OSError, ValueError, TypeError, KeyError, json.JSONDecodeError) as error:
+ raise ArchiveError("formal cron bridge trigger is invalid") from error
+ if trigger.get("schema") != "multica.storage-cron-trigger.v1" or not trigger.get("token"):
+ raise ArchiveError("formal cron bridge trigger schema is invalid")
+ try:
+ receipt = json.loads(Path(str(config["cron_bridge_receipt_path"])).read_text(encoding="utf-8"))
+ except (FileNotFoundError, OSError, json.JSONDecodeError):
+ receipt = {}
+ if receipt.get("token") == trigger["token"]:
+ raise ArchiveError("formal cron bridge token was already consumed")
+ age = (utc_now() - created_at).total_seconds()
+ if age < 0 or age > 120:
+ raise ArchiveError("formal cron bridge trigger is stale")
+ command = subprocess.run(
+ ["/bin/ps", "-p", str(bridge_pid), "-o", "command="],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if command.returncode != 0 or "retention_cron_bridge.py" not in command.stdout:
+ raise ArchiveError("formal cron bridge process is not alive")
+ lineage = process_ancestry(bridge_pid)
+ if not any(Path(str(item["command"])).name == "cron" for item in lineage):
+ raise ArchiveError("formal cron bridge has no live cron ancestor")
+ return trigger, lineage
+
+
+def write_cron_bridge_receipt(
+ config: Dict[str, Any],
+ *,
+ token: Optional[str],
+ status: str,
+ error: Optional[str] = None,
+) -> None:
+ if not token:
+ return
+ atomic_write_json(
+ Path(str(config["cron_bridge_receipt_path"])),
+ {"token": token, "status": status, "recorded_at": utc_now().isoformat(), "error": error},
+ )
+
+
+def atomic_write_failure_report(config: Dict[str, Any], message: str) -> None:
+ path = Path(str(config["report_path"]))
+ last_success_at: Optional[str] = None
+ try:
+ previous = json.loads(path.read_text(encoding="utf-8"))
+ if isinstance(previous, dict):
+ if previous.get("status") == "green":
+ last_success_at = str(previous.get("recorded_at") or "") or None
+ else:
+ last_success_at = previous.get("last_success_at")
+ except (FileNotFoundError, OSError, json.JSONDecodeError):
+ pass
+ atomic_write_json(
+ path,
+ {
+ "schema": "multica.storage-retention-run.v1",
+ "status": "red",
+ "failed_at": utc_now().isoformat(),
+ "last_success_at": last_success_at,
+ "error": message,
+ },
+ )
+
+
+def run_worker(config: Dict[str, Any]) -> Dict[str, Any]:
+ ancestry = process_ancestry()
+ trigger: Optional[Dict[str, Any]] = None
+ cron_lineage: List[Dict[str, Any]] = []
+ if config.get("require_cron_lineage", True):
+ trigger, cron_lineage = verify_cron_bridge(config)
+ config["_verified_cron_token"] = str(trigger["token"])
+ write_cron_bridge_receipt(config, token=str(trigger["token"]), status="running")
+ electron_audit = maybe_run_electron_updater_audit(
+ config,
+ invocation_source="verified-cron-launchd-bridge" if trigger else "manual-worker",
+ )
+ if electron_audit.get("status") == "attention":
+ send_alert(
+ config,
+ "ST-1 Electron updater audit needs review: %d candidates, %.3f GiB; report: %s"
+ % (
+ int(electron_audit.get("candidate_count") or 0),
+ int(electron_audit.get("total_bytes") or 0) / GIB,
+ str(electron_audit.get("report_path") or "missing"),
+ ),
+ )
+ elif electron_audit.get("status") == "red":
+ raise ArchiveError(
+ "ST-1 Electron updater audit was incomplete: %s"
+ % "; ".join(str(value) for value in electron_audit.get("errors") or ["unknown error"])
+ )
+ if config.get("delete_source", False):
+ raise ArchiveError("source deletion requires a producer lease and remains disabled")
+ external_path = Path(str(config["external_path"]))
+ archive_root = Path(str(config["archive_root"]))
+ archive_root.mkdir(parents=True, exist_ok=True)
+ volume_guard = ExternalVolumeGuard(
+ external_path,
+ archive_root,
+ expected_uuid=str(config["external_volume_uuid"]),
+ min_free_bytes=int(float(config.get("external_min_free_gib", 100)) * GIB),
+ )
+ volume_guard.check()
+ canary_root = Path(str(config.get("canary_root") or external_path))
+ canary_root.mkdir(parents=True, exist_ok=True)
+ canary_guard = ExternalVolumeGuard(
+ external_path,
+ canary_root,
+ expected_uuid=str(config["external_volume_uuid"]),
+ min_free_bytes=int(float(config.get("external_min_free_gib", 100)) * GIB),
+ )
+ canary_guard.check()
+ canary = Canary(
+ canary_root,
+ expected_uuid=str(config["external_volume_uuid"]),
+ min_free_bytes=int(float(config.get("external_min_free_gib", 100)) * GIB),
+ volume_path=external_path,
+ postflight=canary_guard.check,
+ ).run()
+ evaluator = GCEvaluator(
+ MulticaIssueClient(),
+ retention_seconds=int(float(config.get("retention_days", 7)) * 86400),
+ recent_write_seconds=int(config.get("recent_write_seconds", 86400)),
+ )
+ candidate_paths = discover_candidates(Path(str(item)) for item in config.get("workspace_roots", []))
+ with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
+ candidates = list(executor.map(evaluator.evaluate, candidate_paths))
+ report: Dict[str, Any] = {
+ "schema": "multica.storage-retention-run.v1",
+ "status": "green",
+ "recorded_at": utc_now().isoformat(),
+ "pid": os.getpid(),
+ "parent_pid": os.getppid(),
+ "invocation_source": "verified-cron-launchd-bridge" if trigger else "manual",
+ "process_ancestry": ancestry,
+ "cron_bridge_ancestry": cron_lineage,
+ "cron_trigger_token": trigger.get("token") if trigger else None,
+ "canary": canary,
+ "electron_updater_audit": electron_audit,
+ "gc_mode": "dry-run",
+ "gc_candidates": candidates,
+ "eligible_count": sum(1 for candidate in candidates if candidate["eligible"]),
+ "archive_enabled": bool(config.get("archive_enabled", False)),
+ "delete_source": bool(config.get("delete_source", False)),
+ "archives": [],
+ }
+ if config.get("archive_enabled", False):
+ manager = ArchiveManager(archive_root, preflight=volume_guard.check)
+ approved = {str(value) for value in config.get("approved_candidates", [])}
+ consumed = consumed_approval_tokens(archive_root)
+ for candidate in candidates:
+ candidate_path = Path(str(candidate["path"]))
+ token = str(candidate.get("approval_token") or "")
+ if not candidate["eligible"] or token not in approved or token in consumed:
+ continue
+ fresh = evaluator.evaluate(candidate_path)
+ if not fresh["eligible"] or fresh.get("approval_token") != candidate.get("approval_token"):
+ continue
+ report["archives"].append(
+ manager.archive(
+ candidate_path,
+ candidate_path.name,
+ delete_source=False,
+ approval_token=token,
+ )
+ )
+ consumed.add(token)
+ atomic_write_json(Path(str(config["report_path"])), report)
+ if trigger:
+ write_cron_bridge_receipt(config, token=str(trigger["token"]), status="green")
+ return report
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--config", required=True)
+ parser.add_argument(
+ "--electron-audit-only",
+ action="store_true",
+ help="force the read-only ST-1 updater audit without cron lineage or external-volume checks",
+ )
+ args = parser.parse_args()
+ config: Dict[str, Any] = {}
+ audit_only_report: Optional[Dict[str, Any]] = None
+ try:
+ config = json.loads(Path(args.config).read_text(encoding="utf-8"))
+ with SingleInstanceLock(Path(str(config["lock_path"]))):
+ if args.electron_audit_only:
+ audit_only_report = maybe_run_electron_updater_audit(
+ config,
+ force=True,
+ invocation_source="manual-audit-only",
+ )
+ else:
+ report = run_worker(config)
+ except BlockingIOError:
+ message = "storage retention worker skipped: another owner holds the single-instance lock"
+ for action in (
+ lambda: atomic_write_failure_report(config, message),
+ lambda: write_cron_bridge_receipt(
+ config,
+ token=str(config.get("_verified_cron_token") or "") or None,
+ status="red",
+ error=message,
+ ),
+ lambda: send_alert(config, message),
+ ):
+ try:
+ action()
+ except Exception:
+ pass
+ print(json.dumps({"status": "locked", "error": message}), file=sys.stderr)
+ return 75
+ except Exception as error:
+ message = "storage retention worker failed closed: %s" % error
+ for action in (
+ lambda: atomic_write_failure_report(config, message),
+ lambda: write_cron_bridge_receipt(
+ config,
+ token=str(config.get("_verified_cron_token") or "") or None,
+ status="red",
+ error=message,
+ ),
+ lambda: send_alert(config, message),
+ ):
+ try:
+ action()
+ except Exception:
+ pass
+ print(json.dumps({"status": "red", "error": message}, ensure_ascii=False), file=sys.stderr)
+ return 1
+ if audit_only_report is not None:
+ print(
+ json.dumps(
+ {
+ "status": audit_only_report["status"],
+ "audit_month": audit_only_report["audit_month"],
+ "candidate_count": audit_only_report["candidate_count"],
+ "stale_candidate_count": audit_only_report["stale_candidate_count"],
+ "total_bytes": audit_only_report["total_bytes"],
+ "report_path": audit_only_report["report_path"],
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ )
+ return 1 if audit_only_report["status"] == "red" else 0
+ print(
+ json.dumps(
+ {
+ "status": report["status"],
+ "recorded_at": report["recorded_at"],
+ "eligible_count": report["eligible_count"],
+ "archive_count": len(report["archives"]),
+ "report_path": str(config["report_path"]),
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/storage-governance/storage_guard.py b/scripts/storage-governance/storage_guard.py
new file mode 100644
index 00000000000..ca22ef7d649
--- /dev/null
+++ b/scripts/storage-governance/storage_guard.py
@@ -0,0 +1,753 @@
+#!/usr/bin/env python3
+"""Minute-level storage guard for a Multica runtime host.
+
+The guard is deliberately small: sample, classify with hysteresis, pause new
+Multica admissions, stop explicitly listed non-critical jobs, and alert. It
+never deletes or archives data.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import re
+import signal
+import subprocess
+import tempfile
+import time
+from dataclasses import asdict, dataclass, replace
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any, Dict, Iterable, List, Optional, Tuple
+
+
+GIB = 1024**3
+
+
+@dataclass(frozen=True)
+class Sample:
+ recorded_at: str
+ internal_free_bytes: int
+ external_free_bytes: Optional[int]
+ swap_used_bytes: int
+ active_task_count: int
+ daemon_pid: Optional[int]
+ daemon_status: str
+ admission_pause_owners: Optional[Tuple[str, ...]] = None
+ shadow_runs_bytes: Optional[int] = None
+ workspace_total_bytes: Optional[int] = None
+ workspace_inflight_bytes: Optional[int] = None
+ workspace_gc_eligible_bytes: Optional[int] = None
+ workspace_gc_backlog_bytes: Optional[int] = None
+ workspace_unclassified_bytes: Optional[int] = None
+ cursor_bytes: Optional[int] = None
+ logs_bytes: Optional[int] = None
+
+
+class CommandRunner:
+ def run(self, argv: List[str], *, tolerate_failure: bool = False) -> Tuple[int, str, str]:
+ try:
+ proc = subprocess.run(argv, capture_output=True, text=True, check=False, timeout=10)
+ except subprocess.TimeoutExpired as error:
+ if tolerate_failure:
+ return 124, "", "command timed out after 10 seconds"
+ raise RuntimeError("command timed out: %s" % argv[0]) from error
+ if proc.returncode != 0 and not tolerate_failure:
+ detail = (proc.stderr or proc.stdout).strip()
+ raise RuntimeError("command failed (%d): %s: %s" % (proc.returncode, argv[0], detail))
+ return proc.returncode, proc.stdout, proc.stderr
+
+ def signal(self, pid: int, sig: int) -> None:
+ os.kill(pid, sig)
+
+
+class SystemCollector:
+ def __init__(self, config: Dict[str, Any], runner: CommandRunner):
+ self.config = config
+ self.runner = runner
+
+ @staticmethod
+ def free_bytes(path: str) -> int:
+ stat = os.statvfs(path)
+ return stat.f_bavail * stat.f_frsize
+
+ def swap_used_bytes(self) -> int:
+ code, stdout, stderr = self.runner.run(["/usr/sbin/sysctl", "vm.swapusage"])
+ if code != 0:
+ raise RuntimeError("sysctl vm.swapusage failed: " + (stderr or stdout).strip())
+ match = re.search(r"\bused\s*=\s*([0-9.]+)([KMGT])", stdout)
+ if not match:
+ raise RuntimeError("could not parse vm.swapusage")
+ units = {"K": 1024, "M": 1024**2, "G": 1024**3, "T": 1024**4}
+ return int(float(match.group(1)) * units[match.group(2)])
+
+ def daemon_status(self) -> Dict[str, Any]:
+ code, stdout, _ = self.runner.run(
+ ["multica", "daemon", "status", "--output", "json"],
+ tolerate_failure=True,
+ )
+ if code != 0:
+ return {"status": "stopped", "active_task_count": 0, "pid": None}
+ try:
+ value = json.loads(stdout)
+ except json.JSONDecodeError:
+ return {"status": "unknown", "active_task_count": 0, "pid": None}
+ return value if isinstance(value, dict) else {"status": "unknown"}
+
+ @staticmethod
+ def directory_size(path: Path, *, deadline: Optional[float] = None) -> Optional[int]:
+ if deadline is not None and time.monotonic() >= deadline:
+ return None
+ if not path.is_dir() or path.is_symlink():
+ return None
+ total = 0
+ try:
+ for current, dirnames, filenames in os.walk(str(path), topdown=True, followlinks=False):
+ if deadline is not None and time.monotonic() >= deadline:
+ return None
+ current_path = Path(current)
+ dirnames[:] = [name for name in dirnames if not (current_path / name).is_symlink()]
+ for name in filenames:
+ if deadline is not None and time.monotonic() >= deadline:
+ return None
+ candidate = current_path / name
+ if not candidate.is_symlink():
+ total += candidate.stat().st_size
+ except OSError:
+ return None
+ return total
+
+ def growth_metrics(self) -> Dict[str, Optional[int]]:
+ deadline = time.monotonic() + max(0.0, float(self.config.get("growth_scan_budget_seconds", 5)))
+
+ def scan(path: object) -> Optional[int]:
+ return self.directory_size(Path(str(path)), deadline=deadline)
+
+ shadow = scan(self.config["shadow_runs_path"]) if self.config.get("shadow_runs_path") else None
+ cursor = scan(self.config["cursor_path"]) if self.config.get("cursor_path") else None
+ log_values = [scan(path) for path in self.config.get("logs_paths", [])]
+ logs = sum(value for value in log_values if value is not None) if log_values and all(value is not None for value in log_values) else None
+ workspace_values = [scan(path) for path in self.config.get("workspace_roots", [])]
+ workspace_total = (
+ sum(value for value in workspace_values if value is not None)
+ if workspace_values and all(value is not None for value in workspace_values)
+ else None
+ )
+
+ eligible = inflight = backlog = None
+ report_path = self.config.get("retention_report_path")
+ if report_path:
+ try:
+ report = json.loads(Path(str(report_path)).read_text(encoding="utf-8"))
+ recorded_at = parse_timestamp(str(report["recorded_at"]))
+ maximum_age = float(self.config.get("retention_report_max_age_seconds", 1800))
+ report_age = (datetime.now(timezone.utc) - recorded_at).total_seconds()
+ if report.get("status") == "green" and 0 <= report_age <= maximum_age:
+ eligible_value = inflight_value = backlog_value = 0
+ for candidate in report.get("gc_candidates") or []:
+ size = int((candidate.get("details") or {}).get("size_bytes") or 0)
+ reasons = " ".join(str(value) for value in candidate.get("reasons") or [])
+ if candidate.get("eligible"):
+ eligible_value += size
+ elif any(
+ marker in reasons
+ for marker in ("not terminal", "nonterminal child", "active run", "open file", "recent write")
+ ):
+ inflight_value += size
+ elif float((candidate.get("details") or {}).get("age_seconds") or 0) >= float(self.config.get("retention_days", 7)) * 86400:
+ backlog_value += size
+ eligible, inflight, backlog = eligible_value, inflight_value, backlog_value
+ except (KeyError, OSError, ValueError, TypeError, json.JSONDecodeError):
+ pass
+ unclassified = None
+ if workspace_total is not None and eligible is not None and inflight is not None and backlog is not None:
+ categorized = eligible + inflight + backlog
+ if categorized <= workspace_total:
+ unclassified = workspace_total - categorized
+ else:
+ eligible = inflight = backlog = None
+ return {
+ "shadow_runs_bytes": shadow,
+ "workspace_total_bytes": workspace_total,
+ "workspace_inflight_bytes": inflight,
+ "workspace_gc_eligible_bytes": eligible,
+ "workspace_gc_backlog_bytes": backlog,
+ "workspace_unclassified_bytes": unclassified,
+ "cursor_bytes": cursor,
+ "logs_bytes": logs,
+ }
+
+ def collect(self) -> Sample:
+ daemon = self.daemon_status()
+ external_free: Optional[int]
+ try:
+ external_free = self.free_bytes(str(self.config["external_path"]))
+ except OSError:
+ external_free = None
+ pid = daemon.get("pid")
+ owners_value = daemon.get("admission_pause_owners")
+ owners = tuple(str(value) for value in owners_value) if isinstance(owners_value, list) else None
+ return Sample(
+ recorded_at=datetime.now(timezone.utc).isoformat(),
+ internal_free_bytes=self.free_bytes(str(self.config["internal_path"])),
+ external_free_bytes=external_free,
+ swap_used_bytes=self.swap_used_bytes(),
+ active_task_count=int(daemon.get("active_task_count") or 0),
+ daemon_pid=int(pid) if pid else None,
+ daemon_status=str(daemon.get("status") or "unknown"),
+ admission_pause_owners=owners,
+ )
+
+ def enrich(self, sample: Sample) -> Sample:
+ cache_path = Path(
+ str(
+ self.config.get("growth_cache_path")
+ or Path(str(self.config["state_path"])).with_name("growth-metrics-cache.json")
+ )
+ )
+ interval = float(self.config.get("growth_scan_interval_seconds", 900))
+ now = datetime.now(timezone.utc)
+ cached = load_json(cache_path, {})
+ try:
+ cached_age = (now - parse_timestamp(str(cached["recorded_at"]))).total_seconds()
+ cached_values = cached["values"]
+ except (KeyError, TypeError, ValueError):
+ cached_age = float("inf")
+ cached_values = {}
+ if cached_age < interval:
+ return replace(sample, **cached_values)
+ values = self.growth_metrics()
+ if any(value is not None for value in values.values()):
+ atomic_write_json(cache_path, {"recorded_at": now.isoformat(), "values": values})
+ return replace(sample, **values)
+ if cached_age < interval * 2:
+ return replace(sample, **cached_values)
+ return replace(sample, **values)
+
+
+def classify_level(
+ free_bytes: int,
+ *,
+ previous: int,
+ level1: int,
+ level1_clear: int,
+ level2: int,
+ level2_clear: int,
+) -> int:
+ if free_bytes <= level2:
+ return 2
+ if previous >= 2 and free_bytes < level2_clear:
+ return 2
+ if free_bytes <= level1:
+ return 1
+ if previous >= 1 and free_bytes < level1_clear:
+ return 1
+ return 0
+
+
+def atomic_write_json(path: Path, value: Dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ descriptor, temp_name = tempfile.mkstemp(prefix=".%s." % path.name, dir=str(path.parent))
+ try:
+ with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+ json.dump(value, handle, ensure_ascii=False, indent=2, sort_keys=True)
+ handle.write("\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temp_name, path)
+ finally:
+ try:
+ os.unlink(temp_name)
+ except FileNotFoundError:
+ pass
+
+
+def append_jsonl(path: Path, value: Dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(value, ensure_ascii=False, sort_keys=True) + "\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+
+
+def parse_timestamp(raw: str) -> datetime:
+ return datetime.fromisoformat(raw.replace("Z", "+00:00"))
+
+
+def build_capacity_report(
+ samples: Iterable[Dict[str, Any]],
+ *,
+ safety_floor_bytes: int,
+ burst_reserve_bytes: int,
+ minimum_hours: float,
+ expected_interval_seconds: float = 3600,
+ maximum_window_hours: float = 72,
+ minimum_coverage: float = 0.8,
+ required_growth_fields: Iterable[str] = (),
+ required_field_max_gap_seconds: float = 1800,
+ discarded_sample_count: int = 0,
+ external_safety_floor_bytes: int = 100 * GIB,
+) -> Dict[str, Any]:
+ ordered_all = sorted(samples, key=lambda item: parse_timestamp(str(item["recorded_at"])))
+ if ordered_all:
+ cutoff = parse_timestamp(str(ordered_all[-1]["recorded_at"])) - timedelta(hours=maximum_window_hours)
+ ordered = [item for item in ordered_all if parse_timestamp(str(item["recorded_at"])) >= cutoff]
+ else:
+ ordered = []
+ report: Dict[str, Any] = {
+ "schema": "multica.storage-capacity.v1",
+ "status": "INCONCLUSIVE",
+ "sample_count": len(ordered),
+ "discarded_sample_count": discarded_sample_count,
+ "observation_hours": 0.0,
+ "coverage_ratio": 0.0,
+ "maximum_gap_seconds": None,
+ "p95_growth_bytes_per_hour": None,
+ "peak_growth_bytes_per_hour": None,
+ "category_p95_growth_bytes_per_hour": {},
+ "field_coverage_ratio": {},
+ "field_maximum_gap_seconds": {},
+ "safety_floor_bytes": safety_floor_bytes,
+ "burst_reserve_bytes": burst_reserve_bytes,
+ "days_remaining": None,
+ "external_safety_floor_bytes": external_safety_floor_bytes,
+ "external_days_remaining": None,
+ }
+ if len(ordered) < 2:
+ return report
+ start = parse_timestamp(str(ordered[0]["recorded_at"]))
+ end = parse_timestamp(str(ordered[-1]["recorded_at"]))
+ observation_hours = max(0.0, (end - start).total_seconds() / 3600)
+ report["observation_hours"] = observation_hours
+ gaps = [
+ (parse_timestamp(str(current["recorded_at"])) - parse_timestamp(str(previous["recorded_at"]))).total_seconds()
+ for previous, current in zip(ordered, ordered[1:])
+ ]
+ maximum_gap = max(gaps) if gaps else None
+ coverage = min(1.0, ((len(ordered) - 1) * expected_interval_seconds) / max(1.0, (end - start).total_seconds()))
+ report["coverage_ratio"] = coverage
+ report["maximum_gap_seconds"] = maximum_gap
+ minimum_samples = math.ceil(minimum_hours * 3600 / expected_interval_seconds * minimum_coverage) + 1
+ required_fields = list(required_growth_fields)
+ fields_ready = True
+ for field in required_fields:
+ available = [item for item in ordered if item.get(field) is not None]
+ coverage_value = len(available) / len(ordered)
+ report["field_coverage_ratio"][field] = coverage_value
+ field_times = [parse_timestamp(str(item["recorded_at"])) for item in available]
+ field_gaps = [(current - previous).total_seconds() for previous, current in zip(field_times, field_times[1:])]
+ field_maximum_gap = max(field_gaps) if field_gaps else None
+ report["field_maximum_gap_seconds"][field] = field_maximum_gap
+ latest_age = (end - field_times[-1]).total_seconds() if field_times else float("inf")
+ if (
+ coverage_value < minimum_coverage
+ or field_maximum_gap is None
+ or field_maximum_gap > required_field_max_gap_seconds
+ or latest_age > required_field_max_gap_seconds
+ ):
+ fields_ready = False
+ if (
+ observation_hours < minimum_hours
+ or len(ordered) < minimum_samples
+ or coverage < minimum_coverage
+ or maximum_gap is None
+ or maximum_gap > expected_interval_seconds * 3
+ or not fields_ready
+ ):
+ return report
+
+ hourly_growth: Dict[str, float] = {}
+ hourly_categories: Dict[str, Dict[str, float]] = {field: {} for field in required_fields}
+ for previous, current in zip(ordered, ordered[1:]):
+ previous_at = parse_timestamp(str(previous["recorded_at"]))
+ current_at = parse_timestamp(str(current["recorded_at"]))
+ elapsed_seconds = (current_at - previous_at).total_seconds()
+ if elapsed_seconds <= 0 or elapsed_seconds > expected_interval_seconds * 3:
+ continue
+ bucket = current_at.replace(minute=0, second=0, microsecond=0).isoformat()
+ delta = int(previous["internal_free_bytes"]) - int(current["internal_free_bytes"])
+ hourly_growth[bucket] = hourly_growth.get(bucket, 0.0) + delta
+ for field in required_fields:
+ if previous.get(field) is None or current.get(field) is None:
+ continue
+ if field.endswith("_free_bytes"):
+ category_delta = int(previous[field]) - int(current[field])
+ else:
+ category_delta = int(current[field]) - int(previous[field])
+ values = hourly_categories[field]
+ values[bucket] = values.get(bucket, 0.0) + category_delta
+ growth_rates = [max(0.0, value) for value in hourly_growth.values()]
+ if not growth_rates:
+ return report
+
+ sorted_rates = sorted(growth_rates)
+ p95 = sorted_rates[max(0, math.ceil(0.95 * len(sorted_rates)) - 1)]
+ report["status"] = "READY"
+ report["p95_growth_bytes_per_hour"] = p95
+ report["peak_growth_bytes_per_hour"] = max(growth_rates)
+ for field, buckets in hourly_categories.items():
+ values = sorted(max(0.0, value) for value in buckets.values())
+ report["category_p95_growth_bytes_per_hour"][field] = (
+ values[max(0, math.ceil(0.95 * len(values)) - 1)] if values else None
+ )
+ latest_free = int(ordered[-1]["internal_free_bytes"])
+ budget = max(0, latest_free - safety_floor_bytes - burst_reserve_bytes)
+ report["days_remaining"] = (budget / p95 / 24) if p95 > 0 else None
+ external_growth = report["category_p95_growth_bytes_per_hour"].get("external_free_bytes")
+ if external_growth is not None and ordered[-1].get("external_free_bytes") is not None:
+ external_budget = max(0, int(ordered[-1]["external_free_bytes"]) - external_safety_floor_bytes)
+ report["external_days_remaining"] = (
+ external_budget / float(external_growth) / 24 if float(external_growth) > 0 else None
+ )
+ return report
+
+
+def load_json(path: Path, default: Dict[str, Any]) -> Dict[str, Any]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except (FileNotFoundError, json.JSONDecodeError, OSError):
+ return dict(default)
+ return value if isinstance(value, dict) else dict(default)
+
+
+def read_jsonl_with_errors(path: Path) -> Tuple[List[Dict[str, Any]], int]:
+ values: List[Dict[str, Any]] = []
+ discarded = 0
+ try:
+ lines = path.read_text(encoding="utf-8").splitlines()
+ except FileNotFoundError:
+ return values, discarded
+ for line in lines:
+ try:
+ value = json.loads(line)
+ except json.JSONDecodeError:
+ discarded += 1
+ continue
+ if isinstance(value, dict) and "recorded_at" in value and "internal_free_bytes" in value:
+ values.append(value)
+ else:
+ discarded += 1
+ return values, discarded
+
+
+def read_jsonl(path: Path) -> List[Dict[str, Any]]:
+ values, _ = read_jsonl_with_errors(path)
+ return values
+
+
+def validate_config(config: Dict[str, Any]) -> None:
+ required = [
+ "internal_path",
+ "external_path",
+ "level1_free_gib",
+ "level1_clear_gib",
+ "level2_free_gib",
+ "level2_clear_gib",
+ "state_path",
+ "metrics_path",
+ "capacity_report_path",
+ ]
+ missing = [key for key in required if key not in config]
+ if missing:
+ raise ValueError("missing config keys: " + ", ".join(missing))
+ level1 = float(config["level1_free_gib"])
+ level1_clear = float(config["level1_clear_gib"])
+ level2 = float(config["level2_free_gib"])
+ level2_clear = float(config["level2_clear_gib"])
+ if not (0 < level2 < level2_clear <= level1 < level1_clear):
+ raise ValueError("watermarks must satisfy level2 < level2_clear <= level1 < level1_clear")
+
+
+def has_message_id(value: Any) -> bool:
+ if isinstance(value, dict):
+ if value.get("message_id"):
+ return True
+ return any(has_message_id(child) for child in value.values())
+ if isinstance(value, list):
+ return any(has_message_id(child) for child in value)
+ return False
+
+
+class Guard:
+ def __init__(self, config: Dict[str, Any], runner: CommandRunner, collector: Any):
+ validate_config(config)
+ self.config = config
+ self.runner = runner
+ self.collector = collector
+ self.uid = int(config.get("uid", os.getuid()))
+
+ def stop_launchagent(self, label: str, actions: List[str]) -> bool:
+ target = "gui/%d/%s" % (self.uid, label)
+ disable_code, _, disable_error = self.runner.run(["launchctl", "disable", target], tolerate_failure=True)
+ bootout_code, _, bootout_error = self.runner.run(["launchctl", "bootout", target], tolerate_failure=True)
+ print_code, _, print_error = self.runner.run(["launchctl", "print", target], tolerate_failure=True)
+ print_not_found = print_code != 0 and any(
+ marker in (print_error or "").lower()
+ for marker in ("not found", "could not find service", "no such process")
+ )
+ if disable_code == 0 and print_not_found:
+ actions.append("launchagent_stopped:" + label)
+ return True
+ detail = (print_error or bootout_error or disable_error).strip()[:120]
+ actions.append("launchagent_stop_failed:%s:%s" % (label, detail))
+ return False
+
+ def pause_admission(self, sample: Sample, state: Dict[str, Any], actions: List[str]) -> bool:
+ code, stdout, stderr = self.runner.run(
+ ["multica", "daemon", "pause", "--owner", "storage-guard", "--output", "json"],
+ tolerate_failure=True,
+ )
+ try:
+ response = json.loads(stdout)
+ except json.JSONDecodeError:
+ response = {}
+ if code == 0 and response.get("owner_paused") is True and response.get("admission_paused") is True:
+ state["legacy_daemon_sigstopped"] = False
+ state["legacy_daemon_pid"] = None
+ state["legacy_daemon_command"] = None
+ state["admission_pause_owned"] = True
+ actions.append("daemon_admission_paused")
+ return True
+ actions.append("daemon_admission_pause_failed:" + stderr.strip()[:200])
+ actions.append("legacy_sigstop_disabled:unsafe_without_shared_claim_barrier")
+ return False
+
+ def resume_admission(self, state: Dict[str, Any], actions: List[str]) -> bool:
+ released_legacy_fallback = False
+ if (state.get("legacy_daemon_sigstopped") or state.get("legacy_daemon_sigstop_intent")) and state.get("legacy_daemon_pid"):
+ pid = int(state["legacy_daemon_pid"])
+ ps_code, command, _ = self.runner.run(
+ ["/bin/ps", "-p", str(pid), "-o", "command="], tolerate_failure=True
+ )
+ expected = str(state.get("legacy_daemon_command") or "")
+ if ps_code != 0 or not expected or command.strip() != expected:
+ actions.append("legacy_daemon_sigcontinue_failed:pid_identity_mismatch")
+ state["resume_pending"] = True
+ return False
+ self.runner.signal(pid, signal.SIGCONT)
+ state["legacy_daemon_sigstopped"] = False
+ state["legacy_daemon_sigstop_intent"] = False
+ state["legacy_daemon_pid"] = None
+ state["legacy_daemon_command"] = None
+ atomic_write_json(Path(str(self.config["state_path"])), state)
+ actions.append("legacy_daemon_sigcontinued")
+ released_legacy_fallback = True
+ if released_legacy_fallback and not state.get("admission_pause_owned"):
+ state["resume_pending"] = False
+ return True
+ code, stdout, stderr = self.runner.run(
+ ["multica", "daemon", "resume", "--owner", "storage-guard", "--output", "json"],
+ tolerate_failure=True,
+ )
+ try:
+ response = json.loads(stdout)
+ except json.JSONDecodeError:
+ response = {}
+ if code != 0 or response.get("owner_paused") is not False:
+ state["resume_pending"] = True
+ actions.append("daemon_admission_resume_failed:" + stderr.strip()[:200])
+ return False
+ state["admission_pause_owned"] = False
+ state["resume_pending"] = False
+ actions.append("daemon_admission_resumed")
+ return True
+
+ def alert(
+ self,
+ sample: Sample,
+ level: int,
+ state: Dict[str, Any],
+ actions: List[str],
+ reason: str,
+ *,
+ alert_key: str,
+ ) -> None:
+ open_id = str(self.config.get("lark_open_id") or "").strip()
+ if not open_id:
+ actions.append("lark_alert_skipped:no_recipient")
+ return
+ now = parse_timestamp(sample.recorded_at)
+ alert_times = state.setdefault("last_alert_at_by_key", {})
+ previous_raw = alert_times.get(alert_key)
+ if previous_raw:
+ previous = parse_timestamp(str(previous_raw))
+ cooldown = float(self.config.get("alert_cooldown_seconds", 3600))
+ if (now - previous).total_seconds() < cooldown:
+ actions.append("lark_alert_suppressed:cooldown")
+ return
+ text = (
+ "[主机储存熔断 L%d] %s；内置盘可用 %.2f GiB，swap %.2f GiB，活跃任务 %d。"
+ % (
+ level,
+ reason,
+ sample.internal_free_bytes / GIB,
+ sample.swap_used_bytes / GIB,
+ sample.active_task_count,
+ )
+ )
+ code, stdout, _ = self.runner.run(
+ [
+ "lark-cli",
+ "im",
+ "+messages-send",
+ "--as",
+ "bot",
+ "--user-id",
+ open_id,
+ "--text",
+ text,
+ "--format",
+ "json",
+ ],
+ tolerate_failure=True,
+ )
+ try:
+ response = json.loads(stdout)
+ except json.JSONDecodeError:
+ response = {}
+ if code == 0 and has_message_id(response):
+ alert_times[alert_key] = sample.recorded_at
+ actions.append("lark_alert_sent")
+ else:
+ actions.append("lark_alert_failed")
+
+ def run_once(self) -> Dict[str, Any]:
+ sample = self.collector.collect()
+ state_path = Path(str(self.config["state_path"]))
+ metrics_path = Path(str(self.config["metrics_path"]))
+ report_path = Path(str(self.config["capacity_report_path"]))
+ state = load_json(
+ state_path,
+ {
+ "level": 0,
+ "legacy_daemon_sigstopped": False,
+ "legacy_daemon_sigstop_intent": False,
+ "legacy_daemon_pid": None,
+ "admission_pause_owned": False,
+ "resume_pending": False,
+ },
+ )
+ if sample.admission_pause_owners is not None:
+ state["admission_pause_owned"] = "storage-guard" in sample.admission_pause_owners
+ previous = int(state.get("level") or 0)
+ level = classify_level(
+ sample.internal_free_bytes,
+ previous=previous,
+ level1=int(float(self.config["level1_free_gib"]) * GIB),
+ level1_clear=int(float(self.config["level1_clear_gib"]) * GIB),
+ level2=int(float(self.config["level2_free_gib"]) * GIB),
+ level2_clear=int(float(self.config["level2_clear_gib"]) * GIB),
+ )
+ actions: List[str] = []
+ enforcement_ok = True
+
+ if level >= 1:
+ for label in self.config.get("observer_labels", []):
+ enforcement_ok = self.stop_launchagent(str(label), actions) and enforcement_ok
+ enforcement_ok = self.pause_admission(sample, state, actions) and enforcement_ok
+ if not enforcement_ok:
+ self.alert(
+ sample,
+ level,
+ state,
+ actions,
+ "一级低水位执行失败，任务入场可能仍开放",
+ alert_key="level1-enforcement-failed",
+ )
+ elif previous >= 1 or state.get("legacy_daemon_sigstopped") or state.get("legacy_daemon_sigstop_intent") or state.get("resume_pending") or state.get("admission_pause_owned"):
+ enforcement_ok = self.resume_admission(state, actions)
+ if not enforcement_ok:
+ level = max(previous, 1)
+ self.alert(
+ sample,
+ level,
+ state,
+ actions,
+ "低水位恢复失败，将持续重试 storage-guard 自有入场屏障",
+ alert_key="level1-resume-failed",
+ )
+
+ if level >= 2:
+ level2_enforcement_ok = True
+ for label in self.config.get("nonproduction_launchagents", []):
+ level2_enforcement_ok = self.stop_launchagent(str(label), actions) and level2_enforcement_ok
+ enforcement_ok = level2_enforcement_ok and enforcement_ok
+ self.alert(
+ sample,
+ level,
+ state,
+ actions,
+ (
+ "内置盘进入二级低水位，已暂停显式列出的非生产任务"
+ if level2_enforcement_ok
+ else "内置盘进入二级低水位，但非生产任务暂停未完全生效"
+ ),
+ alert_key="level2-internal-low-water",
+ )
+
+ external_min = int(float(self.config.get("external_min_free_gib", 100)) * GIB)
+ if sample.external_free_bytes is None:
+ self.alert(
+ sample,
+ max(level, 1),
+ state,
+ actions,
+ "外置归档卷不可用",
+ alert_key="external-unavailable",
+ )
+ elif sample.external_free_bytes < external_min:
+ self.alert(
+ sample,
+ max(level, 1),
+ state,
+ actions,
+ "外置归档卷低于配置水位 (%.2f GiB)" % (sample.external_free_bytes / GIB),
+ alert_key="external-low-water",
+ )
+
+ if hasattr(self.collector, "enrich"):
+ sample = self.collector.enrich(sample)
+ state.update(
+ {
+ "level": level,
+ "previous_level": previous,
+ "last_sample": asdict(sample),
+ "last_actions": actions,
+ "enforcement_status": "verified" if enforcement_ok else "failed",
+ }
+ )
+ append_jsonl(metrics_path, asdict(sample))
+ samples, discarded = read_jsonl_with_errors(metrics_path)
+ report = build_capacity_report(
+ samples,
+ safety_floor_bytes=int(float(self.config.get("safety_floor_gib", 25)) * GIB),
+ burst_reserve_bytes=int(float(self.config.get("burst_reserve_gib", 10)) * GIB),
+ minimum_hours=float(self.config.get("minimum_observation_hours", 48)),
+ expected_interval_seconds=float(self.config.get("expected_interval_seconds", 60)),
+ maximum_window_hours=float(self.config.get("maximum_observation_hours", 72)),
+ minimum_coverage=float(self.config.get("minimum_sample_coverage", 0.8)),
+ required_growth_fields=self.config.get("required_growth_fields", []),
+ required_field_max_gap_seconds=float(self.config.get("required_field_max_gap_seconds", 1800)),
+ discarded_sample_count=discarded,
+ external_safety_floor_bytes=int(float(self.config.get("external_min_free_gib", 100)) * GIB),
+ )
+ atomic_write_json(report_path, report)
+ atomic_write_json(state_path, state)
+ return {"level": level, "previous_level": previous, "sample": asdict(sample), "actions": actions, "capacity": report}
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Sample and protect a Multica runtime host")
+ parser.add_argument("--config", required=True, type=Path)
+ args = parser.parse_args()
+ config = json.loads(args.config.read_text(encoding="utf-8"))
+ runner = CommandRunner()
+ result = Guard(config, runner, SystemCollector(config, runner)).run_once()
+ print(json.dumps(result, ensure_ascii=False, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/storage-governance/test_retention_cron_bridge.py b/scripts/storage-governance/test_retention_cron_bridge.py
new file mode 100644
index 00000000000..529a009789b
--- /dev/null
+++ b/scripts/storage-governance/test_retention_cron_bridge.py
@@ -0,0 +1,52 @@
+from __future__ import annotations
+
+import fcntl
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from retention_cron_bridge import ( # noqa: E402
+ SingleInstanceLock,
+ launchctl_kickstart_command,
+ receipt_exit_code,
+ record_lock_collision,
+)
+
+
+class CronBridgeTest(unittest.TestCase):
+ def test_kickstart_does_not_force_kill_an_existing_retention_run(self) -> None:
+ command = launchctl_kickstart_command("com.multica.storage-retention", uid=501)
+ self.assertEqual(
+ command,
+ ["/bin/launchctl", "kickstart", "gui/501/com.multica.storage-retention"],
+ )
+ self.assertNotIn("-k", command)
+
+ def test_bridge_single_instance_lock_is_nonblocking(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "bridge.lock"
+ with SingleInstanceLock(path):
+ descriptor = path.open("a+")
+ try:
+ with self.assertRaises(BlockingIOError):
+ fcntl.flock(descriptor.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ finally:
+ descriptor.close()
+
+ def test_running_receipt_keeps_bridge_waiting(self) -> None:
+ self.assertIsNone(receipt_exit_code({"token": "t", "status": "running"}, "t"))
+ self.assertEqual(receipt_exit_code({"token": "t", "status": "green"}, "t"), 0)
+ self.assertEqual(receipt_exit_code({"token": "t", "status": "red"}, "t"), 1)
+
+ def test_lock_collision_writes_machine_readable_alert(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "alerts.jsonl"
+ record_lock_collision(path)
+ self.assertIn("previous bridge is still running", path.read_text())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/scripts/storage-governance/test_retention_worker.py b/scripts/storage-governance/test_retention_worker.py
new file mode 100644
index 00000000000..41366ac21cb
--- /dev/null
+++ b/scripts/storage-governance/test_retention_worker.py
@@ -0,0 +1,675 @@
+from __future__ import annotations
+
+import fcntl
+import io
+import json
+import sys
+import tempfile
+import unittest
+from contextlib import redirect_stdout
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from retention_worker import ( # noqa: E402
+ ArchiveError,
+ ArchiveManager,
+ Canary,
+ ExternalVolumeGuard,
+ GCEvaluator,
+ SingleInstanceLock,
+ _latest_mtime_size_and_unsafe_symlinks,
+ audit_electron_updaters,
+ consumed_approval_tokens,
+ main,
+ maybe_run_electron_updater_audit,
+ run_worker,
+ send_alert,
+ tree_manifest,
+ verify_cron_bridge,
+ write_cron_bridge_receipt,
+)
+
+
+class FakeIssueClient:
+ def __init__(self, issue: dict, runs: list[dict], children: list[dict] | None = None):
+ self.issue = issue
+ self.runs = runs
+ self.children = children or []
+
+ def get_issue(self, issue_id: str) -> dict:
+ return dict(self.issue)
+
+ def get_runs(self, issue_id: str) -> list[dict]:
+ return list(self.runs)
+
+ def get_children(self, issue_id: str) -> list[dict]:
+ return list(self.children)
+
+
+def make_tree(root: Path) -> None:
+ (root / "nested" / "deeper").mkdir(parents=True)
+ (root / "root.txt").write_text("root payload\n", encoding="utf-8")
+ (root / "nested" / "data.bin").write_bytes(bytes(range(64)))
+ (root / "nested" / "deeper" / "empty").write_bytes(b"")
+
+
+class CanaryTest(unittest.TestCase):
+ def test_uuid_is_checked_before_writing_representative_tree(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ destination = Path(tmp)
+ canary = Canary(
+ destination,
+ expected_uuid="expected",
+ min_free_bytes=1,
+ uuid_reader=lambda _: "wrong",
+ free_bytes_reader=lambda _: 10_000,
+ )
+ with self.assertRaises(ArchiveError):
+ canary.run()
+ self.assertFalse((destination / ".multica-storage-canary").exists())
+
+ def test_external_low_water_is_red_before_writing(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ destination = Path(tmp)
+ with self.assertRaises(ArchiveError):
+ Canary(
+ destination,
+ expected_uuid="volume-uuid",
+ min_free_bytes=100,
+ uuid_reader=lambda _: "volume-uuid",
+ free_bytes_reader=lambda _: 99,
+ ).run()
+ self.assertFalse((destination / ".multica-storage-canary").exists())
+
+ def test_writes_and_verifies_nested_tree_with_counts_bytes_and_hashes(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ destination = Path(tmp)
+ result = Canary(
+ destination,
+ expected_uuid="volume-uuid",
+ min_free_bytes=1,
+ uuid_reader=lambda _: "volume-uuid",
+ free_bytes_reader=lambda _: 10_000,
+ ).run()
+ self.assertEqual(result["status"], "green")
+ self.assertGreaterEqual(result["manifest"]["file_count"], 3)
+ self.assertGreater(result["manifest"]["total_bytes"], 0)
+ self.assertGreaterEqual(len(result["manifest"]["sample_hashes"]), 2)
+ self.assertTrue(Path(result["path"], "nested", "deeper", "empty").is_file())
+
+ def test_postflight_volume_rebind_failure_prevents_green_marker(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ destination = Path(tmp)
+
+ def rebound() -> dict:
+ raise ArchiveError("archive volume UUID changed before commit")
+
+ with self.assertRaisesRegex(ArchiveError, "UUID changed"):
+ Canary(
+ destination,
+ expected_uuid="volume-uuid",
+ min_free_bytes=1,
+ uuid_reader=lambda _: "volume-uuid",
+ free_bytes_reader=lambda _: 10_000,
+ postflight=rebound,
+ ).run()
+ self.assertEqual(list(destination.glob("**/CANARY.json")), [])
+
+
+class ArchiveTransactionTest(unittest.TestCase):
+ def test_manifest_hashes_every_file_not_only_samples(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ for index in range(17):
+ (root / ("f%02d.bin" % index)).write_bytes(bytes([index]) * 32)
+ before = tree_manifest(root)
+ target = root / "f08.bin"
+ stat = target.stat()
+ target.write_bytes(b"x" * 32)
+ import os
+
+ os.utime(target, ns=(stat.st_atime_ns, stat.st_mtime_ns))
+ after = tree_manifest(root)
+ self.assertEqual(before["files"], after["files"])
+ self.assertNotEqual(before["content_hashes"], after["content_hashes"])
+
+ def test_source_delete_is_refused_without_a_shared_producer_lease(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ source = root / "source"
+ destination = root / "archive"
+ source.mkdir()
+ make_tree(source)
+
+ with self.assertRaisesRegex(ArchiveError, "producer lease"):
+ ArchiveManager(destination).archive(source, "candidate", delete_source=True)
+ self.assertTrue(source.exists())
+
+ def test_every_injected_failure_preserves_source(self) -> None:
+ for phase in ("after_copy", "after_verify", "after_rename", "after_complete"):
+ with self.subTest(phase=phase), tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ source = root / "source"
+ source.mkdir()
+ make_tree(source)
+ with self.assertRaises(ArchiveError):
+ ArchiveManager(root / "archive", fail_at=phase).archive(
+ source, "candidate", delete_source=False
+ )
+ self.assertTrue(source.exists())
+ self.assertEqual(tree_manifest(source)["file_count"], 3)
+
+ def test_committed_archive_is_rehashed_after_complete_marker(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ source = root / "source"
+ source.mkdir()
+ make_tree(source)
+
+ def corrupt_committed(final: Path) -> None:
+ (final / "root.txt").write_text("corrupt\n", encoding="utf-8")
+
+ with self.assertRaisesRegex(ArchiveError, "committed archive changed"):
+ ArchiveManager(root / "archive").archive(
+ source,
+ "candidate",
+ delete_source=False,
+ post_commit_hook=corrupt_committed,
+ )
+ self.assertTrue(source.exists())
+ self.assertEqual(list((root / "archive").glob("*/COMPLETE.json")), [])
+ self.assertEqual(consumed_approval_tokens(root / "archive"), set())
+
+ def test_single_instance_lock_is_nonblocking(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ lock_path = Path(tmp) / "worker.lock"
+ with SingleInstanceLock(lock_path):
+ descriptor = lock_path.open("a+")
+ try:
+ with self.assertRaises(BlockingIOError):
+ fcntl.flock(descriptor.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ finally:
+ descriptor.close()
+
+
+class GCGateTest(unittest.TestCase):
+ def make_candidate(self, root: Path, *, completed: datetime) -> tuple[Path, str, str]:
+ workspace_id = "workspace-uuid"
+ issue_id = "issue-uuid"
+ task_id = "abcdef12-0000-0000-0000-000000000000"
+ candidate = root / workspace_id / task_id[:8]
+ (candidate / "workdir" / ".multica").mkdir(parents=True)
+ (candidate / ".gc_meta.json").write_text(
+ json.dumps(
+ {
+ "kind": "issue",
+ "issue_id": issue_id,
+ "workspace_id": workspace_id,
+ "completed_at": completed.isoformat(),
+ }
+ ),
+ encoding="utf-8",
+ )
+ (candidate / "workdir" / ".multica" / "daemon_task_context.json").write_text(
+ json.dumps({"managed_by": "multica-daemon-task", "issue_id": issue_id}),
+ encoding="utf-8",
+ )
+ (candidate / "payload.txt").write_text("payload", encoding="utf-8")
+ old = completed.timestamp()
+ for path in sorted(candidate.rglob("*"), reverse=True):
+ path.touch() if path.is_file() else None
+ if path.exists():
+ import os
+
+ os.utime(path, (old, old), follow_symlinks=False)
+ import os
+
+ os.utime(candidate, (old, old))
+ return candidate, issue_id, task_id
+
+ def test_all_gates_must_pass_for_dry_run_candidate(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ now = datetime(2026, 8, 10, tzinfo=timezone.utc)
+ completed = now - timedelta(days=8)
+ candidate, issue_id, task_id = self.make_candidate(Path(tmp), completed=completed)
+ client = FakeIssueClient(
+ {"id": issue_id, "workspace_id": "workspace-uuid", "status": "done", "metadata": {}},
+ [
+ {
+ "id": task_id,
+ "issue_id": issue_id,
+ "workspace_id": "workspace-uuid",
+ "status": "completed",
+ "completed_at": completed.isoformat(),
+ "work_dir": str(candidate / "workdir"),
+ }
+ ],
+ [{"status": "done"}],
+ )
+ result = GCEvaluator(
+ client,
+ now=lambda: now,
+ open_file_checker=lambda _: False,
+ retention_seconds=7 * 86400,
+ recent_write_seconds=3600,
+ ).evaluate(candidate)
+ self.assertTrue(result["eligible"])
+ self.assertEqual(result["reasons"], [])
+ self.assertEqual(result["run_id"], task_id)
+ self.assertEqual(len(result["approval_token"]), 64)
+
+ client.runs.append({"id": "other", "status": "dispatched", "work_dir": "/other"})
+ rejected = GCEvaluator(
+ client,
+ now=lambda: now,
+ open_file_checker=lambda _: False,
+ retention_seconds=7 * 86400,
+ recent_write_seconds=3600,
+ ).evaluate(candidate)
+ self.assertFalse(rejected["eligible"])
+ self.assertIn("active run", " ".join(rejected["reasons"]))
+
+ def test_nonterminal_pin_open_file_recent_write_and_identity_mismatch_fail_closed(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ now = datetime(2026, 8, 10, tzinfo=timezone.utc)
+ completed = now - timedelta(days=8)
+ candidate, issue_id, task_id = self.make_candidate(Path(tmp), completed=completed)
+ client = FakeIssueClient(
+ {
+ "id": issue_id,
+ "workspace_id": "different-workspace",
+ "status": "in_review",
+ "metadata": {"gc_pin": True},
+ },
+ [{"id": task_id, "status": "running", "work_dir": str(candidate / "wrong")}],
+ [{"status": "todo"}],
+ )
+ (candidate / "recent.txt").write_text("new", encoding="utf-8")
+ import os
+
+ os.utime(candidate / "recent.txt", (now.timestamp(), now.timestamp()))
+ os.symlink(str(Path(tmp).parent / "outside"), candidate / "outside-link")
+ result = GCEvaluator(
+ client,
+ now=lambda: now,
+ open_file_checker=lambda _: True,
+ retention_seconds=7 * 86400,
+ recent_write_seconds=3600,
+ ).evaluate(candidate)
+ self.assertFalse(result["eligible"])
+ reasons = " ".join(result["reasons"])
+ for expected in (
+ "terminal",
+ "pin",
+ "child",
+ "open file",
+ "recent write",
+ "identity",
+ "active run",
+ "out-of-bound symlink",
+ ):
+ self.assertIn(expected, reasons)
+
+
+class ExternalVolumeGuardTest(unittest.TestCase):
+ def test_binds_archive_root_device_uuid_and_candidate_budget(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ external = Path(tmp) / "external"
+ archive = external / "archive"
+ archive.mkdir(parents=True)
+ guard = ExternalVolumeGuard(
+ external,
+ archive,
+ expected_uuid="volume-uuid",
+ min_free_bytes=100,
+ uuid_reader=lambda _: "volume-uuid",
+ free_bytes_reader=lambda _: 1210,
+ )
+ self.assertEqual(guard.check(100)["required_reserve_bytes"], 210)
+
+ outside = Path(tmp) / "outside"
+ outside.mkdir()
+ with self.assertRaises(ArchiveError):
+ ExternalVolumeGuard(
+ external,
+ outside,
+ expected_uuid="volume-uuid",
+ min_free_bytes=1,
+ uuid_reader=lambda _: "volume-uuid",
+ free_bytes_reader=lambda _: 100,
+ ).check()
+
+ def test_size_scan_counts_regular_file_payload(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ (root / "payload.bin").write_bytes(b"x" * 4096)
+ _, total_bytes, _ = _latest_mtime_size_and_unsafe_symlinks(root)
+ self.assertGreaterEqual(total_bytes, 4096)
+
+ def test_consumed_approval_tokens_are_read_from_complete_markers(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ archive = Path(tmp)
+ completed = archive / "candidate-1"
+ completed.mkdir()
+ (completed / "COMPLETE.json").write_text(
+ json.dumps({"approval_token": "one-time-token"}),
+ encoding="utf-8",
+ )
+ self.assertEqual(consumed_approval_tokens(archive), {"one-time-token"})
+
+
+class CronBridgeVerificationTest(unittest.TestCase):
+ def make_config(self, root: Path, token: str = "fresh-token") -> dict:
+ trigger = root / "trigger.json"
+ trigger.write_text(
+ json.dumps(
+ {
+ "schema": "multica.storage-cron-trigger.v1",
+ "token": token,
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "bridge_pid": 4321,
+ }
+ ),
+ encoding="utf-8",
+ )
+ return {
+ "cron_bridge_trigger_path": str(trigger),
+ "cron_bridge_receipt_path": str(root / "receipt.json"),
+ }
+
+ @mock.patch("retention_worker.process_ancestry")
+ @mock.patch("retention_worker.subprocess.run")
+ def test_accepts_fresh_token_with_live_cron_ancestry(self, run: mock.Mock, ancestry: mock.Mock) -> None:
+ with tempfile.TemporaryDirectory() as tmp, mock.patch.dict(
+ "os.environ", {"MULTICA_STORAGE_CRON_BRIDGE": "1"}
+ ):
+ config = self.make_config(Path(tmp))
+ run.return_value = mock.Mock(
+ returncode=0,
+ stdout="/usr/bin/python3 /opt/multica/retention_cron_bridge.py --trigger ...\n",
+ )
+ ancestry.return_value = [
+ {"pid": 4321, "parent_pid": 123, "command": "python3"},
+ {"pid": 123, "parent_pid": 1, "command": "/usr/sbin/cron"},
+ ]
+
+ trigger, lineage = verify_cron_bridge(config)
+
+ self.assertEqual(trigger["token"], "fresh-token")
+ self.assertEqual(lineage[-1]["command"], "/usr/sbin/cron")
+
+ @mock.patch("retention_worker.process_ancestry")
+ @mock.patch("retention_worker.subprocess.run")
+ def test_rejects_token_already_consumed_by_a_receipt(self, run: mock.Mock, ancestry: mock.Mock) -> None:
+ with tempfile.TemporaryDirectory() as tmp, mock.patch.dict(
+ "os.environ", {"MULTICA_STORAGE_CRON_BRIDGE": "1"}
+ ):
+ root = Path(tmp)
+ config = self.make_config(root, token="replayed-token")
+ Path(config["cron_bridge_receipt_path"]).write_text(
+ json.dumps({"token": "replayed-token", "status": "green"}),
+ encoding="utf-8",
+ )
+ run.return_value = mock.Mock(
+ returncode=0,
+ stdout="/usr/bin/python3 /opt/multica/retention_cron_bridge.py --trigger ...\n",
+ )
+ ancestry.return_value = [
+ {"pid": 4321, "parent_pid": 123, "command": "python3"},
+ {"pid": 123, "parent_pid": 1, "command": "/usr/sbin/cron"},
+ ]
+
+ with self.assertRaisesRegex(ArchiveError, "already consumed"):
+ verify_cron_bridge(config)
+
+ def test_receipt_is_bound_to_verified_token_not_reread_trigger(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config = self.make_config(root, token="newer-trigger")
+ write_cron_bridge_receipt(config, token="verified-token", status="green")
+ receipt = json.loads(Path(config["cron_bridge_receipt_path"]).read_text())
+ self.assertEqual(receipt["token"], "verified-token")
+
+
+class AlertDeliveryTest(unittest.TestCase):
+ @mock.patch("retention_worker.subprocess.run")
+ def test_success_exit_without_message_id_is_recorded_as_delivery_failure(self, run: mock.Mock) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ run.return_value = mock.Mock(returncode=0, stdout="{}", stderr="")
+ config = {
+ "lark_open_id": "ou_test",
+ "alert_log_path": str(root / "alerts.jsonl"),
+ }
+ send_alert(config, "disk full")
+ alert = json.loads((root / "alerts.jsonl").read_text().splitlines()[-1])
+ self.assertEqual(alert["message"], "lark alert delivery failed")
+
+
+class ElectronUpdaterAuditTest(unittest.TestCase):
+ def make_config(self, root: Path) -> tuple[dict, Path]:
+ home = root / "home"
+ home.mkdir()
+ return (
+ {
+ "electron_updater_audit": {
+ "enabled": True,
+ "timezone": "Asia/Shanghai",
+ "day_of_month": 15,
+ "home_path": str(home),
+ "report_dir": str(root / "reports"),
+ "patterns": [
+ {"label": "electron_updater_cache", "glob": "Library/Caches/*-updater"},
+ {"label": "pending_update_zip", "glob": "Library/Caches/*-updater/*.zip"},
+ ],
+ "warn_total_gib": 5,
+ "warn_candidate_gib": 1,
+ "stale_days": 45,
+ "stale_min_mib": 100,
+ }
+ },
+ home,
+ )
+
+ def test_discovers_deduplicates_and_does_not_modify_updater_residue(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config, home = self.make_config(root)
+ updater = home / "Library" / "Caches" / "demo-updater"
+ updater.mkdir(parents=True)
+ (updater / "update.zip").write_bytes(b"z" * 128)
+ (updater / "metadata.json").write_bytes(b"m" * 64)
+ before = {
+ path.relative_to(updater).as_posix(): (path.stat().st_size, path.stat().st_mtime_ns)
+ for path in updater.iterdir()
+ }
+
+ report = audit_electron_updaters(
+ config,
+ now=lambda: datetime(2026, 8, 5, tzinfo=timezone.utc),
+ invocation_source="test",
+ )
+
+ self.assertEqual(report["schema"], "multica.st1-electron-updater-audit.v1")
+ self.assertEqual(report["audit_month"], "2026-08")
+ self.assertEqual(report["status"], "green")
+ self.assertEqual(report["candidate_count"], 1)
+ self.assertEqual(report["total_bytes"], 192)
+ self.assertEqual(report["candidates"][0]["file_count"], 2)
+ self.assertEqual(
+ report["candidates"][0]["matched_labels"],
+ ["electron_updater_cache", "pending_update_zip"],
+ )
+ after = {
+ path.relative_to(updater).as_posix(): (path.stat().st_size, path.stat().st_mtime_ns)
+ for path in updater.iterdir()
+ }
+ self.assertEqual(after, before)
+ self.assertEqual(json.loads(Path(report["report_path"]).read_text()), report)
+
+ def test_records_a_symlink_match_without_following_or_counting_its_target(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config, home = self.make_config(root)
+ cache = home / "Library" / "Caches"
+ cache.mkdir(parents=True)
+ outside = root / "outside"
+ outside.mkdir()
+ (outside / "payload.bin").write_bytes(b"x" * 4096)
+ (cache / "escape-updater").symlink_to(outside, target_is_directory=True)
+
+ report = audit_electron_updaters(
+ config,
+ now=lambda: datetime(2026, 8, 5, tzinfo=timezone.utc),
+ invocation_source="test",
+ )
+
+ self.assertEqual(report["status"], "green")
+ self.assertEqual(report["candidate_count"], 0)
+ self.assertEqual(report["total_bytes"], 0)
+ self.assertEqual(report["errors"], [])
+ self.assertEqual(len(report["excluded_symlink_matches"]), 1)
+ self.assertEqual(
+ report["excluded_symlink_matches"][0]["path"],
+ str(cache.resolve() / "escape-updater"),
+ )
+ self.assertIn("not traversed", report["excluded_symlink_matches"][0]["reason"])
+ self.assertEqual((outside / "payload.bin").read_bytes(), b"x" * 4096)
+
+ def test_rejects_unbounded_recursive_globs(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ config, _ = self.make_config(Path(tmp))
+ config["electron_updater_audit"]["patterns"] = [
+ {"label": "too_broad", "glob": "Library/**/update.zip"}
+ ]
+
+ with self.assertRaisesRegex(ArchiveError, "recursive"):
+ audit_electron_updaters(config)
+
+ def test_marks_large_or_stale_candidates_for_attention_without_cleanup(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config, home = self.make_config(root)
+ audit_config = config["electron_updater_audit"]
+ audit_config["warn_total_gib"] = 0.0000001
+ audit_config["warn_candidate_gib"] = 0.0000001
+ audit_config["stale_min_mib"] = 0.0001
+ updater = home / "Library" / "Caches" / "old-updater"
+ updater.mkdir(parents=True)
+ payload = updater / "update.zip"
+ payload.write_bytes(b"x" * 512)
+ old = datetime(2026, 5, 1, tzinfo=timezone.utc).timestamp()
+ import os
+
+ os.utime(payload, (old, old))
+ os.utime(updater, (old, old))
+
+ report = audit_electron_updaters(
+ config,
+ now=lambda: datetime(2026, 8, 5, tzinfo=timezone.utc),
+ invocation_source="test",
+ )
+
+ self.assertEqual(report["status"], "attention")
+ reasons = " ".join(report["attention_reasons"])
+ self.assertIn("total", reasons)
+ self.assertIn("candidate", reasons)
+ self.assertIn("stale", reasons)
+ self.assertTrue(payload.exists())
+
+ def test_month_gate_runs_once_and_force_bypasses_day_and_existing_evidence(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config, home = self.make_config(root)
+ updater = home / "Library" / "Caches" / "demo-updater"
+ updater.mkdir(parents=True)
+ (updater / "update.zip").write_bytes(b"z")
+ before_due = lambda: datetime(2026, 8, 5, tzinfo=timezone.utc)
+ on_due = lambda: datetime(2026, 8, 15, tzinfo=timezone.utc)
+
+ skipped = maybe_run_electron_updater_audit(config, now=before_due)
+ self.assertEqual(skipped["status"], "skipped")
+ self.assertIn("before day", skipped["reason"])
+
+ forced = maybe_run_electron_updater_audit(config, now=before_due, force=True)
+ self.assertEqual(forced["status"], "green")
+ report_path = Path(forced["report_path"])
+ self.assertTrue(report_path.is_file())
+
+ already_recorded = maybe_run_electron_updater_audit(config, now=on_due)
+ self.assertEqual(already_recorded["status"], "skipped")
+ self.assertIn("already exists", already_recorded["reason"])
+
+ rerun = maybe_run_electron_updater_audit(config, now=on_due, force=True)
+ self.assertEqual(rerun["status"], "green")
+ self.assertEqual(rerun["audit_month"], "2026-08")
+
+ def test_formal_worker_audits_before_external_volume_checks(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ archive = root / "external" / "archive"
+ archive.mkdir(parents=True)
+ config = {
+ "require_cron_lineage": False,
+ "delete_source": False,
+ "external_path": str(root / "external"),
+ "archive_root": str(archive),
+ "external_volume_uuid": "volume-uuid",
+ "external_min_free_gib": 1,
+ "workspace_roots": [],
+ }
+ events: list[str] = []
+
+ def audit(*args: object, **kwargs: object) -> dict:
+ events.append("audit")
+ return {"status": "skipped", "reason": "test"}
+
+ def external_check(*args: object, **kwargs: object) -> dict:
+ events.append("external")
+ raise ArchiveError("stop after ordering proof")
+
+ with mock.patch(
+ "retention_worker.maybe_run_electron_updater_audit", side_effect=audit
+ ), mock.patch.object(ExternalVolumeGuard, "check", side_effect=external_check):
+ with self.assertRaisesRegex(ArchiveError, "ordering proof"):
+ run_worker(config)
+
+ self.assertEqual(events, ["audit", "external"])
+
+ def test_audit_only_cli_bypasses_cron_lineage_and_external_volume(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ config, home = self.make_config(root)
+ config.update(
+ {
+ "require_cron_lineage": True,
+ "lock_path": str(root / "worker.lock"),
+ "external_path": str(root / "missing-external"),
+ "archive_root": str(root / "missing-external" / "archive"),
+ }
+ )
+ updater = home / "Library" / "Caches" / "demo-updater"
+ updater.mkdir(parents=True)
+ (updater / "update.zip").write_bytes(b"z")
+ config_path = root / "config.json"
+ config_path.write_text(json.dumps(config), encoding="utf-8")
+ output = io.StringIO()
+
+ with mock.patch.object(
+ sys,
+ "argv",
+ ["retention_worker.py", "--config", str(config_path), "--electron-audit-only"],
+ ), redirect_stdout(output):
+ exit_code = main()
+
+ self.assertEqual(exit_code, 0)
+ summary = json.loads(output.getvalue())
+ self.assertEqual(summary["status"], "green")
+ self.assertTrue(Path(summary["report_path"]).is_file())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/scripts/storage-governance/test_storage_guard.py b/scripts/storage-governance/test_storage_guard.py
new file mode 100644
index 00000000000..4d6d2382ae4
--- /dev/null
+++ b/scripts/storage-governance/test_storage_guard.py
@@ -0,0 +1,458 @@
+from __future__ import annotations
+
+import json
+import plistlib
+import signal
+import sys
+import tempfile
+import unittest
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from storage_guard import ( # noqa: E402
+ Guard,
+ Sample,
+ build_capacity_report,
+ classify_level,
+ SystemCollector,
+)
+
+
+GIB = 1024**3
+
+
+class LaunchdContractTest(unittest.TestCase):
+ def test_guard_ignores_ambient_python_environment(self) -> None:
+ plist_path = Path(__file__).with_name("com.multica.storage-guard.plist.example")
+ with plist_path.open("rb") as handle:
+ program_arguments = plistlib.load(handle)["ProgramArguments"]
+
+ self.assertEqual(program_arguments[:2], ["/usr/bin/python3", "-E"])
+
+
+def base_config(root: Path) -> dict:
+ return {
+ "internal_path": "/",
+ "external_path": "/Volumes/MacMini-HotSSD",
+ "level1_free_gib": 25,
+ "level1_clear_gib": 28,
+ "level2_free_gib": 18,
+ "level2_clear_gib": 21,
+ "external_min_free_gib": 100,
+ "safety_floor_gib": 25,
+ "burst_reserve_gib": 10,
+ "minimum_observation_hours": 48,
+ "observer_labels": ["ai.multica.ws2512.m0-shadow-observer"],
+ "nonproduction_launchagents": ["com.example.nonproduction"],
+ "lark_open_id": "ou_test",
+ "alert_cooldown_seconds": 3600,
+ "legacy_sigstop_fallback": True,
+ "state_path": str(root / "state.json"),
+ "metrics_path": str(root / "metrics.jsonl"),
+ "capacity_report_path": str(root / "capacity.json"),
+ }
+
+
+class FakeCollector:
+ def __init__(self, sample: Sample):
+ self.sample = sample
+
+ def collect(self) -> Sample:
+ return self.sample
+
+
+class FakeRunner:
+ def __init__(
+ self,
+ *,
+ admission_supported: bool = True,
+ resume_supported: bool = True,
+ launchagent_stops: bool = True,
+ ):
+ self.admission_supported = admission_supported
+ self.resume_supported = resume_supported
+ self.launchagent_stops = launchagent_stops
+ self.calls: list[tuple[str, ...]] = []
+ self.signals: list[tuple[int, int]] = []
+
+ def run(self, argv: list[str], *, tolerate_failure: bool = False) -> tuple[int, str, str]:
+ self.calls.append(tuple(argv))
+ if argv[:3] == ["multica", "daemon", "pause"]:
+ if not self.admission_supported:
+ return 1, "", "endpoint returned 404"
+ return 0, json.dumps({"owner_paused": True, "admission_paused": True}), ""
+ if argv[:3] == ["multica", "daemon", "resume"]:
+ if not self.resume_supported:
+ return 1, "", "endpoint unavailable"
+ return 0, json.dumps({"owner_paused": False, "admission_paused": False}), ""
+ if argv[:3] == ["multica", "daemon", "status"]:
+ return 0, json.dumps({"pid": 4321, "status": "running", "active_task_count": 2}), ""
+ if argv[:3] == ["/bin/ps", "-p", "4321"]:
+ return 0, "/opt/homebrew/bin/multica daemon start --foreground\n", ""
+ if argv[:2] == ["launchctl", "print"]:
+ return (1, "", "not found") if self.launchagent_stops else (0, "running", "")
+ if argv and argv[0] == "lark-cli":
+ return 0, json.dumps({"data": {"message_id": "om_test"}}), ""
+ return 0, "{}", ""
+
+ def signal(self, pid: int, sig: int) -> None:
+ self.signals.append((pid, sig))
+
+
+class LevelClassificationTest(unittest.TestCase):
+ def test_hysteresis_prevents_flapping(self) -> None:
+ self.assertEqual(classify_level(30 * GIB, previous=0, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 0)
+ self.assertEqual(classify_level(24 * GIB, previous=0, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 1)
+ self.assertEqual(classify_level(26 * GIB, previous=1, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 1)
+ self.assertEqual(classify_level(29 * GIB, previous=1, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 0)
+ self.assertEqual(classify_level(17 * GIB, previous=1, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 2)
+ self.assertEqual(classify_level(19 * GIB, previous=2, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 2)
+ self.assertEqual(classify_level(22 * GIB, previous=2, level1=25 * GIB, level1_clear=28 * GIB, level2=18 * GIB, level2_clear=21 * GIB), 1)
+
+
+class GuardActionTest(unittest.TestCase):
+ def make_sample(
+ self,
+ free_gib: int,
+ timestamp: datetime,
+ active_tasks: int = 2,
+ admission_pause_owners: tuple[str, ...] = (),
+ ) -> Sample:
+ return Sample(
+ recorded_at=timestamp.isoformat(),
+ internal_free_bytes=free_gib * GIB,
+ external_free_bytes=1500 * GIB,
+ swap_used_bytes=9 * GIB,
+ active_task_count=active_tasks,
+ daemon_pid=4321,
+ daemon_status="running",
+ admission_pause_owners=admission_pause_owners,
+ )
+
+ def test_level1_stops_observer_and_pauses_admission_without_killing_tasks(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ runner = FakeRunner()
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ result = Guard(base_config(root), runner, FakeCollector(self.make_sample(24, now))).run_once()
+
+ self.assertEqual(result["level"], 1)
+ self.assertIn(("launchctl", "disable", "gui/501/ai.multica.ws2512.m0-shadow-observer"), runner.calls)
+ self.assertIn(("launchctl", "bootout", "gui/501/ai.multica.ws2512.m0-shadow-observer"), runner.calls)
+ self.assertIn(("multica", "daemon", "pause", "--owner", "storage-guard", "--output", "json"), runner.calls)
+ self.assertEqual(runner.signals, [])
+ state = json.loads(Path(base_config(root)["state_path"]).read_text())
+ self.assertEqual(state["level"], 1)
+ self.assertFalse(state["legacy_daemon_sigstopped"])
+
+ def test_old_daemon_refuses_sigstop_with_active_tasks(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ runner = FakeRunner(admission_supported=False)
+ Guard(cfg, runner, FakeCollector(self.make_sample(24, now))).run_once()
+ self.assertEqual(runner.signals, [])
+ self.assertTrue(
+ any(
+ "legacy_sigstop_disabled:unsafe" in action
+ for action in json.loads(Path(cfg["state_path"]).read_text())["last_actions"]
+ )
+ )
+
+ def test_old_daemon_never_uses_sigstop_even_when_idle(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ runner = FakeRunner(admission_supported=False)
+ result = Guard(cfg, runner, FakeCollector(self.make_sample(24, now, active_tasks=0))).run_once()
+ self.assertEqual(runner.signals, [])
+ self.assertEqual(result["level"], 1)
+ self.assertEqual(json.loads(Path(cfg["state_path"]).read_text())["enforcement_status"], "failed")
+
+ def test_persisted_sigstop_intent_is_reconciled_after_space_recovers(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ Path(cfg["state_path"]).write_text(
+ json.dumps(
+ {
+ "level": 0,
+ "legacy_daemon_sigstop_intent": True,
+ "legacy_daemon_sigstopped": False,
+ "legacy_daemon_pid": 4321,
+ "legacy_daemon_command": "/opt/homebrew/bin/multica daemon start --foreground",
+ }
+ ),
+ encoding="utf-8",
+ )
+ runner = FakeRunner(admission_supported=False)
+ result = Guard(
+ cfg,
+ runner,
+ FakeCollector(self.make_sample(30, datetime(2026, 8, 3, tzinfo=timezone.utc))),
+ ).run_once()
+ self.assertIn((4321, signal.SIGCONT), runner.signals)
+ self.assertEqual(result["level"], 0)
+
+ def test_daemon_owner_reconciles_when_local_pause_state_was_not_committed(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ runner = FakeRunner()
+ result = Guard(
+ cfg,
+ runner,
+ FakeCollector(
+ self.make_sample(
+ 30,
+ datetime(2026, 8, 3, tzinfo=timezone.utc),
+ admission_pause_owners=("storage-guard",),
+ )
+ ),
+ ).run_once()
+ self.assertIn(
+ ("multica", "daemon", "resume", "--owner", "storage-guard", "--output", "json"),
+ runner.calls,
+ )
+ self.assertEqual(result["level"], 0)
+
+ def test_level2_launchagent_failure_is_not_reported_as_verified(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ runner = FakeRunner(launchagent_stops=False)
+ Guard(
+ cfg,
+ runner,
+ FakeCollector(self.make_sample(17, datetime(2026, 8, 3, tzinfo=timezone.utc))),
+ ).run_once()
+ state = json.loads(Path(cfg["state_path"]).read_text())
+ self.assertEqual(state["enforcement_status"], "failed")
+ self.assertTrue(any("launchagent_stop_failed" in action for action in state["last_actions"]))
+
+ def test_resume_failure_remains_pending_and_retries(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ first = FakeRunner()
+ Guard(cfg, first, FakeCollector(self.make_sample(24, now))).run_once()
+
+ failing = FakeRunner(resume_supported=False)
+ result = Guard(cfg, failing, FakeCollector(self.make_sample(30, now + timedelta(minutes=1)))).run_once()
+ self.assertEqual(result["level"], 1)
+ state = json.loads(Path(cfg["state_path"]).read_text())
+ self.assertTrue(state["resume_pending"])
+ self.assertEqual(state["enforcement_status"], "failed")
+
+ def test_level2_pauses_explicit_nonproduction_jobs_and_sends_lark_alert(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ runner = FakeRunner()
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ result = Guard(base_config(root), runner, FakeCollector(self.make_sample(17, now))).run_once()
+
+ self.assertEqual(result["level"], 2)
+ self.assertIn(("launchctl", "disable", "gui/501/com.example.nonproduction"), runner.calls)
+ self.assertTrue(any(call and call[0] == "lark-cli" for call in runner.calls))
+
+ def test_every_run_appends_one_machine_readable_sample(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ cfg = base_config(root)
+ now = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ Guard(cfg, FakeRunner(), FakeCollector(self.make_sample(30, now))).run_once()
+ lines = Path(cfg["metrics_path"]).read_text().splitlines()
+ self.assertEqual(len(lines), 1)
+ self.assertEqual(json.loads(lines[0])["internal_free_bytes"], 30 * GIB)
+
+
+class CapacityReportTest(unittest.TestCase):
+ def test_report_is_inconclusive_before_48_hours(self) -> None:
+ start = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ samples = [
+ {"recorded_at": start.isoformat(), "internal_free_bytes": 40 * GIB},
+ {"recorded_at": (start + timedelta(hours=24)).isoformat(), "internal_free_bytes": 36 * GIB},
+ ]
+ report = build_capacity_report(samples, safety_floor_bytes=25 * GIB, burst_reserve_bytes=10 * GIB, minimum_hours=48)
+ self.assertEqual(report["status"], "INCONCLUSIVE")
+ self.assertIsNone(report["days_remaining"])
+
+ def test_two_samples_across_48_hours_do_not_fake_ready_coverage(self) -> None:
+ start = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ report = build_capacity_report(
+ [
+ {"recorded_at": start.isoformat(), "internal_free_bytes": 40 * GIB},
+ {"recorded_at": (start + timedelta(hours=48)).isoformat(), "internal_free_bytes": 39 * GIB},
+ ],
+ safety_floor_bytes=25 * GIB,
+ burst_reserve_bytes=10 * GIB,
+ minimum_hours=48,
+ expected_interval_seconds=3600,
+ )
+ self.assertEqual(report["status"], "INCONCLUSIVE")
+ self.assertLess(report["coverage_ratio"], 0.1)
+
+ def test_report_uses_p95_observed_hourly_growth_and_reserves(self) -> None:
+ start = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ samples = []
+ free = 80 * GIB
+ for hour in range(50):
+ samples.append({"recorded_at": (start + timedelta(hours=hour)).isoformat(), "internal_free_bytes": free})
+ free -= 1 * GIB
+ report = build_capacity_report(samples, safety_floor_bytes=25 * GIB, burst_reserve_bytes=10 * GIB, minimum_hours=48)
+ self.assertEqual(report["status"], "READY")
+ self.assertAlmostEqual(report["p95_growth_bytes_per_hour"], float(GIB))
+ self.assertAlmostEqual(report["peak_growth_bytes_per_hour"], float(GIB))
+ self.assertEqual(report["days_remaining"], 0.0)
+
+ def test_one_missing_category_sample_uses_field_coverage_instead_of_poisoning_window(self) -> None:
+ start = datetime(2026, 8, 3, tzinfo=timezone.utc)
+ samples = []
+ for hour in range(50):
+ samples.append(
+ {
+ "recorded_at": (start + timedelta(hours=hour)).isoformat(),
+ "internal_free_bytes": (80 - hour) * GIB,
+ "cursor_bytes": None if hour == 25 else hour * GIB,
+ }
+ )
+ report = build_capacity_report(
+ samples,
+ safety_floor_bytes=25 * GIB,
+ burst_reserve_bytes=10 * GIB,
+ minimum_hours=48,
+ expected_interval_seconds=3600,
+ required_growth_fields=["cursor_bytes"],
+ required_field_max_gap_seconds=3 * 3600,
+ )
+ self.assertEqual(report["status"], "READY")
+ self.assertGreater(report["field_coverage_ratio"]["cursor_bytes"], 0.9)
+
+
+class SystemCollectorTest(unittest.TestCase):
+ def test_growth_metrics_respect_shared_scan_budget(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ workspace = Path(tmp) / "workspaces"
+ workspace.mkdir()
+ (workspace / "payload.bin").write_bytes(b"payload")
+ collector = SystemCollector(
+ {
+ "growth_scan_budget_seconds": 0,
+ "workspace_roots": [str(workspace)],
+ "logs_paths": [],
+ },
+ mock.Mock(),
+ )
+
+ self.assertIsNone(collector.growth_metrics()["workspace_total_bytes"])
+
+ def test_fast_collect_does_not_scan_growth_directories(self) -> None:
+ runner = mock.Mock()
+ runner.run.side_effect = [
+ (0, json.dumps({"status": "running", "pid": 1, "active_task_count": 0}), ""),
+ (0, "vm.swapusage: total = 0.00M used = 0.00M free = 0.00M", ""),
+ ]
+ collector = SystemCollector({"internal_path": "/", "external_path": "/"}, runner)
+ with mock.patch.object(collector, "growth_metrics", side_effect=AssertionError("slow scan on fast path")):
+ sample = collector.collect()
+ self.assertEqual(sample.active_task_count, 0)
+
+ def test_stale_green_retention_report_is_not_capacity_evidence(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ report = root / "report.json"
+ report.write_text(
+ json.dumps(
+ {
+ "status": "green",
+ "recorded_at": "2000-01-01T00:00:00+00:00",
+ "gc_candidates": [{"eligible": True, "details": {"size_bytes": 123}}],
+ }
+ ),
+ encoding="utf-8",
+ )
+ collector = SystemCollector(
+ {
+ "retention_report_path": str(report),
+ "retention_report_max_age_seconds": 1800,
+ "workspace_roots": [],
+ "logs_paths": [],
+ },
+ mock.Mock(),
+ )
+ metrics = collector.growth_metrics()
+ self.assertIsNone(metrics["workspace_gc_eligible_bytes"])
+
+ def test_future_green_retention_report_is_not_capacity_evidence(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ report = root / "report.json"
+ report.write_text(
+ json.dumps(
+ {
+ "status": "green",
+ "recorded_at": "2099-01-01T00:00:00+00:00",
+ "gc_candidates": [{"eligible": True, "details": {"size_bytes": 123}}],
+ }
+ ),
+ encoding="utf-8",
+ )
+ collector = SystemCollector(
+ {
+ "retention_report_path": str(report),
+ "retention_report_max_age_seconds": 1800,
+ "workspace_roots": [],
+ "logs_paths": [],
+ },
+ mock.Mock(),
+ )
+ self.assertIsNone(collector.growth_metrics()["workspace_gc_eligible_bytes"])
+
+ def test_workspace_categories_conserve_file_payload_bytes(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ workspace = root / "workspaces"
+ workspace.mkdir()
+ (workspace / "payload.bin").write_bytes(b"x" * 100)
+ report = root / "report.json"
+ report.write_text(
+ json.dumps(
+ {
+ "status": "green",
+ "recorded_at": datetime.now(timezone.utc).isoformat(),
+ "gc_candidates": [{"eligible": True, "details": {"size_bytes": 40}}],
+ }
+ ),
+ encoding="utf-8",
+ )
+ collector = SystemCollector(
+ {
+ "retention_report_path": str(report),
+ "retention_report_max_age_seconds": 1800,
+ "workspace_roots": [str(workspace)],
+ "logs_paths": [],
+ },
+ mock.Mock(),
+ )
+ metrics = collector.growth_metrics()
+ total = metrics["workspace_total_bytes"]
+ categories = sum(
+ int(metrics[key] or 0)
+ for key in (
+ "workspace_gc_eligible_bytes",
+ "workspace_inflight_bytes",
+ "workspace_gc_backlog_bytes",
+ "workspace_unclassified_bytes",
+ )
+ )
+ self.assertEqual(categories, total)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/server/cmd/multica/cmd_daemon.go b/server/cmd/multica/cmd_daemon.go
index 724b033f4ae..0f168573ab9 100644
--- a/server/cmd/multica/cmd_daemon.go
+++ b/server/cmd/multica/cmd_daemon.go
@@ -8,6 +8,7 @@ import (
"io"
"log/slog"
"net/http"
+ "net/url"
"os"
"os/exec"
"path/filepath"
@@ -50,6 +51,24 @@ var daemonStatusCmd = &cobra.Command{
RunE: runDaemonStatus,
}
+var daemonPauseCmd = &cobra.Command{
+ Use: "pause",
+ Short: "Pause admission of new tasks without cancelling active tasks",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ return runDaemonAdmission(cmd, true)
+ },
+}
+
+var daemonResumeCmd = &cobra.Command{
+ Use: "resume",
+ Short: "Resume admission of new tasks",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ return runDaemonAdmission(cmd, false)
+ },
+}
+
var daemonProbeRuntimesCmd = &cobra.Command{
Use: "probe-runtimes",
Short: "Probe locally configured runtimes for the Desktop app",
@@ -104,6 +123,10 @@ func init() {
daemonLogsCmd.Flags().IntP("lines", "n", 50, "Number of lines to show")
daemonStatusCmd.Flags().String("output", "table", "Output format: table or json")
+ daemonPauseCmd.Flags().String("output", "table", "Output format: table or json")
+ daemonResumeCmd.Flags().String("output", "table", "Output format: table or json")
+ daemonPauseCmd.Flags().String("owner", "manual", "Independent pause owner (for example: manual or storage-guard)")
+ daemonResumeCmd.Flags().String("owner", "manual", "Release only this pause owner")
// restart shares all the same flags as start
rf := daemonRestartCmd.Flags()
@@ -132,6 +155,8 @@ func init() {
daemonCmd.AddCommand(daemonStopCmd)
daemonCmd.AddCommand(daemonRestartCmd)
daemonCmd.AddCommand(daemonStatusCmd)
+ daemonCmd.AddCommand(daemonPauseCmd)
+ daemonCmd.AddCommand(daemonResumeCmd)
daemonCmd.AddCommand(daemonProbeRuntimesCmd)
daemonCmd.AddCommand(daemonLogsCmd)
daemonCmd.AddCommand(daemonDiskUsageCmd)
@@ -1055,6 +1080,67 @@ func requestDaemonShutdown(healthPort int) error {
return nil
}
+type daemonAdmissionState struct {
+ AdmissionPaused bool `json:"admission_paused"`
+ AdmissionPauseOwners []string `json:"admission_pause_owners"`
+ Owner string `json:"owner"`
+ OwnerPaused bool `json:"owner_paused"`
+ ClaimsInFlight int `json:"claims_in_flight"`
+ ActiveTaskCount int64 `json:"active_task_count"`
+}
+
+func requestDaemonAdmission(client *http.Client, baseURL string, paused bool, owner string) (daemonAdmissionState, error) {
+ action := "resume"
+ if paused {
+ action = "pause"
+ }
+ endpoint := strings.TrimRight(baseURL, "/") + "/admission/" + action + "?" + url.Values{"owner": {owner}}.Encode()
+ req, err := http.NewRequest(http.MethodPost, endpoint, nil)
+ if err != nil {
+ return daemonAdmissionState{}, err
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ return daemonAdmissionState{}, err
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+ return daemonAdmissionState{}, fmt.Errorf("daemon admission endpoint returned status %d", resp.StatusCode)
+ }
+ var state daemonAdmissionState
+ if err := json.NewDecoder(resp.Body).Decode(&state); err != nil {
+ return daemonAdmissionState{}, fmt.Errorf("decode daemon admission response: %w", err)
+ }
+ return state, nil
+}
+
+func runDaemonAdmission(cmd *cobra.Command, paused bool) error {
+ profile := resolveProfile(cmd)
+ port := healthPortForProfile(profile)
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer cancel()
+ if !daemonAlive(checkDaemonHealthOnPort(ctx, port)) {
+ return fmt.Errorf("daemon is not running")
+ }
+
+ client := &http.Client{Timeout: 5 * time.Second}
+ owner, _ := cmd.Flags().GetString("owner")
+ state, err := requestDaemonAdmission(client, fmt.Sprintf("http://127.0.0.1:%d", port), paused, owner)
+ if err != nil {
+ return err
+ }
+ output, _ := cmd.Flags().GetString("output")
+ if output == "json" {
+ return cli.PrintJSON(cmd.OutOrStdout(), state)
+ }
+ action := "resumed"
+ if paused {
+ action = "paused"
+ }
+ fmt.Fprintf(cmd.OutOrStdout(), "Daemon admission %s (active tasks: %d, claims draining: %d).\n", action, state.ActiveTaskCount, state.ClaimsInFlight)
+ return nil
+}
+
// --- daemon status ---
func runDaemonStatus(cmd *cobra.Command, _ []string) error {
diff --git a/server/cmd/multica/cmd_daemon_test.go b/server/cmd/multica/cmd_daemon_test.go
index 136cfc44834..10ee78712e2 100644
--- a/server/cmd/multica/cmd_daemon_test.go
+++ b/server/cmd/multica/cmd_daemon_test.go
@@ -69,6 +69,53 @@ func TestPrintDaemonStatusIncludesCLIVersion(t *testing.T) {
}
}
+func TestRequestDaemonAdmissionUsesLocalPostAndDecodesState(t *testing.T) {
+ t.Parallel()
+
+ var gotMethod, gotPath, gotOwner string
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ gotMethod = r.Method
+ gotPath = r.URL.Path
+ gotOwner = r.URL.Query().Get("owner")
+ w.Header().Set("Content-Type", "application/json")
+ _, _ = w.Write([]byte(`{"admission_paused":true,"admission_pause_owners":["storage-guard"],"owner":"storage-guard","owner_paused":true,"claims_in_flight":0,"active_task_count":2}`))
+ }))
+ defer srv.Close()
+
+ state, err := requestDaemonAdmission(srv.Client(), srv.URL, true, "storage-guard")
+ if err != nil {
+ t.Fatalf("requestDaemonAdmission: %v", err)
+ }
+ if gotMethod != http.MethodPost || gotPath != "/admission/pause" {
+ t.Fatalf("request = %s %s, want POST /admission/pause", gotMethod, gotPath)
+ }
+ if gotOwner != "storage-guard" || !state.OwnerPaused {
+ t.Fatalf("owner = %q, state = %+v", gotOwner, state)
+ }
+ if !state.AdmissionPaused || state.ClaimsInFlight != 0 || state.ActiveTaskCount != 2 {
+ t.Fatalf("state = %+v, want paused with two active tasks and no draining claims", state)
+ }
+}
+
+func TestRequestDaemonAdmissionAcceptsDrainPending(t *testing.T) {
+ t.Parallel()
+
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusAccepted)
+ _, _ = w.Write([]byte(`{"admission_paused":true,"owner":"manual","owner_paused":true,"claims_in_flight":1,"active_task_count":2}`))
+ }))
+ defer srv.Close()
+
+ state, err := requestDaemonAdmission(srv.Client(), srv.URL, true, "manual")
+ if err != nil {
+ t.Fatalf("requestDaemonAdmission: %v", err)
+ }
+ if !state.AdmissionPaused || state.ClaimsInFlight != 1 {
+ t.Fatalf("state = %+v, want paused with one draining claim", state)
+ }
+}
+
func TestBuildDaemonStartArgsForwardsCodexHandshakeTimeout(t *testing.T) {
cmd := &cobra.Command{}
cmd.Flags().Duration("codex-handshake-timeout", 0, "")
diff --git a/server/internal/daemon/admission.go b/server/internal/daemon/admission.go
new file mode 100644
index 00000000000..3d9fb7ad615
--- /dev/null
+++ b/server/internal/daemon/admission.go
@@ -0,0 +1,127 @@
+package daemon
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "regexp"
+ "sort"
+)
+
+const defaultAdmissionOwner = "manual"
+
+var admissionOwnerPattern = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,63}$`)
+
+type admissionPauseDiskState struct {
+ Owners []string `json:"owners"`
+}
+
+func normalizeAdmissionOwner(owner string) (string, error) {
+ if owner == "" {
+ owner = defaultAdmissionOwner
+ }
+ if !admissionOwnerPattern.MatchString(owner) {
+ return "", fmt.Errorf("invalid admission owner %q", owner)
+ }
+ return owner, nil
+}
+
+func (d *Daemon) admissionStatePath() string {
+ if d.cfg.WorkspacesRoot == "" {
+ return ""
+ }
+ return filepath.Join(d.cfg.WorkspacesRoot, ".multica-admission-pauses.json")
+}
+
+func cloneAdmissionOwners(source map[string]struct{}) map[string]struct{} {
+ result := make(map[string]struct{}, len(source))
+ for owner := range source {
+ result[owner] = struct{}{}
+ }
+ return result
+}
+
+func sortedAdmissionOwners(source map[string]struct{}) []string {
+ owners := make([]string, 0, len(source))
+ for owner := range source {
+ owners = append(owners, owner)
+ }
+ sort.Strings(owners)
+ return owners
+}
+
+func (d *Daemon) persistAdmissionOwnersLocked(next map[string]struct{}) error {
+ path := d.admissionStatePath()
+ if path == "" {
+ return nil
+ }
+ if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
+ return fmt.Errorf("create admission state directory: %w", err)
+ }
+ payload, err := json.Marshal(admissionPauseDiskState{Owners: sortedAdmissionOwners(next)})
+ if err != nil {
+ return err
+ }
+ temporary, err := os.CreateTemp(filepath.Dir(path), ".admission-pauses-*.tmp")
+ if err != nil {
+ return fmt.Errorf("create admission state temp file: %w", err)
+ }
+ temporaryName := temporary.Name()
+ defer os.Remove(temporaryName)
+ if err := temporary.Chmod(0o600); err != nil {
+ temporary.Close()
+ return err
+ }
+ if _, err := temporary.Write(append(payload, '\n')); err != nil {
+ temporary.Close()
+ return err
+ }
+ if err := temporary.Sync(); err != nil {
+ temporary.Close()
+ return err
+ }
+ if err := temporary.Close(); err != nil {
+ return err
+ }
+ if err := os.Rename(temporaryName, path); err != nil {
+ return fmt.Errorf("commit admission state: %w", err)
+ }
+ directory, err := os.Open(filepath.Dir(path))
+ if err != nil {
+ return fmt.Errorf("open admission state directory: %w", err)
+ }
+ defer directory.Close()
+ if err := directory.Sync(); err != nil {
+ return fmt.Errorf("sync admission state directory: %w", err)
+ }
+ return nil
+}
+
+func (d *Daemon) loadAdmissionOwners() error {
+ path := d.admissionStatePath()
+ if path == "" {
+ return nil
+ }
+ payload, err := os.ReadFile(path)
+ if os.IsNotExist(err) {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ var state admissionPauseDiskState
+ if err := json.Unmarshal(payload, &state); err != nil {
+ return err
+ }
+ owners := make(map[string]struct{}, len(state.Owners))
+ for _, owner := range state.Owners {
+ normalized, err := normalizeAdmissionOwner(owner)
+ if err != nil {
+ return err
+ }
+ owners[normalized] = struct{}{}
+ }
+ d.admissionPauseOwners = owners
+ return nil
+}
diff --git a/server/internal/daemon/daemon.go b/server/internal/daemon/daemon.go
index 47610072ea6..543014ee3ab 100644
--- a/server/internal/daemon/daemon.go
+++ b/server/internal/daemon/daemon.go
@@ -376,9 +376,10 @@ type Daemon struct {
// or any task is in handleTask. Together that closes the fetch-then-claim
// race where a new task slipping in during the release-metadata fetch
// would be cancelled by triggerRestart's root-ctx cancel.
- claimMu sync.Mutex
- pauseClaims bool // when true, the batch poller skips claiming
- claimsInFlight int // pollers that have decided to claim but haven't yet handed the task off to handleTask
+ claimMu sync.Mutex
+ pauseClaims bool // auto-update barrier; successful upgrades keep it set until process restart
+ admissionPauseOwners map[string]struct{} // independently owned, restart-persistent operator barriers
+ claimsInFlight int // pollers that have decided to claim but haven't yet handed the task off to handleTask
activeEnvRootsMu sync.Mutex
activeEnvRootsCond *sync.Cond // signalled when an in-flight env-root GC mutation finishes
@@ -462,6 +463,14 @@ func New(cfg Config, logger *slog.Logger) *Daemon {
reconcile: newReconcileBroadcaster(),
workspaceChanges: newWorkspaceChangeSignal(),
wsRPC: newWSRPCClient(wsRPCResponseGrace),
+ admissionPauseOwners: make(map[string]struct{}),
+ }
+ if err := d.loadAdmissionOwners(); err != nil {
+ // A corrupt/unreadable persisted barrier must fail closed. The owner is
+ // intentionally not persisted over the corrupt file; an operator must
+ // repair/remove it before admission can resume.
+ d.admissionPauseOwners["state-load-error"] = struct{}{}
+ logger.Error("admission pause state failed to load; admission remains paused", "error", err)
}
d.activeEnvRootsCond = sync.NewCond(&d.activeEnvRootsMu)
d.activeCodexStoresCond = sync.NewCond(&d.activeCodexStoresMu)
@@ -3352,7 +3361,7 @@ func (d *Daemon) reportUpdateResultWithRetry(ctx context.Context, runtimeID, upd
func (d *Daemon) tryEnterClaim() bool {
d.claimMu.Lock()
defer d.claimMu.Unlock()
- if d.pauseClaims {
+ if d.pauseClaims || len(d.admissionPauseOwners) > 0 {
return false
}
d.claimsInFlight++
diff --git a/server/internal/daemon/health.go b/server/internal/daemon/health.go
index d50e16e2c57..d6591ec0053 100644
--- a/server/internal/daemon/health.go
+++ b/server/internal/daemon/health.go
@@ -25,14 +25,16 @@ type HealthResponse struct {
// lifecycle CLI (`daemon start/stop`) acts on the host process namespace,
// so a foreign-OS daemon can't be started/stopped by the app even though
// /health is reachable. See #3916.
- OS string `json:"os"`
- Uptime string `json:"uptime"`
- DaemonID string `json:"daemon_id"`
- DeviceName string `json:"device_name"`
- ServerURL string `json:"server_url"`
- CLIVersion string `json:"cli_version"`
- ActiveTaskCount int64 `json:"active_task_count"`
- Agents []string `json:"agents"`
+ OS string `json:"os"`
+ Uptime string `json:"uptime"`
+ DaemonID string `json:"daemon_id"`
+ DeviceName string `json:"device_name"`
+ ServerURL string `json:"server_url"`
+ CLIVersion string `json:"cli_version"`
+ ActiveTaskCount int64 `json:"active_task_count"`
+ AdmissionPaused bool `json:"admission_paused"`
+ AdmissionPauseOwners []string `json:"admission_pause_owners"`
+ Agents []string `json:"agents"`
// SkippedAgents maps a provider that WAS discovered on this machine to the
// reason the last registration round dropped it (version undetectable,
// below the minimum supported version). Purely diagnostic, and omitted when
@@ -103,18 +105,20 @@ func (d *Daemon) healthHandler(startedAt time.Time) http.HandlerFunc {
}
resp := HealthResponse{
- Status: status,
- PID: os.Getpid(),
- OS: runtime.GOOS,
- Uptime: time.Since(startedAt).Truncate(time.Second).String(),
- DaemonID: d.cfg.DaemonID,
- DeviceName: d.cfg.DeviceName,
- ServerURL: d.cfg.ServerBaseURL,
- CLIVersion: d.cfg.CLIVersion,
- ActiveTaskCount: d.activeTasks.Load(),
- Agents: agents,
- SkippedAgents: d.skippedAgentsSnapshot(),
- Workspaces: wsList,
+ Status: status,
+ PID: os.Getpid(),
+ OS: runtime.GOOS,
+ Uptime: time.Since(startedAt).Truncate(time.Second).String(),
+ DaemonID: d.cfg.DaemonID,
+ DeviceName: d.cfg.DeviceName,
+ ServerURL: d.cfg.ServerBaseURL,
+ CLIVersion: d.cfg.CLIVersion,
+ ActiveTaskCount: d.activeTasks.Load(),
+ AdmissionPaused: d.isAdmissionPaused(),
+ AdmissionPauseOwners: d.admissionPauseOwnersSnapshot(),
+ Agents: agents,
+ SkippedAgents: d.skippedAgentsSnapshot(),
+ Workspaces: wsList,
}
w.Header().Set("Content-Type", "application/json")
@@ -122,6 +126,68 @@ func (d *Daemon) healthHandler(startedAt time.Time) http.HandlerFunc {
}
}
+func (d *Daemon) isAdmissionPaused() bool {
+ d.claimMu.Lock()
+ defer d.claimMu.Unlock()
+ return len(d.admissionPauseOwners) > 0
+}
+
+func (d *Daemon) admissionPauseOwnersSnapshot() []string {
+ d.claimMu.Lock()
+ defer d.claimMu.Unlock()
+ return sortedAdmissionOwners(d.admissionPauseOwners)
+}
+
+// admissionHandler changes only the operator-controlled claim barrier. Pausing
+// does not cancel active tasks. A 202 response means a claim that began before
+// the pause is still draining; the barrier already blocks every later claim.
+func (d *Daemon) admissionHandler(paused bool) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodPost {
+ http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+ return
+ }
+ owner, err := normalizeAdmissionOwner(r.URL.Query().Get("owner"))
+ if err != nil {
+ http.Error(w, err.Error(), http.StatusBadRequest)
+ return
+ }
+
+ d.claimMu.Lock()
+ next := cloneAdmissionOwners(d.admissionPauseOwners)
+ if paused {
+ next[owner] = struct{}{}
+ } else {
+ delete(next, owner)
+ }
+ if err := d.persistAdmissionOwnersLocked(next); err != nil {
+ d.claimMu.Unlock()
+ http.Error(w, "persist admission barrier: "+err.Error(), http.StatusInternalServerError)
+ return
+ }
+ d.admissionPauseOwners = next
+ claimsInFlight := d.claimsInFlight
+ owners := sortedAdmissionOwners(next)
+ _, ownerPaused := next[owner]
+ d.claimMu.Unlock()
+
+ statusCode := http.StatusOK
+ if paused && claimsInFlight > 0 {
+ statusCode = http.StatusAccepted
+ }
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(statusCode)
+ _ = json.NewEncoder(w).Encode(map[string]any{
+ "admission_paused": len(owners) > 0,
+ "admission_pause_owners": owners,
+ "owner": owner,
+ "owner_paused": ownerPaused,
+ "claims_in_flight": claimsInFlight,
+ "active_task_count": d.activeTasks.Load(),
+ })
+ }
+}
+
// shutdownHandler triggers a graceful daemon shutdown by cancelling the
// top-level context. Used by `multica daemon stop` so we don't depend on
// OS-signal delivery, which is unreliable on Windows once the daemon is
@@ -150,6 +216,8 @@ func (d *Daemon) serveHealth(ctx context.Context, ln net.Listener, startedAt tim
mux := http.NewServeMux()
mux.HandleFunc("/health", d.healthHandler(startedAt))
mux.HandleFunc("/shutdown", d.shutdownHandler())
+ mux.HandleFunc("/admission/pause", d.admissionHandler(true))
+ mux.HandleFunc("/admission/resume", d.admissionHandler(false))
mux.HandleFunc("/repo/checkout", d.repoCheckoutHandler())
srv := &http.Server{Handler: mux}
diff --git a/server/internal/daemon/health_test.go b/server/internal/daemon/health_test.go
index 5c5c31e6ed4..46975dc0aa4 100644
--- a/server/internal/daemon/health_test.go
+++ b/server/internal/daemon/health_test.go
@@ -53,6 +53,9 @@ func TestHealthHandlerReportsCLIVersionAndActiveTaskCount(t *testing.T) {
if got, want := raw["active_task_count"], float64(3); got != want {
t.Errorf("active_task_count key: got %v, want %v", got, want)
}
+ if got, want := raw["admission_paused"], false; got != want {
+ t.Errorf("admission_paused key: got %v, want %v", got, want)
+ }
if got, want := raw["status"], "running"; got != want {
t.Errorf("status key: got %v, want %q", got, want)
}
@@ -75,6 +78,137 @@ func TestHealthHandlerReportsCLIVersionAndActiveTaskCount(t *testing.T) {
if resp.ActiveTaskCount != 3 {
t.Errorf("ActiveTaskCount: got %d, want 3", resp.ActiveTaskCount)
}
+ if resp.AdmissionPaused {
+ t.Error("AdmissionPaused: got true, want false")
+ }
+}
+
+func TestAdmissionHandlerPausesAndResumesNewClaims(t *testing.T) {
+ t.Parallel()
+
+ d := &Daemon{}
+
+ pauseRec := httptest.NewRecorder()
+ d.admissionHandler(true).ServeHTTP(
+ pauseRec,
+ httptest.NewRequest(http.MethodPost, "/admission/pause", nil),
+ )
+ if pauseRec.Code != http.StatusOK {
+ t.Fatalf("pause status: got %d, want 200: %s", pauseRec.Code, pauseRec.Body.String())
+ }
+ if d.tryEnterClaim() {
+ d.exitClaim()
+ t.Fatal("claim entered while admission was paused")
+ }
+
+ resumeRec := httptest.NewRecorder()
+ d.admissionHandler(false).ServeHTTP(
+ resumeRec,
+ httptest.NewRequest(http.MethodPost, "/admission/resume", nil),
+ )
+ if resumeRec.Code != http.StatusOK {
+ t.Fatalf("resume status: got %d, want 200: %s", resumeRec.Code, resumeRec.Body.String())
+ }
+ if !d.tryEnterClaim() {
+ t.Fatal("claim remained blocked after admission resumed")
+ }
+ d.exitClaim()
+}
+
+func TestAdmissionResumeDoesNotClearAutoUpdateBarrier(t *testing.T) {
+ t.Parallel()
+
+ d := &Daemon{pauseClaims: true, admissionPauseOwners: map[string]struct{}{"manual": {}}}
+ rec := httptest.NewRecorder()
+ d.admissionHandler(false).ServeHTTP(
+ rec,
+ httptest.NewRequest(http.MethodPost, "/admission/resume", nil),
+ )
+ if rec.Code != http.StatusOK {
+ t.Fatalf("resume status: got %d, want 200: %s", rec.Code, rec.Body.String())
+ }
+ if d.tryEnterClaim() {
+ d.exitClaim()
+ t.Fatal("manual resume cleared the independent auto-update claim barrier")
+ }
+}
+
+func TestAdmissionPauseReportsClaimDrainPending(t *testing.T) {
+ t.Parallel()
+
+ d := &Daemon{claimsInFlight: 1}
+ rec := httptest.NewRecorder()
+ d.admissionHandler(true).ServeHTTP(
+ rec,
+ httptest.NewRequest(http.MethodPost, "/admission/pause", nil),
+ )
+ if rec.Code != http.StatusAccepted {
+ t.Fatalf("pause status: got %d, want 202: %s", rec.Code, rec.Body.String())
+ }
+ var body map[string]any
+ if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
+ t.Fatalf("decode response: %v", err)
+ }
+ if got, want := body["claims_in_flight"], float64(1); got != want {
+ t.Fatalf("claims_in_flight: got %v, want %v", got, want)
+ }
+ if d.tryEnterClaim() {
+ d.exitClaim()
+ t.Fatal("new claim entered while a pre-pause claim was draining")
+ }
+}
+
+func TestAdmissionHandlerRejectsNonPost(t *testing.T) {
+ t.Parallel()
+
+ d := &Daemon{}
+ rec := httptest.NewRecorder()
+ d.admissionHandler(true).ServeHTTP(
+ rec,
+ httptest.NewRequest(http.MethodGet, "/admission/pause", nil),
+ )
+ if rec.Code != http.StatusMethodNotAllowed {
+ t.Fatalf("status: got %d, want 405", rec.Code)
+ }
+ if d.isAdmissionPaused() {
+ t.Fatal("GET request changed admission state")
+ }
+}
+
+func TestAdmissionOwnersAreIndependentAndPersistAcrossRestart(t *testing.T) {
+ t.Parallel()
+
+ root := t.TempDir()
+ d := &Daemon{cfg: Config{WorkspacesRoot: root}, admissionPauseOwners: make(map[string]struct{})}
+ request := func(path string) map[string]any {
+ rec := httptest.NewRecorder()
+ d.admissionHandler(strings.Contains(path, "/pause")).ServeHTTP(
+ rec,
+ httptest.NewRequest(http.MethodPost, path, nil),
+ )
+ if rec.Code != http.StatusOK {
+ t.Fatalf("%s status: %d: %s", path, rec.Code, rec.Body.String())
+ }
+ var body map[string]any
+ if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
+ t.Fatal(err)
+ }
+ return body
+ }
+ request("/admission/pause?owner=manual")
+ request("/admission/pause?owner=storage-guard")
+ body := request("/admission/resume?owner=storage-guard")
+ if body["admission_paused"] != true || body["owner_paused"] != false {
+ t.Fatalf("storage resume cleared manual owner: %v", body)
+ }
+
+ restarted := &Daemon{cfg: Config{WorkspacesRoot: root}, admissionPauseOwners: make(map[string]struct{})}
+ if err := restarted.loadAdmissionOwners(); err != nil {
+ t.Fatal(err)
+ }
+ if got := restarted.admissionPauseOwnersSnapshot(); len(got) != 1 || got[0] != "manual" {
+ t.Fatalf("persisted owners = %v, want [manual]", got)
+ }
}
// TestHealthHandlerReportsStartingUntilReady pins the liveness/readiness split: