storage(swm): bound public snapshot retention with reference-aware garbage collection

[branarakic-agent]

Problem

FileWorkspacePublicSnapshotStore persists public Shared Working Memory (SWM) snapshots as immutable N-Quads files under <dataDir>/swm-public-snapshots.

Snapshots are written by both publishing and synchronization paths. Content addressing avoids storing byte-identical snapshots more than once, but any change produces a new digest and a new complete snapshot. The store currently exposes write/read/page operations without a corresponding retention, deletion, or capacity-management lifecycle.

On a long-running node, the directory can therefore grow without bound. Eventually the filesystem can run out of space, preventing the DKG node and other colocated services from writing. This can cascade into failures that appear unrelated to SWM synchronization.

These files are separate from the materialized triple store:

• the triple store contains indexed/materialized graph state;
• the snapshot store contains immutable payloads used for verification, synchronization, peer serving, and recovery;
• materialization alone does not guarantee that the exact historical payload can be reconstructed from the triple store.

This means deleting every snapshot immediately after materialization would be unsafe, but retaining every snapshot indefinitely is also unsafe.

Expected behavior

Public snapshot storage should have a bounded, observable lifecycle that:

• never removes snapshots that are active, pending, pinned, or required by current manifests/metadata;
• distinguishes locally published snapshots from peer-fetched cache entries;
• allows materialized and recoverable snapshots to become eligible for eviction after a grace period;
• prevents snapshot growth from exhausting the node filesystem;
• handles an evicted peer snapshot as a recoverable cache miss where the protocol permits refetching;
• fails gracefully before ENOSPC can cascade into unrelated node services.

Proposed solution

1. Add snapshot lifecycle metadata

Maintain a bounded/persistent index containing at least:

• digest/ref and byte size;
• creation and last-access timestamps;
• source: locally published or peer-fetched;
• materialization/verification status;
• current reference, pending-operation, and pin status;
• replication or refetchability status when known.

2. Implement reference-aware mark-and-sweep GC

Mark snapshots that must remain available, including those referenced by current manifests, pending publication/finalization, active synchronization, or explicit operator pins.

A snapshot should become sweep-eligible only when it is:

1. successfully verified and materialized;
2. not marked by any active reference or operation;
3. older than the configured grace period; and
4. safely recoverable, replicated, or covered by the policy for its source type.

Peer-fetched snapshots can use LRU/age-based eviction once eligible. Locally published snapshots should receive stronger protection until durability or sufficient replication is established.

3. Add capacity controls

Suggested configuration surface:

• maxBytes;
• high and low watermarks;
• minimum retention/grace period;
• GC interval;
• separate policies for local and peer-fetched snapshots;
• dry-run/report-only mode for rollout.

GC should run periodically and before writes when the high watermark is reached, sweeping enough eligible data to return below the low watermark. If insufficient eligible data exists, the node should enter a clear degraded state rather than consume the filesystem's final free space.

4. Add observability and operator tooling

Expose:

• snapshot count and total bytes;
• marked, pinned, and GC-eligible bytes;
• locally published versus peer-cached bytes;
• last successful GC and reclaimed bytes;
• failed eviction/refetch counts;
• filesystem watermark status.

Provide an operator command for a dry-run GC report and an explicit safe sweep. Documentation should recommend a dedicated filesystem or object store for durable snapshot storage where appropriate.

5. Preserve correctness under concurrency and restart

GC must coordinate with concurrent reads, writes, publication, and sync. Deletion should be atomic from the store's perspective, tolerate interrupted sweeps, and rebuild or reconcile its index after restart without requiring all snapshots to be loaded into memory.

Acceptance criteria

• Snapshot storage can be configured with a byte budget and watermarks.
• Referenced, pending, pinned, and in-use snapshots are never swept.
• Eligible materialized peer snapshots are evicted oldest-first when capacity is exceeded.
• Locally published snapshots are retained until the configured durability/replication condition is satisfied.
• An evicted recoverable snapshot follows the existing network recovery path on demand.
• Concurrent reads/writes and interrupted GC cannot leave partial snapshot files or corrupt lifecycle metadata.
• Startup reconciliation is bounded and tested against an existing populated directory.
• Disk-pressure and GC metrics are exposed through node health/telemetry.
• The node reports a specific capacity/degraded error before filesystem exhaustion.
• Unit and integration tests cover marking, grace periods, watermarks, local-versus-peer policy, restart, and refetch-after-eviction.

Related work

• perf(sync): cache validation of unchanged public snapshot files #1960 optimizes repeated validation of unchanged cached snapshots. That reduces read/CPU amplification but does not bound retained bytes; the two improvements are complementary.