Add opt-in Storage & retention with durable cleanup

[ymichael]

Human comments

What was wrong

BB had no unified way to see storage use across machines, reclaim thread files or abandoned development instances, or opt into automatic retention. Cleanup tied only to live notifications could also lose work while a plugin was disabled, reloading, or its machine was offline.

What changed

• Add the experimental, default-disabled Storage & retention plugin with a machine overview, cached disk measurements, thread/worktree/development breakdowns, explicit scans, and matching bb storage commands. Reports include hidden threads and link development data to its source checkout where possible.
• Offer file clearing, archived-file and large-file cleanup, orphan removal, failed worktree cleanup retry, and development-instance removal. Bulk work runs in tracked background jobs with progress and retryable errors. Different thread clears can run concurrently; conflicting maintenance is serialized per machine. Host workers validate paths and recheck eligibility before deleting files.
• Add hourly thread archiving/deletion, both defaulting to Never. Pinned members exempt their lifecycle group. Two separate switches default off: delete thread storage after future archives, and delete development data whose checkout is missing. Settings save immediately and quietly; failed saves restore the previous value. Destructive bulk actions keep inline confirmations, and running development instances require stop-and-remove confirmation.
• Archive storage cleanup persists pending work, waits for the shared 30-second undo grace, and retries once the thread stops and its machine reconnects. Unarchiving cancels it. File clearing preserves conversations and uploaded attachment originals. Development cleanup checks filesystem state on plugin startup, machine reconnect, SDK reconnect, and hourly when enabled; it can clean existing missing-checkout data. A live removal notification re-measures only that machine's ~/.bb-dev, queued behind any scan or cleanup already running there; later scans retry offline/busy machines or failed cleanup. Disabling the plugin stops its schedules and cancels active workers while retaining saved state.
• Announce successful provider environment removal after its lifecycle transition commits. Notifications are ephemeral; startup, reconnect, and scheduled filesystem scans recover missed notifications. Provider removal success does not guarantee the path was deleted. No core database tables or migration are added.
• Keep policy, report assembly, scan caching, and cleanup scheduling in the plugin/server. Host workers own filesystem measurement and deletion. Save development checkout metadata at launch. Canonicalize deleted checkout paths correctly and sweep processes for a whole batch of directories. Protocol version is 230 and Plugin SDK version is 0.6.27. Update CLI guides, plugin skills, configuration docs, and the Plugin Guide.

Public Plugin SDK changes

1. New event: bb.events.on("experimental_environment.removed", handler) receives { removal } after successful provider removal commits. The payload contains environmentId, removedAt, hostId, path, and providerOwnedPath; delivery is ephemeral. PluginThreadEventPayloads and the fake-host harness support it.
2. Changed host helper: experimental_killProcessesWithCwdUnder({ directories, graceMs? }) replaces the singular directory input with a directory list. Existing callers must pass directories: [directory]. Each sweep enumerates processes for the whole batch, including deleted paths resolved through their nearest existing ancestor. It uses SIGTERM then SIGKILL; Windows cwd enumeration remains a no-op.
3. New host helper: experimental_readProcessIdentity(pid) from @get-bb/plugin-sdk/host returns { command, startedAt } or null, the same probe bb's launcher uses before stopping a recorded PID. Storage & retention uses it to confirm a development instance's recorded launcher (entry path and start time) before signalling it.

Plugin-owned RPC and data contracts

• Inspection: state, hosts, host, scanHost, scanAll, and preview.
• Configuration: configure accepts archive/delete thresholds plus deleteStorageOnArchive and deleteDevDataOnCheckoutRemoval.
• Cleanup: clearThread, clearArchivedFiles, clearLargeFiles, removeOrphans, retryWorktreeCleanup, and removeDevInstances. The latter accepts all missing-checkout entries or selected names with explicit stopRunning.
• Background jobs: startClearLargeFiles, startClearArchivedFiles, and startCleanup (orphans, development, or worktrees). Machine responses expose cleanup progress/completion/failure.
• The plugin owns SQLite scan snapshots and pending archive cleanup; policy and the latest retention run use plugin KV storage. Development cleanup uses filesystem reconciliation and needs no removal-history tables, cursor, or persistent scan queue.

How you verified

• Turbo lint/typecheck pass for storage, server, CLI, SDK, database, Plugin SDK, and Plugin Guide.
• Tests pass: storage 69, focused server lifecycle/guide tests 164, database 588, SDK 109, CLI environment commands 35, Plugin SDK 365, and Plugin Guide 75.
• Development-cleanup regression exercises real host-worker filesystem cleanup after reload without removal history, machine and SDK reconnect, live removal notification, retry after failure, and disabled/offline behavior. Existing and unidentified source checkouts remain intact. Temporarily removing startup reconciliation makes the test fail because orphaned data remains; restoring it passes.
• Review hardening, each with a regression test that fails without the fix: the retention sweep re-selects every group right before archiving or deleting it (a thread pinned or unarchived mid-sweep is kept); stop-and-remove verifies the recorded launcher before signalling a PID; pending archive cleanups are dropped once their thread or machine returns 404; environment removal no longer rescans every machine.
• After rebasing onto main, turbo run lint typecheck test --filter='...[origin/main]' passes except builtin-plugins.test.ts > surfaces builtin app build failures…, a file-watch rebuild test unrelated to this plugin that times out only under full parallel load and passes 3/3 in isolation.
• Drizzle generation reports no schema changes and nothing to migrate. git diff --check passes.
• Earlier branch verification covered process sweeps, development path recovery, desktop/compact layouts, disposable cleanup, and app tests. No new live-browser or simulator pass is claimed for this backend simplification.
• CI fixes on this branch also align marketplace defaults and make Codex fixtures/lifecycle assertions tolerate Windows process-ID reuse.

Known limits: automation targets have no special retention exemption; pin threads that must be kept. Reports are cached snapshots, and filesystem eligibility is rechecked before cleanup. Windows does not enumerate working directories for process sweeps.

AGENT GENERATED

[patleeman]

Since this adds automatic thread deletion, I want to raise a need it touches.

What I'm trying to achieve: delete old threads to keep bb.db in check (by hand today, and with retention policies like these once this lands) without permanently losing their transcripts. When a thread is deleted, I'd like its full history to still exist somewhere on disk, in a plain file format like JSONL, so I can grep it, read it, or bring it back into context later.

Why it matters for me:

• My bb.db is 2.1 GB. The events table is about 1.1 GB of data plus about 400 MB of indexes. 83% of the event data (910 MB) belongs to 813 archived threads, 519 MB of it in threads archived more than 30 days ago.
• Deleting old archived threads is the only meaningful way to get that space back. Since deletion is permanent, I've been holding off.
• An automatic delete policy makes this sharper: threads would disappear without anyone looking at them first.

Why I can't cover it with a plugin today: thread.deleted fires after the soft delete, and the server doesn't wait for handlers. For a thread with no environment, the hard delete (and the cascade through events) runs in the same request, so the events are already gone by the time a handler could read them. For threads with an environment, the window lasts only until the daemon confirms storage deletion. The only reliable option left is to mirror every thread's events to disk continuously, just in case.

Related: #3598 (undelete).

AGENT GENERATED

[ymichael]

Since this adds automatic thread deletion, I want to raise a need it touches.

What I'm trying to achieve: delete old threads to keep bb.db in check (by hand today, and with retention policies like these once this lands) without permanently losing their transcripts. When a thread is deleted, I'd like its full history to still exist somewhere on disk, in a plain file format like JSONL, so I can grep it, read it, or bring it back into context later.

Why it matters for me:

• My bb.db is 2.1 GB. The events table is about 1.1 GB of data plus about 400 MB of indexes. 83% of the event data (910 MB) belongs to 813 archived threads, 519 MB of it in threads archived more than 30 days ago.
• Deleting old archived threads is the only meaningful way to get that space back. Since deletion is permanent, I've been holding off.
• An automatic delete policy makes this sharper: threads would disappear without anyone looking at them first.

Why I can't cover it with a plugin today: thread.deleted fires after the soft delete, and the server doesn't wait for handlers. For a thread with no environment, the hard delete (and the cascade through events) runs in the same request, so the events are already gone by the time a handler could read them. For threads with an environment, the window lasts only until the daemon confirms storage deletion. The only reliable option left is to mirror every thread's events to disk continuously, just in case.

Related: #3598 (undelete).

AGENT GENERATED

@patleeman tracking your feedback here: #4714 - I'm curious what you'd find useful in the snapshot / feel free to comment on the issue to clarify anything I missed