fix(checkpoints): reclaim checkpoint file data on workspace archive#1003
Conversation
Checkpoint file snapshots (`checkpoint_files` / `checkpoint_blobs`) were only ever pruned on checkpoint *insert*, via the insert-time retention sweep. Archived workspaces receive no further inserts, so their snapshot data was frozen in the DB forever — on large repos this grew the DB to well over a gigabyte, ~89% of which belonged to archived workspaces. Archiving now purges all of a workspace's `checkpoint_files`, GC's any blobs that lose their last reference, and drains the freed pages. The conversation survives: `conversation_checkpoints` and `turn_tool_activities` are kept, so chat history and tool timelines still render. `has_file_state` is a derived EXISTS column, so it flips to false and the restore path degrades to git-commit restore (the state archiving preserved anyway) instead of erroring. This closes the same gap the FK cascade already handles for the hard-delete path. Wired into both archive paths: `ops::workspace::archive` (GUI/CLI/IPC) and `scm::auto_archive_workspace` (PR-merge auto-archive). Closes #1002
There was a problem hiding this comment.
Pull request overview
This PR fixes unbounded SQLite growth caused by archived workspaces retaining all checkpoint file snapshots indefinitely, by purging checkpoint_files on archive, garbage-collecting orphaned checkpoint_blobs, and draining freed pages via incremental vacuum. It also adds targeted regression tests and documents the new archive behavior/trade-off (conversation preserved; per-turn file restore no longer available after restore from an archived workspace).
Changes:
- Add
Database::purge_workspace_checkpoint_files(workspace_id)to delete a workspace’s checkpoint file snapshot rows, GC orphan blobs, and run incremental vacuum. - Wire the purge into both manual archive (
src/ops/workspace.rs) and PR-merge auto-archive (src-tauri/src/commands/scm.rs). - Add regression tests covering reclamation + cross-workspace shared blob preservation, and update checkpoints docs to describe the new archive semantics.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/ops/workspace.rs |
Call snapshot purge during workspace archive; adds regression test ensuring archive reclaims file snapshots but keeps conversation checkpoint metadata. |
src/db/checkpoint.rs |
Introduces purge_workspace_checkpoint_files plus focused DB-level tests for reclamation and cross-workspace blob sharing safety. |
src-tauri/src/commands/scm.rs |
Adds best-effort purge to the PR-merge auto-archive path. |
site/src/content/docs/features/checkpoints-and-forking.mdx |
Documents that archiving purges file snapshots (conversation preserved; restore falls back to git-commit). |
| // Reclaim checkpoint file/blob bytes now. An archived workspace | ||
| // receives no further snapshot inserts, so the insert-time retention | ||
| // sweep never fires for it again — without this, its full-tree | ||
| // snapshots stay frozen in the DB forever (#1002). The conversation | ||
| // checkpoints and tool timelines survive; only file-restore | ||
| // snapshots go away, and the restore path degrades gracefully via | ||
| // the derived `has_file_state` guard. | ||
| db.purge_workspace_checkpoint_files(&workspace_id)?; | ||
| db.update_workspace_status(&workspace_id, &WorkspaceStatus::Archived, None)?; |
There was a problem hiding this comment.
Good catch — fixed in d072472. Reordered so the purge runs only after update_workspace_status succeeds. Now a failed status flip returns early via ? before any purge (snapshots stay intact), and if the purge itself fails the row is already Archived so the bytes are simply reclaimed on a later archive/delete. The purge is also now best-effort (logs a warning) rather than ?, since the archive has already committed once the row is Archived.
The single-transaction alternative isn't viable here: purge_workspace_checkpoint_files ends with an incremental vacuum, and SQLite forbids VACUUM/incremental_vacuum inside a transaction (see the note at src/db/mod.rs:124).
| // Reclaim checkpoint file/blob bytes on archive — archived | ||
| // workspaces never prune again on their own (#1002). Keeps | ||
| // the conversation; drops only file-restore snapshots. | ||
| let _ = db.purge_workspace_checkpoint_files(&ws_id); | ||
| db.update_workspace_status( |
There was a problem hiding this comment.
Fixed in d072472 — same reorder here. The status flip now happens first, and purge_workspace_checkpoint_files runs only inside if archived { ... }, so when the mutation fails (mutation_ok == false) the snapshots are never touched. Matches the ordering fix on the ops::workspace::archive path.
Address PR review: the checkpoint-file purge ran before `update_workspace_status`, so a failed status flip would leave an Active workspace stripped of its file-restore snapshots (data-loss regression). Reorder both archive paths (`ops::workspace::archive` and `scm::auto_archive_workspace`) to purge only after the row is successfully marked Archived. A failed flip now returns early before any purge (snapshots intact); a failed purge leaves an already-Archived workspace whose bytes are reclaimed on a later archive/delete. The purge is best-effort so a reclaim hiccup can't fail an archive that already committed. A single wrapping transaction isn't viable: the purge ends with an incremental vacuum, which SQLite forbids inside a transaction.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1003 +/- ##
==========================================
+ Coverage 81.12% 81.24% +0.12%
==========================================
Files 124 124
Lines 47637 47959 +322
==========================================
+ Hits 38644 38965 +321
- Misses 8993 8994 +1 🚀 New features to boost your workflow:
|
…int data The per-archive purge (#1002) only reclaims on the Active -> Archived transition, so workspaces that were already archived before it shipped keep their checkpoint file snapshots frozen in the DB — which is the bulk of the reported bloat (e.g. ~1.5 GB / 89% of rows belonging to already- archived workspaces). Nothing reclaimed those retroactively. Add a one-time, post-boot background sweep modeled on the existing #940 blob backfill: - Database::purge_archived_workspaces_checkpoint_files() deletes checkpoint_files for every workspace currently `archived` (batched per-workspace, matching the live path's granularity) and GCs orphaned blobs once. Cross-workspace shared blobs are preserved. - run_archived_checkpoint_sweep is a one-shot gated by app_settings.checkpoint_archived_sweep_done. When it purges rows it re-arms the shared checkpoint_blob_space_reclaim_done marker so the existing post-backfill VACUUM drains this freelist too — a single file rewrite reclaims both the #940 dedup and #1002 purge pages (a full VACUUM is required; INCREMENTAL auto-vacuum only trims the file tail on the fragmented freelists these DBs carry). - main.rs runs it between the backfill and the space reclaim, after boot acknowledgement so it can't trip the updater's boot-probation window. A sweep failure is non-fatal to the reclaim and retries next launch. Also correct the archive doc wording: snapshot-only checkpoints (the norm, commit_hash = None) drop the per-turn "restore files" affordance outright (the rollback UI hides it); only legacy commit-hash checkpoints fall back to a git-commit restore. Refs #1002
| // the failure modes safe — and the purge is best-effort because the | ||
| // archive has already committed, so a reclaim hiccup just defers the | ||
| // space saving to the next archive/delete or manual vacuum rather than | ||
| // failing an archive that actually happened. (A single transaction | ||
| // isn't an option: the purge ends with an incremental vacuum, which |
There was a problem hiding this comment.
Good catch — the PR description was stale (written before the reorder in d072472). Updated it to match the code: ops::workspace::archive flips the row to Archived first, then purges as best-effort (logs a warning and continues). A failed status flip returns early before any purge, so a workspace is never left Active without its snapshots; a failed purge just defers reclamation to a later archive/delete. The code is the intended behavior — only the description was wrong.
|
|
||
| A per-workspace retention cap bounds how many recent checkpoints retain their file-restore snapshot. The cap defaults to **50** and is configurable under **Settings → Storage → Checkpoint file retention** (range `[1, 1000]`). Older checkpoints stay visible in chat history and tool-activity timelines, but the "restore files" affordance is dropped for them — only the `checkpoint_files` reference rows are pruned, `conversation_checkpoints` and `turn_tool_activities` survive. | ||
|
|
||
| **Archiving reclaims file snapshots.** The retention cap only runs when a new checkpoint is written, and an archived workspace records no more checkpoints — so its snapshots would otherwise stay frozen in the database indefinitely. To keep the database bounded, archiving a workspace purges all of its `checkpoint_files` (and garbage-collects any blobs that lose their last reference), then drains the freed pages. The conversation itself is untouched: `conversation_checkpoints` and `turn_tool_activities` survive, so chat history and tool timelines still render after a restore. Only the per-turn file-restore affordance is dropped for snapshot-only checkpoints (the norm) — the rollback UI simply hides the "restore files" option for them; legacy checkpoints that recorded a commit hash still fall back to a git-commit restore. Restoring the whole workspace still rebuilds its worktree from the preserved branch tip regardless. Deleting a workspace already reclaimed this data via the foreign-key cascade; this closes the same gap for the archive path. |
There was a problem hiding this comment.
Fixed in 23346aa. Reworded so it no longer implies immediate shrinkage: the per-archive path runs a bounded incremental vacuum that returns freed pages to SQLite's reuse pool immediately (so the DB stops growing), but the on-disk file may not shrink until a full VACUUM compacts a fragmented freelist — which the one-time sweep does once, or a manual VACUUM on demand.
| /// Drop **all** snapshot file rows for a workspace's checkpoints, GC any | ||
| /// `checkpoint_blobs` that lose their last reference, and drain the freed | ||
| /// pages. Returns the number of `checkpoint_files` rows deleted. | ||
| /// |
There was a problem hiding this comment.
Fixed in 23346aa. The doc comment now says the method runs a best-effort incremental vacuum and explicitly notes it's bounded (INCREMENTAL_VACUUM_PAGES_AFTER_DELETE) and does not guarantee the file shrinks — a fragmented freelist only fully compacts under VACUUM, which the one-time sweep pairs with its purge.
Address PR review: the per-archive purge runs a bounded incremental vacuum, not a full drain — it makes freed pages reusable but doesn't guarantee the on-disk file shrinks (fragmented freelists only compact under VACUUM). Soften the "drains the freed pages" claim in both the purge_workspace_checkpoint_files doc comment and the checkpoints docs page, and cross-reference the one-time sweep's full VACUUM. No behavior change.
Summary
Fixes #1002. Conversation-checkpoint file snapshots (
checkpoint_files/checkpoint_blobs) were only pruned on checkpoint insert, via the insert-time retention sweep. Archived workspaces record no more checkpoints, so their snapshots stayed frozen in the database forever. On the reporter's large repo (~6k tracked files) this grewclaudette.dbto 1.8 GB, ~89% of which belonged to already-archived workspaces.This PR does two things:
Reclaim on archive (going forward). Archiving a workspace now purges all of its
checkpoint_files, garbage-collects anycheckpoint_blobsthat lose their last reference, and runs a bounded incremental vacuum. Freed pages become reusable immediately so the DB stops growing; the file on disk may not shrink until a fullVACUUMcompacts it (see below). The conversation is preserved —conversation_checkpointsandturn_tool_activitiesare kept, so chat history and tool timelines still render after a restore.has_file_stateis a derivedEXISTS(...)column, so for snapshot-only checkpoints (the norm) the per-turn "restore files" affordance is dropped (the rollback UI hides it); legacy commit-hash checkpoints still fall back to a git-commit restore. Restoring the whole workspace rebuilds its worktree from the preserved branch tip regardless.Reclaim the existing bloat (one-time). A post-boot background sweep (modeled on the Checkpoint storage: checkpoint_files stores undeduplicated file copies → 6.9 GB DB (98% duplicate) #940 blob backfill) purges the
checkpoint_filesof workspaces that were already archived before this shipped, then a singleVACUUMreturns the freed pages to the OS. Gated by a completion marker so it runs exactly once. This is what shrinks a database that grew before archive-time reclamation existed — incremental auto-vacuum only trims the file tail, so the fragmented freelists these DBs carry need a fullVACUUM.flowchart LR subgraph Boot[one-time post-boot maintenance] direction LR BF[legacy blob backfill] --> SW[archived-workspace sweep] --> RC[single VACUUM<br/>reclaims both freelists] end A[Archive workspace<br/>branch kept] --> P[purge checkpoint_files<br/>+ GC orphan blobs<br/>+ incremental vacuum]Wired into both live archive paths:
ops::workspace::archive(GUI / CLI / IPC) andscm::auto_archive_workspace(PR-merge auto-archive).Complexity Notes
Archivedfirst, then purge.ops::workspace::archiveruns the purge as best-effort (logs a warning and continues), so a reclaim hiccup can't fail an archive that already committed; a failed status flip returns early before any purge, so a workspace is never left Active without its snapshots.scm::auto_archive_workspacepurges only insideif archived { … }. A single wrapping transaction isn't viable — the purge ends with an incremental vacuum, and SQLite forbidsVACUUMinside a transaction.checkpoint_blob_space_reclaim_donemarker (exactly asrun_backfilldoes after a dedup migration), so the existing post-backfillVACUUMdrains both the Checkpoint storage: checkpoint_files stores undeduplicated file copies → 6.9 GB DB (98% duplicate) #940 dedup freelist and this purge's freelist in one file rewrite. Ordering inmain.rsis backfill → sweep → reclaim, all after boot acknowledgement so it can't trip the updater's boot-probation window.NOT EXISTSkeeps any blob a non-archived workspace still references. Pinned by tests.Test Steps
cargo test -p claudette --lib— all pass (incl. the new tests).cargo test -p claudette --lib purge_workspace_checkpoint_filescargo test -p claudette --lib purge_archived_workspacescargo test -p claudette --lib archive_purges_checkpoint_files_but_keeps_conversationcargo test -p claudette --lib checkpoint_backfill::tests(sweep purges archived / keeps active / marker-gated / re-arms reclaim)checkpoint_filesare gone while its chat history + checkpoint rows survive:0.archived-workspace checkpoint sweep complete issue=1002 purged_rows=<n>and the DB shrinks after the post-bootVACUUM.Checklist
site/.../checkpoints-and-forking.mdx)