Skip to content

Checkpoints: a second set of names for the same bytes - #9

Merged
BhaveshThapar merged 1 commit into
mainfrom
checkpoints
Aug 24, 2026
Merged

BhaveshThapar merged 1 commit into
mainfrom
checkpoints

Conversation

@BhaveshThapar

Copy link
Copy Markdown
Owner

What

Db::checkpoint(dir) writes a complete, openable database into another directory by hard-linking the live SSTables and writing a fresh manifest naming them.

The bytes are not moved, so a checkpoint of a gigabyte costs milliseconds — and the source may go on to compact and unlink its own names, because a file's contents outlive its last link rather than its first.

  • Fs gains hard_link, which refuses to link onto an existing name.
  • Manifest::write_fresh writes a complete manifest into a directory that has none, and refuses one that already has.
  • The MemTable is flushed on the way in, and only flushed: Db::flush would also drain every pending compaction, making the stall a function of how many SSTables happened to accumulate rather than of how much is buffered.
  • The live set is read under the flush lock, so a compaction that published mid-checkpoint cannot leave a manifest naming a table nobody linked.

What turned out not to be needed

A read-only Manifest view. The live table set is already in memory under a lock the checkpoint takes anyway, so there is nothing to read the manifest for.

Tests

  • a_checkpoint_shares_its_bytes_with_the_source — same inode, link count ≥ 2. It is a link, not a copy.
  • a_checkpoint_survives_the_source_losing_the_names_it_linked — three rounds of rewrite-and-compact on the source, then the linked names removed, then the checkpoint opens with its original contents. The removal is direct rather than waiting for size-tiered compaction to choose that run: what a compaction does to a merged table is exactly this, and depending on a heuristic to pick the right one would make the test's coverage a function of the heuristic.
  • Plus: buffered writes are included, later writes are not, a second checkpoint over the first is refused, and the source is unaffected.

All tests green, clippy --all-targets --all-features -D warnings and fmt clean.

Db::checkpoint writes a complete, openable database into another directory by
hard-linking the live SSTables and writing a fresh manifest naming them. The
bytes are not moved, so a checkpoint of a gigabyte costs milliseconds — and the
source may go on to compact and unlink its own names, because a file's contents
outlive its last link rather than its first.

The MemTable is flushed on the way in, and only flushed. Db::flush would also
drain every pending compaction, which is a merge of unbounded size and would
make the stall a function of how many SSTables happened to accumulate rather
than of how much is buffered.

The live set is read under the flush lock, so a compaction that published
mid-checkpoint cannot leave a manifest naming a table nobody linked. And a
checkpoint into a directory that is not empty is refused, twice — once before
anything is linked, once by the manifest — because a directory describing two
different states is worse than a failed checkpoint.

A read-only manifest view turned out not to be needed. The live table set is
already in memory under a lock the checkpoint takes anyway, so there is nothing
to read the manifest for; the note that said otherwise was written before the
flush lock was there to use.

Tests assert the mechanism rather than describing it: the two paths share an
inode and the link count is at least two, so it is a link and not a copy; and a
checkpoint opens with its original contents after the source has been churned
through three rounds of rewrite-and-compact and then had the names it linked
removed. The removal is direct rather than waiting for size-tiered compaction to
choose that particular run — what a compaction does to a merged table is exactly
this, and depending on a heuristic to pick the right one would make the test's
coverage a function of the heuristic.
@BhaveshThapar
BhaveshThapar merged commit ad581d4 into main Aug 24, 2026
1 check failed
@BhaveshThapar
BhaveshThapar deleted the checkpoints branch August 24, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant