Repository navigation
Checkpoints: a second set of names for the same bytes - #9
Merged
Merged
Conversation
Db::checkpoint writes a complete, openable database into another directory by hard-linking the live SSTables and writing a fresh manifest naming them. The bytes are not moved, so a checkpoint of a gigabyte costs milliseconds — and the source may go on to compact and unlink its own names, because a file's contents outlive its last link rather than its first. The MemTable is flushed on the way in, and only flushed. Db::flush would also drain every pending compaction, which is a merge of unbounded size and would make the stall a function of how many SSTables happened to accumulate rather than of how much is buffered. The live set is read under the flush lock, so a compaction that published mid-checkpoint cannot leave a manifest naming a table nobody linked. And a checkpoint into a directory that is not empty is refused, twice — once before anything is linked, once by the manifest — because a directory describing two different states is worse than a failed checkpoint. A read-only manifest view turned out not to be needed. The live table set is already in memory under a lock the checkpoint takes anyway, so there is nothing to read the manifest for; the note that said otherwise was written before the flush lock was there to use. Tests assert the mechanism rather than describing it: the two paths share an inode and the link count is at least two, so it is a link and not a copy; and a checkpoint opens with its original contents after the source has been churned through three rounds of rewrite-and-compact and then had the names it linked removed. The removal is direct rather than waiting for size-tiered compaction to choose that particular run — what a compaction does to a merged table is exactly this, and depending on a heuristic to pick the right one would make the test's coverage a function of the heuristic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Db::checkpoint(dir)writes a complete, openable database into another directory by hard-linking the live SSTables and writing a fresh manifest naming them.The bytes are not moved, so a checkpoint of a gigabyte costs milliseconds — and the source may go on to compact and unlink its own names, because a file's contents outlive its last link rather than its first.
Fsgainshard_link, which refuses to link onto an existing name.Manifest::write_freshwrites a complete manifest into a directory that has none, and refuses one that already has.Db::flushwould also drain every pending compaction, making the stall a function of how many SSTables happened to accumulate rather than of how much is buffered.What turned out not to be needed
A read-only
Manifestview. The live table set is already in memory under a lock the checkpoint takes anyway, so there is nothing to read the manifest for.Tests
a_checkpoint_shares_its_bytes_with_the_source— same inode, link count ≥ 2. It is a link, not a copy.a_checkpoint_survives_the_source_losing_the_names_it_linked— three rounds of rewrite-and-compact on the source, then the linked names removed, then the checkpoint opens with its original contents. The removal is direct rather than waiting for size-tiered compaction to choose that run: what a compaction does to a merged table is exactly this, and depending on a heuristic to pick the right one would make the test's coverage a function of the heuristic.All tests green,
clippy --all-targets --all-features -D warningsandfmtclean.