Add freeze() to drop per-shard codes/residuals for read-only indexes#45
Merged
Conversation
After indexing, FastPlaid keeps both the per-shard files
({i}.codes.npy, {i}.residuals.npy) used by update/delete and a merged
file used at search time, roughly doubling on-disk storage. For a
read-only index the shards are redundant.
freeze() ensures the merged file is up to date, deletes every per-shard
codes/residuals file, and writes "frozen": true into metadata.json. The
loader skips the chunk scan when frozen and mmaps the merged file
directly. Subsequent update/delete calls raise; search is unaffected.
unfreeze() slices merged_codes.npy / merged_residuals.npy back into
per-shard {i}.codes.npy / {i}.residuals.npy using each shard's
doclens.{i}.json, then drops the frozen flag from metadata.json so
update/delete work again.
Shard .npy files are written via a small helper that matches
tch::Tensor::write_npy byte-for-byte (Rust-style shape formatting,
explicit byte order, 16-byte prologue alignment). metadata.json is
re-pretty-printed with two-space indent to match serde_json's
to_writer_pretty output.
Adds test_freeze_unfreeze_roundtrip_byte_identical which creates a
multi-shard index, copies the directory, freezes+unfreezes the
original, and asserts every file (except the regenerated merged_*
manifests) is byte-identical to the snapshot.
Python text-mode open() translates \n to \r\n on Windows, breaking byte equality against the Rust serde_json::to_writer_pretty output (which always uses \n). Pass newline="\n" explicitly so the freeze/unfreeze roundtrip is byte-identical on all platforms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FastPlaid.freeze()which deletes the per-shard{i}.codes.npy/{i}.residuals.npyfiles and sets"frozen": trueinmetadata.json. The merged file remains and becomes the sole storage, cutting on-disk usage roughly in half.FastPlaid.unfreeze()as the inverse: slicesmerged_codes.npy/merged_residuals.npyback into per-shard files usingdoclens.{i}.json, then drops thefrozenkey. The roundtrip is byte-identical — every file matches the original.mmapthe merged file directly (slightly faster reload, less peak RAM).update()anddelete()on a frozen index raiseRuntimeError— both paths operate per shard and can't run without re-sharding (callunfreeze()first).Motivation
After indexing, FastPlaid keeps two copies of the same compressed data on disk:
{i}.codes.npy/{i}.residuals.npy(used byupdate.rs/delete.rs)merged_codes.npy/merged_residuals.npy(used at search time viaStridedTensor)For workloads that won't mutate the index again, the shards are dead weight.
freeze()makes that explicit and reclaims the space without any Rust changes.unfreeze()is there as an escape hatch if you change your mind.Byte-identical roundtrip
unfreeze()reconstructs shards via a small helper that matchestch::Tensor::write_npybyte-for-byte:(R,C,)(trailing comma, no spaces) instead ofnp.save's(R, C).<byte order even for 1-byte types (np.savewrites|).metadata.jsonis written withindent=2to matchserde_json::to_writer_pretty. Thefrozenkey is fully removed onunfreeze()so the file matches its pre-freeze bytes exactly.What's untouched
create.rs) and the chunk-then-merge flow are unchanged. Peak RAM during create is preserved.StridedTensor.Test plan
test_freeze_removes_shards_and_search_still_works— search results identical before and afterfreeze(), shard files gone, merged files retained.test_freeze_persists_across_reload— a frozen index reopens correctly in a newFastPlaidinstance and returns the same results.test_freeze_update_raises—update()on a frozen index raisesRuntimeError.test_freeze_delete_raises—delete()on a frozen index raisesRuntimeError.test_freeze_idempotent— callingfreeze()twice is a no-op.test_freeze_unfreeze_roundtrip_byte_identical— create a multi-shard index, copy the directory, freeze+unfreeze the original, assert every file is byte-identical to the snapshot viafilecmp.cmp(shallow=False).test_unfreeze_restores_update_capability— afterunfreeze(),update()works again.test_unfreeze_idempotent— callingunfreeze()on a non-frozen index is a no-op.TestBasicCreateAndSearchandTestUpdatesuites still pass.