feat!: retain indexed text and serve sequence bases by id#12
Merged
Conversation
added 2 commits
July 24, 2026 17:29
Add `sequence(SeqId)` / `sequence_by_header(&str)` on both `FmIndex` and `BidirFmIndex`, returning the bases of one indexed sequence as a borrowed slice in O(1) with the trailing sentinel excluded. An FM-index could previously only recover text by LF-walking backwards one symbol at a time, so callers needing random-access substrings — an aligner rescoring a read against a candidate diagonal, say — had to keep a full second copy of every reference alongside the index. Retaining the concatenated text instead costs ~n bytes and gives up the build-time peak reduction that dropping it bought, but makes those substrings free. Only the forward half of a `BidirFmIndex` keeps text; the reverse half's copy is a redundant reversal that is never served, so it is released at build time and the overhead stays at ~n rather than ~2n. The text is stored unpacked. The alphabet fits in a nibble, but a packed representation could not be borrowed without unpacking into a fresh allocation per call, which is the cost these accessors exist to avoid. BREAKING CHANGE: the serialized layout gains a `text` field, so indexes written by 0.2.0 and earlier are rejected by `from_bytes` and must be rebuilt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
sequence(SeqId)andsequence_by_header(&str)to bothFmIndexandBidirFmIndex, returning the bases of one indexed sequence as a borrowed slice in O(1), with the trailing sentinel excluded.Until now an FM-index could only recover text by LF-walking backwards one symbol at a time. Callers that need random-access substrings — an aligner rescoring a read against a candidate diagonal, say — therefore had to keep a full second copy of every reference alongside the index. This retains the concatenated text instead, so those substrings are free.
The trade is deliberate and worth stating plainly:
drop(text)bought is given up.Only the forward half of a
BidirFmIndexretains text. The reverse half's copy is a redundant reversal that is never served — every accessor delegates tofwd— so it is released at build time viaforget_text(), keeping the overhead at ~n rather than ~2n. A test asserts this.The text is stored unpacked. The alphabet fits in a nibble, but a packed representation could not be borrowed without unpacking into a fresh allocation on every call — precisely the cost these accessors exist to avoid.
Bases are returned as alphabet codes (
A = 1,C = 2, …), not ASCII, matching how the index stores them.encode_byte/encode_char/decode_charare now re-exported at the crate root so callers can cross that boundary without reaching into thealphabetmodule.Breaking
The serialized layout gains a
textfield, so indexes written by 0.2.0 and earlier are rejected byfrom_bytesand must be rebuilt. Version bumped to 0.3.0.Related issues
Follows up #9 / #10 (the
SeqIdchange). Together these let a downstream aligner drop both its header→id map and its copy of every reference sequence.Type of change
Checklist
cargo fmt --all -- --checkpassescargo clippy --all-features -- -D warningspassescargo testpasses (CPU)cargo test --features gpupasses locally, if a GPU path changedtextconsistently with the CPU pathCHANGELOG.mdfor user-visible changes — filed under a new[0.3.0]heading rather than[Unreleased], matching how 0.2.0 was recordeddocs/src/reference/api.mdnow lists the new accessorsTesting
Every CI step run locally on Linux, plus the
deploy.ymlsteps, all green:cargo fmt --checkcargo clippy --all-features -- -D warningscargo build --all-featurescargo testcargo test --features gpucargo build --target wasm32-unknown-unknown --features wasm --releasemdbook buildClippy and build were re-run after
touching the modified files, to confirm the#[cfg(feature = "gpu")]paths were genuinely rechecked rather than served from cache.Six new tests in
src/fm_index/bidir_index.rs:sequence_round_trips_every_reference— three references of different lengths, so a boundary off-by-one cannot pass by lucksequence_excludes_the_sentinel— no separator leaks into a returned slicesequence_out_of_range_is_nonesequence_by_header_agrees_with_sequence_by_idsequence_survives_serialization— guards the new field's round-tripreverse_half_carries_no_duplicate_text— pins the ~n rather than ~2n invariantDownstream verification
Integrated against the PREMISE aligner, which was refactored onto this API. On a fixture of pure uppercase ACGTN references, all four output TSVs are byte-identical before and after, and the whole parallel alignment stage got 8.2% faster (p = 0.00) with the reference-side lookups removed.