Skip to content

feat!: retain indexed text and serve sequence bases by id#12

Merged
sriram98v merged 2 commits into
mainfrom
feat/serve-sequences
Jul 25, 2026
Merged

feat!: retain indexed text and serve sequence bases by id#12
sriram98v merged 2 commits into
mainfrom
feat/serve-sequences

Conversation

@sriram98v

Copy link
Copy Markdown
Owner

Summary

Adds sequence(SeqId) and sequence_by_header(&str) to both FmIndex and BidirFmIndex, returning the bases of one indexed sequence as a borrowed slice in O(1), with the trailing sentinel excluded.

Until now an FM-index could only recover text by LF-walking backwards one symbol at a time. Callers that need random-access substrings — an aligner rescoring a read against a candidate diagonal, say — therefore had to keep a full second copy of every reference alongside the index. This retains the concatenated text instead, so those substrings are free.

The trade is deliberate and worth stating plainly:

  • Cost: ~n bytes of resident and serialized size, and the build-time peak-memory reduction that the old drop(text) bought is given up.
  • Benefit: callers stop duplicating the references. For the downstream aligner this is net-neutral on disk (it already stored those bytes itself) and removes a whole side table from its hot loop.

Only the forward half of a BidirFmIndex retains text. The reverse half's copy is a redundant reversal that is never served — every accessor delegates to fwd — so it is released at build time via forget_text(), keeping the overhead at ~n rather than ~2n. A test asserts this.

The text is stored unpacked. The alphabet fits in a nibble, but a packed representation could not be borrowed without unpacking into a fresh allocation on every call — precisely the cost these accessors exist to avoid.

Bases are returned as alphabet codes (A = 1, C = 2, …), not ASCII, matching how the index stores them. encode_byte / encode_char / decode_char are now re-exported at the crate root so callers can cross that boundary without reaching into the alphabet module.

Breaking

The serialized layout gains a text field, so indexes written by 0.2.0 and earlier are rejected by from_bytes and must be rebuilt. Version bumped to 0.3.0.

Related issues

Follows up #9 / #10 (the SeqId change). Together these let a downstream aligner drop both its header→id map and its copy of every reference sequence.

Type of change

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor / internal change
  • Documentation

Checklist

  • cargo fmt --all -- --check passes
  • cargo clippy --all-features -- -D warnings passes
  • cargo test passes (CPU)
  • cargo test --features gpu passes locally, if a GPU path changed
  • Added/updated CPU-vs-GPU parity tests for any new GPU kernel — n/a, no new kernel; the GPU build path is updated to populate and then release text consistently with the CPU path
  • Updated CHANGELOG.md for user-visible changes — filed under a new [0.3.0] heading rather than [Unreleased], matching how 0.2.0 was recorded
  • Updated docs / README where relevant — docs/src/reference/api.md now lists the new accessors

Testing

Every CI step run locally on Linux, plus the deploy.yml steps, all green:

step command result
fmt cargo fmt --check pass
clippy cargo clippy --all-features -- -D warnings pass
build cargo build --all-features pass
test cargo test pass — 114 lib + 34 integration + 2 doc
gpu cargo test --features gpu pass — +91 GPU tests, real device
deploy cargo build --target wasm32-unknown-unknown --features wasm --release pass
deploy mdbook build pass

Clippy and build were re-run after touching the modified files, to confirm the #[cfg(feature = "gpu")] paths were genuinely rechecked rather than served from cache.

Six new tests in src/fm_index/bidir_index.rs:

  • sequence_round_trips_every_reference — three references of different lengths, so a boundary off-by-one cannot pass by luck
  • sequence_excludes_the_sentinel — no separator leaks into a returned slice
  • sequence_out_of_range_is_none
  • sequence_by_header_agrees_with_sequence_by_id
  • sequence_survives_serialization — guards the new field's round-trip
  • reverse_half_carries_no_duplicate_text — pins the ~n rather than ~2n invariant

Downstream verification

Integrated against the PREMISE aligner, which was refactored onto this API. On a fixture of pure uppercase ACGTN references, all four output TSVs are byte-identical before and after, and the whole parallel alignment stage got 8.2% faster (p = 0.00) with the reference-side lookups removed.

sriram98v added 2 commits July 24, 2026 17:29
Add `sequence(SeqId)` / `sequence_by_header(&str)` on both `FmIndex` and
`BidirFmIndex`, returning the bases of one indexed sequence as a borrowed
slice in O(1) with the trailing sentinel excluded.

An FM-index could previously only recover text by LF-walking backwards one
symbol at a time, so callers needing random-access substrings — an aligner
rescoring a read against a candidate diagonal, say — had to keep a full
second copy of every reference alongside the index. Retaining the
concatenated text instead costs ~n bytes and gives up the build-time peak
reduction that dropping it bought, but makes those substrings free.

Only the forward half of a `BidirFmIndex` keeps text; the reverse half's
copy is a redundant reversal that is never served, so it is released at
build time and the overhead stays at ~n rather than ~2n.

The text is stored unpacked. The alphabet fits in a nibble, but a packed
representation could not be borrowed without unpacking into a fresh
allocation per call, which is the cost these accessors exist to avoid.

BREAKING CHANGE: the serialized layout gains a `text` field, so indexes
written by 0.2.0 and earlier are rejected by `from_bytes` and must be
rebuilt.
@sriram98v
sriram98v merged commit f0085f5 into main Jul 25, 2026
4 checks passed
@sriram98v
sriram98v deleted the feat/serve-sequences branch July 25, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant