genomeGenerate: libsais suffix-array builder, selected by --limitGenomeGenerateRAM

CHANGELOG.md

@@ -13,6 +13,46 @@ Sections commonly used: Features, Bug fixes, Other changes.
 
 ### Features
 
+- **`--limitGenomeGenerateRAM` now picks the suffix-array builder.** The flag
+  was accepted and warned about; it now decides between two constructions of
+  the same array:
+
+  - **libsais** (new dependency, see Bumps) when its estimated peak fits the
+    limit. It is an in-memory SA-IS and needs roughly `17` bytes per
+    suffix-array entry, that is `34 × n_genome` for both strands.
+  - **caps-sa's external-memory path** otherwise, which spills to disk and
+    needs a fraction of the RAM.
+
+  `--genomeSAsparseD` above 1 always takes caps-sa, the only builder with a
+  sparse arm. The chosen path and both numbers are logged, so the decision is
+  visible in the run rather than inferred.
+
+  Output cannot differ: the suffix array of a text is unique, and both arms
+  sort the same per-segment sentinel-transformed text. That is checked at four
+  levels, the last of which is the shipping path rather than a unit-test-only
+  one: against caps-sa on three fixtures, with the `i32` large-alphabet arm
+  forced, through the same `PackedStreamWriter` `genomeGenerate` writes with,
+  and on real genomes.
+
+  Construction time and peak RSS, `--runThreadN 8`, same machine, warm cache,
+  `SA` and `SAindex` byte-identical in every row:
+
+  | genome | libsais | caps-sa | ratio |
+  |---|---|---|---|
+  | yeast R64-1-1, 12 Mb | 1.0 s / 0.39 GB | 1.7 s / 0.22 GB | 1.6x |
+  | human chr1, 249 Mb | 27.4 s / 6.5 GB | 47.2 s / 5.6 GB | 1.7x |
+  | GRCh38 primary, 3.1 Gb | 641.6 s / 86.8 GB | 1040.4 s / 14.4 GB | 1.6x |
+
+  The ratio holds across three orders of magnitude even though libsais runs
+  single-threaded here: caps-sa buys its small footprint with about six times
+  the total CPU work for the same array (on GRCh38, 726 s of user time against
+  6135 s).
+
+  The mammalian row needed `--limitGenomeGenerateRAM 0` to produce. At the
+  31 GB default a GRCh38 build is declined, because libsais would need an
+  estimated 106.7 GB against a measured 86.8 GB peak, so it takes the caps-sa
+  path unless the user states they have the RAM.
+
 - **STARsolo single-cell quantification (`--soloType`)** — the 10x
   Chromium / plate-based count-matrix pipeline, ported from STAR and
   verified against real STARsolo (#90).
@@ -116,6 +156,13 @@ Sections commonly used: Features, Bug fixes, Other changes.
 - `caps-sa` → `0.5` (adds `build_ext_mem_for_filter*`; see the caps-sa
   v0.5.0 release notes).
 
+- **New: `libsais` 0.2.0** (`MIT OR Apache-2.0`), wrapping the C library of the
+  same name. The C is vendored in `libsais-sys` and compiled by `cc`: no system
+  library, no `bindgen`, no `libclang`. Declared `default-features = false`,
+  which turns off the crate's default `openmp` feature, so nothing outside the
+  vendored sources is required and the C builds single-threaded. Adds
+  `bytemuck` and `either`, plus `cc` as a build dependency.
+
 ### Other
 
 - New module `index::packed_stream` — bit-for-bit-compatible streaming

Cargo.toml

@@ -65,6 +65,9 @@ mimalloc = { version = "0.1", default-features = false }
 libmimalloc-sys = { version = "0.1.49", features = ["extended"] } # mi_option_set (purge_delay); see main.rs
 libdeflater = "1.25.2"
 noodles-bgzf = { version = "0.49", features = ["libdeflate"] }
+# SA-IS suffix-array construction. `default-features = false` drops OpenMP, so
+# the C core builds with plain `cc` on every CI target including Windows MSVC.
+libsais = { version = "0.2.0", default-features = false }
 
 [dev-dependencies]
 assert_cmd = "2"

src/index/mod.rs

@@ -159,23 +159,56 @@ impl GenomeIndex {
         let sa_buf = BufWriter::with_capacity(8 * 1024 * 1024, sa_file);
         let mut sa_writer = PackedStreamWriter::new(sa_buf, word_length);
 
-        log::info!("Building suffix array...");
-        let (got_gbit, got_gmask, n_entries) = sa_build::build_streaming(
-            &genome,
-            params.temp_dir.as_deref(),
-            params.genome_sa_sparse_d as u64,
-            |packed_value| {
-                // Emit is now lightweight: just bit-pack into the SA
-                // file. caps-sa's phase-4 emit loop is single-threaded,
-                // so anything we do here serialises the whole build.
-                // The SAindex is built afterwards via a parallel pass
-                // over the on-disk SA.
+        // Which builder runs is decided by `--limitGenomeGenerateRAM`. libsais
+        // is an in-memory SA-IS and needs room for the suffix array, its own
+        // working array and the text at once; caps-sa's external-memory path
+        // needs a fraction of that and spills to disk. Both emit the same
+        // packed values, because the suffix array of a text is unique, so the
+        // choice costs nothing in output and everything in resources.
+        //
+        // The sparse builder has no libsais equivalent, so `--genomeSAsparseD`
+        // above 1 always takes the caps-sa path.
+        let sa_entries = 2 * genome.n_genome;
+        let use_libsais = params.genome_sa_sparse_d <= 1
+            && sa_build::libsais_fits_in_ram(sa_entries, params.limit_genome_generate_ram);
+
+        let (got_gbit, got_gmask, n_entries) = if use_libsais {
+            log::info!(
+                "Building suffix array with libsais (estimated peak {:.1} GB, --limitGenomeGenerateRAM {:.1} GB)...",
+                sa_build::libsais_peak_bytes(sa_entries) as f64 / 1e9,
+                params.limit_genome_generate_ram as f64 / 1e9
+            );
+            let sa = sa_build::build_libsais(&genome)?;
+            let n_entries = sa.len();
+            for i in 0..n_entries {
                 sa_writer
-                    .write_one(packed_value)
+                    .write_one(sa.get(i))
                     .map_err(|e| Error::io(e, &sa_path))?;
-                Ok(())
-            },
-        )?;
+            }
+            (gstrand_bit, gstrand_mask, n_entries)
+        } else {
+            log::info!(
+                "Building suffix array with caps-sa external memory (libsais would need an estimated {:.1} GB, --limitGenomeGenerateRAM {:.1} GB)...",
+                sa_build::libsais_peak_bytes(sa_entries) as f64 / 1e9,
+                params.limit_genome_generate_ram as f64 / 1e9
+            );
+            sa_build::build_streaming(
+                &genome,
+                params.temp_dir.as_deref(),
+                params.genome_sa_sparse_d as u64,
+                |packed_value| {
+                    // Emit is now lightweight: just bit-pack into the SA
+                    // file. caps-sa's phase-4 emit loop is single-threaded,
+                    // so anything we do here serialises the whole build.
+                    // The SAindex is built afterwards via a parallel pass
+                    // over the on-disk SA.
+                    sa_writer
+                        .write_one(packed_value)
+                        .map_err(|e| Error::io(e, &sa_path))?;
+                    Ok(())
+                },
+            )?
+        };
         debug_assert_eq!(got_gbit, gstrand_bit);
         debug_assert_eq!(got_gmask, gstrand_mask);
         log::info!("Suffix array streamed to disk: {n_entries} entries");