sramic

Warning

Under active development.

sramic (pronounced “ceramic”) is a metadata-first Nextflow pipeline for finding public SRA/ENA sequencing runs, evaluating their contextual metadata, and optionally retrieving only the runs that pass your selection rules.

It is designed for questions that depend on where, when, or from what host a sample was collected. Instead of downloading everything first, sramic scores metadata, records why each run passed or failed, estimates transfer size, and then downloads the selected subset.

Retrieval can use Kingfisher for multi-source download and extraction, or sracha-rs for fast, resumable NCBI download, Rust-native decode, and gzip-compressed FASTQ output.

At a glance

manifest or ENA taxon query
            │
            ▼
 normalize and validate metadata
            │
            ▼
 score and select accessions ────── rejected records + reasons
            │
            ▼
 Kingfisher technical annotation + size estimate
            │
            ├── Kingfisher → FASTQ/FASTA/SRA using ordered source fallbacks
            └── sracha-rs  → resumable NCBI transfer → compressed FASTQ
            │
            ▼
 linked Parquet metadata, artifacts, and ML-ready tables

Key properties:

• Metadata is selected before genomic data is transferred.
• Every selected artifact remains linked to its run and contextual metadata.
• Submitted values are preserved; normalized values and conflicts are separate.
• One --threads value controls the local CPU pool and downloader concurrency.
• Downloads can use either Kingfisher or sracha-rs.

Installation

Requirements are Nextflow 26.04.6+, Java 11+, and Conda/Mamba, Docker, or Apptainer for managed dependencies.

For development:

git clone https://github.com/microbemarsh/sramic.git
cd sramic
conda env create -f environment.yml
conda activate sramic-dev
sramic --help

The root environment includes the runtime, tests, Kingfisher, and sracha-rs, and installs sramic in editable mode. The smaller environments/environment.yml is used by Nextflow's Conda profile.

An existing Python environment can install the launcher with:

python -m pip install .

Quick start

Run the offline test profile without network access:

sramic -profile test

Select runs by NCBI Taxonomy ID and download compressed FASTQ with Kingfisher:

sramic \
  --taxid 86661 \
  --min_metadata_score 70 \
  --download_sequences true \
  --download_backend kingfisher \
  --threads 16 \
  --max_accessions 100 \
  --outdir results \
  -profile conda

Select from a manifest and produce compressed FASTQ with sracha-rs, using SRA-lite as the smaller temporary source representation:

sramic \
  --manifest accessions.csv \
  --download_sequences true \
  --download_backend sracha \
  --sracha_format sralite \
  --threads 16 \
  --outdir results \
  -profile conda

Choosing a downloader

Kingfisher

Kingfisher is the default. It can try ENA, cloud, and SRA Toolkit sources in order and can produce SRA, FASTQ, compressed FASTQ, FASTA, or compressed FASTA.

--download_backend kingfisher \
--kingfisher_methods ena-ftp,aws-http,prefetch \
--kingfisher_output_format fastq.gz

Use Kingfisher when you want extracted reads ready for downstream tools or want fallbacks across several public sources.

sracha-rs

sracha-rs is a fast SRA downloader and converter written in Rust. It supports resumable downloads, integrity checks, parallel transfers, multithreaded decoding, and parallel gzip compression. See the sracha-rs documentation for its full standalone feature set and CLI.

sramic uses sracha get, which downloads, decodes, and emits .fastq.gz in one operation:

--download_backend sracha \
--sracha_format sra       # or sralite

--sracha_format selects the temporary source representation, not the final file type. sra preserves submitted quality scores; sralite downloads less data but uses simplified quality scores. Both produce compressed FASTQ and the temporary archive is removed by sracha-rs after successful decoding.

Need	Recommended backend
Ready-to-use compressed FASTQ	Kingfisher or sracha-rs
FASTA or uncompressed FASTQ	Kingfisher
Ordered ENA/cloud/SRA Toolkit fallbacks	Kingfisher
Resumable NCBI transfer with integrated Rust decode	sracha-rs
Smaller transfer with simplified quality scores	sracha-rs with sralite

Regardless of download backend, sramic currently uses Kingfisher annotation for technical metadata and the pre-download size estimate.

Threads and resources

Supply the resource budget once:

--threads 16

For local execution this caps Nextflow's shared CPU pool. Light tasks use one CPU, medium tasks use up to four, and sequence acquisition can use the full budget for Kingfisher transfer/extraction or sracha-rs download, decode, and compression. Advanced users can override only the transfer component with --kingfisher_download_threads or --sracha_connections.

Slurm profiles translate process CPU requests into scheduler resources. Site account, QoS, partition, and email settings remain configurable through the --slurm_* parameters.

Input and selection

Exactly one primary input mode is required:

• --taxid: positive NCBI Taxonomy ID, including descendants through ENA.
• --taxon_name: taxon-name query.
• --manifest: CSV, TSV, compressed delimited text, or Parquet.

--taxon_id remains a compatibility alias for --taxid. A manifest must contain an archive accession column, and downloads require run accessions.

Common selection controls include:

Purpose	Parameters
Overall/context scores	--min_metadata_score, --min_geography_score, --min_date_score, --min_host_score
Required metadata	--require_country, --require_coordinates, --require_collection_date, --require_host, --require_isolation_source
Sequencing filters	--library_strategy, --library_source, --library_selection, --platform, --instrument_model
Context filters	--country, collection-year bounds, --host_tax_id
Run limit	--max_accessions

Host metadata is not required by default because environmental samples often have no meaningful host.

Metadata scoring

The exported score totals 100:

Component	Points	Examples
Geography	35	country, locality, coordinates, coordinate/country agreement
Collection date	20	valid year/month/day and plausibility
Host	20	host identity, taxonomy, tissue, health, demographics
Sample context	10	isolation source and environment
Sequencing	10	library, instrument, counts, file availability
Consistency	5	accession and taxonomy/study agreement

Likely corrections, such as coordinate swaps, are suggestions in separate fields. Source disagreements and rejection reasons are retained rather than silently resolved.

Size-only preflight

Estimate selected-run transfer size without downloading sequence data:

sramic \
  --manifest accessions.csv \
  --estimate_download_size true \
  --outdir results \
  -profile conda

This produces per-run estimates and a summary containing total bytes/GiB, unknown-size runs, and metadata coverage. Retrieval performs the same preflight automatically. Extracted FASTQ/FASTA can require more disk space than the archive estimate.

Outputs

results/
├── metadata/       raw, normalized, scored, selected, and technical metadata
├── raw_reads/      optional accession-keyed downloads and retrieval report
├── final/          curated/rejected metadata, artifact manifest, and GeoJSON
├── training/       linked ML-oriented Parquet tables
├── report/         summary HTML
└── pipeline_info/  execution reports, versions, and parameters

The primary interfaces are:

• final/sramic_curated_metadata.parquet: selected contextual metadata.
• final/sramic_all_metadata.parquet: every scored row and its outcome.
• final/sramic_manifest.parquet: one linked row per local or remote artifact.
• raw_reads/raw_read_report.csv: per-run retrieval status and failures.
• training/dataset_manifest.json: table versions, row counts, and join keys.

All artifact and training tables retain stable identifiers. Set --output_tsv true for human-readable TSV mirrors; Parquet remains the primary machine-learning format.

The training directory separates observations, hosts, technical covariates, artifacts, sequence units, genome clusters, detections, embedding inputs, and leakage-aware split groups. Missing detections remain unknown and are never converted into biological absences.

Scaling and resuming

Use bounded taxonomic or accession shards for corpus-scale work rather than one all-SRA invocation. Keep shared_identifier in every shard and use partitioned Parquet datasets for aggregation.

Resume interrupted work with Nextflow:

sramic -resume [parameters...]

Keep work/ and .nextflow.log. Successful tasks and cached metadata requests will be reused. Downloader failures are recorded per run so a failed transfer does not erase completed pipeline work.

Troubleshooting and limitations

• Empty selection: inspect scored_metadata_with_pass.csv and failure reasons, then relax one threshold at a time.
• API interruption: rerun with -resume to reuse successful cached requests.
• Scheduler rejection: check the Slurm account, QoS, and partition values.
• Coordinate validation defaults to syntax/range checks; country-boundary matching requires a compatible local boundary dataset.
• sramic selects archive runs from metadata. It does not search by nucleotide sequence or retrieve preassembled SRA contigs.
• Live archive schemas can change; record tool versions and parameters for each release or dataset build.

Testing

pytest -q

The test profile provides an offline Nextflow integration fixture.

Citation and license

Citation metadata are in CITATION.cff. sramic is released under the MIT License; see LICENSE.