Motivation
Sizing and triaging a large run currently means reaching into per-sample JSON
artifacts one at a time. The numbers we want are already recorded — e.g. derep
JSON carries total_reads and unique_sequences as top-level header fields — but
there is no way to get them as a table across a run.
Concretely, the unique fraction per sample drives:
- Memory sizing. Whether
dada-pooled is feasible or the run must use
dada-pseudo depends on the resident unique count, not the read count. On a
recent high-diversity dataset the unique fraction ranged ~39–55%, which is the
difference between a comfortable per-sample footprint and an infeasible pooled
one. That check currently requires manual derep runs.
- Diversity triage. Unique/read ratio and singleton fraction are cheap proxies
for sample diversity, which is relevant to error-model quality (see the
error-model fit diagnostics work).
Proposal
A subcommand that reads per-sample JSON artifacts across a directory and emits a
tabular summary (TSV, plus JSON). Analogous to the existing summary-merge, which
already does this for summary outputs.
Sketch — deliberately underspecified, needs design:
dada2-rs <name> --input derep/ --output run_table.tsv
Open design questions
- Scope of inputs.
derep only, or a general roll-up over several artifact
kinds (derep, dada, filter-and-trim, remove-primers)? A single table
joined per sample across stages would be the most useful version and the most
work — it is essentially a run-level "reads through the pipeline" table, which
R DADA2 users build by hand today.
- Relationship to
summary-merge. Extend it to accept other artifact types,
or add a sibling subcommand? Overlapping but not identical: summary-merge
consumes FASTQ quality metrics, this consumes post-processing counts.
- Derived columns. Emit raw counts only, or also computed ratios
(unique fraction, singleton fraction, retention % per stage)? Ratios are the
thing people actually read, but they bake in choices better left to downstream
analysis.
- Cost. Header fields sit before the large arrays in these JSONs, so a
bounded read per file should suffice — worth confirming the layout guarantee
holds for every artifact type rather than relying on field ordering.
Notes
Not urgent; filing to capture the idea. Wants a design pass before implementation.
Motivation
Sizing and triaging a large run currently means reaching into per-sample JSON
artifacts one at a time. The numbers we want are already recorded — e.g.
derepJSON carries
total_readsandunique_sequencesas top-level header fields — butthere is no way to get them as a table across a run.
Concretely, the unique fraction per sample drives:
dada-pooledis feasible or the run must usedada-pseudodepends on the resident unique count, not the read count. On arecent high-diversity dataset the unique fraction ranged ~39–55%, which is the
difference between a comfortable per-sample footprint and an infeasible pooled
one. That check currently requires manual
derepruns.for sample diversity, which is relevant to error-model quality (see the
error-model fit diagnostics work).
Proposal
A subcommand that reads per-sample JSON artifacts across a directory and emits a
tabular summary (TSV, plus JSON). Analogous to the existing
summary-merge, whichalready does this for
summaryoutputs.Sketch — deliberately underspecified, needs design:
Open design questions
dereponly, or a general roll-up over several artifactkinds (
derep,dada,filter-and-trim,remove-primers)? A single tablejoined per sample across stages would be the most useful version and the most
work — it is essentially a run-level "reads through the pipeline" table, which
R DADA2 users build by hand today.
summary-merge. Extend it to accept other artifact types,or add a sibling subcommand? Overlapping but not identical:
summary-mergeconsumes FASTQ quality metrics, this consumes post-processing counts.
(unique fraction, singleton fraction, retention % per stage)? Ratios are the
thing people actually read, but they bake in choices better left to downstream
analysis.
bounded read per file should suffice — worth confirming the layout guarantee
holds for every artifact type rather than relying on field ordering.
Notes
Not urgent; filing to capture the idea. Wants a design pass before implementation.