Skip to content

Tabular roll-up of per-sample run metrics (unique fraction, reads through pipeline) #104

Description

@cjfields

Motivation

Sizing and triaging a large run currently means reaching into per-sample JSON
artifacts one at a time. The numbers we want are already recorded — e.g. derep
JSON carries total_reads and unique_sequences as top-level header fields — but
there is no way to get them as a table across a run.

Concretely, the unique fraction per sample drives:

  • Memory sizing. Whether dada-pooled is feasible or the run must use
    dada-pseudo depends on the resident unique count, not the read count. On a
    recent high-diversity dataset the unique fraction ranged ~39–55%, which is the
    difference between a comfortable per-sample footprint and an infeasible pooled
    one. That check currently requires manual derep runs.
  • Diversity triage. Unique/read ratio and singleton fraction are cheap proxies
    for sample diversity, which is relevant to error-model quality (see the
    error-model fit diagnostics work).

Proposal

A subcommand that reads per-sample JSON artifacts across a directory and emits a
tabular summary (TSV, plus JSON). Analogous to the existing summary-merge, which
already does this for summary outputs.

Sketch — deliberately underspecified, needs design:

dada2-rs <name> --input derep/ --output run_table.tsv

Open design questions

  1. Scope of inputs. derep only, or a general roll-up over several artifact
    kinds (derep, dada, filter-and-trim, remove-primers)? A single table
    joined per sample across stages would be the most useful version and the most
    work — it is essentially a run-level "reads through the pipeline" table, which
    R DADA2 users build by hand today.
  2. Relationship to summary-merge. Extend it to accept other artifact types,
    or add a sibling subcommand? Overlapping but not identical: summary-merge
    consumes FASTQ quality metrics, this consumes post-processing counts.
  3. Derived columns. Emit raw counts only, or also computed ratios
    (unique fraction, singleton fraction, retention % per stage)? Ratios are the
    thing people actually read, but they bake in choices better left to downstream
    analysis.
  4. Cost. Header fields sit before the large arrays in these JSONs, so a
    bounded read per file should suffice — worth confirming the layout guarantee
    holds for every artifact type rather than relying on field ordering.

Notes

Not urgent; filing to capture the idea. Wants a design pass before implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions