Skip to content

filter-and-trim: no primer-sequence-aware trimming; fixed --trim-left cannot handle heterogeneity spacers #113

Description

@cjfields

Problem

filter-and-trim removes leading bases by fixed offset only
(src/filter_trim.rs:192, let start = p.trim_left;). There is no
sequence-aware primer matching anywhere in the module.

That is fine when the primer begins at a constant position. It breaks on the
now-common library designs that insert heterogeneity spacers — 0–N nt of
variable padding before the forward primer, used to boost base diversity on
patterned flow cells (Fadrosh-style).

With variable offsets there is no correct --trim-left:

  • too small → spacer bases remain on some reads
  • too large → real biological bases are removed from others

Both produce the same biological sequence as several distinct ASVs.

Measured impact

On a public NovaSeq 6000 16S run (BioProject PRJNA1504839), 341F was found at
5 distinct offsets (0–4 nt) in 99.96% of merged reads, with the forward
primer retained. Stripping spacer + primer post-hoc collapsed:

arm post-chimera ASVs after stripping collapse
loess 26,097 15,849 39.3%
binned-qual 35,314 19,019 46.1%

So roughly 4 in 10 reported ASVs were offset duplicates of another ASV. It also
inflated an errfun A/B: a −26.1% ASV difference was −16.7% once corrected.

Current workaround

Use cutadapt (or R's removePrimers()) upstream for primer removal, then
filter-and-trim for quality filtering only. This is what our R workflow does.
Worth documenting explicitly either way, since --trim-left looks like it is
the primer-removal knob and silently is not.

Possible directions

  • Add primer-sequence-aware trimming: --primer-fwd / --primer-rev accepting
    IUPAC-degenerate sequences, matched within a small search window with a
    mismatch allowance, trimming through the match end. This is what
    removePrimers() does and what the data above requires.
  • Optionally --max-primer-mismatch and an orientation check.
  • At minimum: document that --trim-left is a positional trim and is not
    safe for spacer-containing or variable-offset designs, and point at cutadapt.

Notes

  • Degenerate primers (341F is CCTACGGGNGGCWGCAG) mean a plain literal match is
    insufficient; IUPAC expansion is required.
  • Detection is cheap as a diagnostic even before trimming is implemented: search
    a known primer across the first ~30 nt and report the offset distribution. A
    multi-offset result is the signal that positional trimming is unsafe. This
    would pair naturally with the read-overlap QC screen discussed in the binned
    quality findings.

Context: docs/findings/binned-quality-illumina-novaseq.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    low priorityLower priority tickets; good to check but not necessary

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions