Skip to content

feat: add wide variant-TSV header and per-pop validation - #128

Merged
ameynert merged 2 commits into
feat/tsv-source-metadatafrom
feat/tsv-variant-validation
Sep 11, 2026
Merged

ameynert merged 2 commits into
feat/tsv-source-metadatafrom
feat/tsv-variant-validation

Conversation

@ameynert

@ameynert ameynert commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

What

Append wide variant-TSV validation to create_duckdb_from_tsv.py: a header check
(validate_variants_header) and a legend-driven, null-safe per-population validator
(read_and_validate_variants), plus a variants.tsv fixture and a parametrized refusal-table test.

Why

The pure-Python TSV path must reject malformed input up front. Validation is vectorized (polars) and
null-safe:

  • Header: the fixed contig,pos,ref,alt plus AC_<pop>/AF_<pop> per population must be present;
    an AC_/AF_ column naming an unknown population is rejected; other extra columns (e.g. rsid)
    are permitted and ignored.
  • Rows: pos >= 1; contig a GRCh38 main-contig token; ref/alt are A/C/G/T/N; each pop's
    AC_/AF_ is both-present-or-both-empty with AC >= 0 and AF in [0, 1]; at least one defined
    AF per row. Failures name the first offending (contig,pos,ref,alt); a null-derived comparison
    counts as a failure (fill_null(False)), never a silent pass.

Notes

Related to #114

@ameynert
ameynert deployed to github-actions-snakemake-linting September 4, 2026 20:00 — with GitHub Actions Active
@ameynert
ameynert force-pushed the feat/tsv-variant-validation branch from 5f8d349 to 4c981ba Compare September 4, 2026 21:22
@ameynert
ameynert deployed to github-actions-snakemake-linting September 4, 2026 21:22 — with GitHub Actions Active
@ameynert
ameynert force-pushed the feat/tsv-variant-validation branch from 4c981ba to f3c7e0a Compare September 8, 2026 21:00
@ameynert
ameynert deployed to github-actions-snakemake-linting September 8, 2026 21:00 — with GitHub Actions Active
@ameynert
ameynert marked this pull request as ready for review September 9, 2026 17:56
@ameynert
ameynert requested a review from znorgaard September 9, 2026 17:56
Comment thread divref/tests/tools/test_create_duckdb_from_tsv.py
Comment thread divref/divref/tools/create_duckdb_from_tsv.py
Comment thread divref/divref/tools/create_duckdb_from_tsv.py Outdated
Comment thread divref/divref/tools/create_duckdb_from_tsv.py
Comment thread divref/divref/tools/create_duckdb_from_tsv.py Outdated
Comment thread divref/divref/tools/create_duckdb_from_tsv.py Outdated
Comment thread divref/divref/tools/create_duckdb_from_tsv.py
@znorgaard znorgaard assigned ameynert and unassigned znorgaard Sep 11, 2026
@ameynert
ameynert force-pushed the feat/tsv-variant-validation branch from f3c7e0a to d0b84b6 Compare September 11, 2026 17:22
@ameynert
ameynert added this pull request to stack #138 September 11, 2026 17:22
@ameynert
ameynert deployed to github-actions-snakemake-linting September 11, 2026 17:22 — with GitHub Actions Active
ameynert added a commit that referenced this pull request Sep 11, 2026
Address review on #128:
- Rename the validation helper `_assert` to `_validate` (it raises ValueError,
  not AssertionError).
- Make `_FIXED_COLUMNS` a frozenset.
- Report the number of failing rows in the validation error message.
- Trim the redundant second sentence from the `mask` arg docstring.

Kept the extra-column passthrough (downstream reads columns by name, so extras
are harmless) and `tsv_source/variants.tsv` as a committed, reused fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ameynert
ameynert force-pushed the feat/tsv-variant-validation branch from d0b84b6 to 0734f26 Compare September 11, 2026 17:57
@ameynert
ameynert deployed to github-actions-snakemake-linting September 11, 2026 17:57 — with GitHub Actions Active
Address review on #128:
- Rename the validation helper `_assert` to `_validate` (it raises ValueError,
  not AssertionError).
- Make `_FIXED_COLUMNS` a frozenset.
- Report the number of failing rows in the validation error message.
- Trim the redundant second sentence from the `mask` arg docstring.

Kept the extra-column passthrough (downstream reads columns by name, so extras
are harmless) and `tsv_source/variants.tsv` as a committed, reused fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ameynert
ameynert force-pushed the feat/tsv-variant-validation branch from 0734f26 to 513fc9c Compare September 11, 2026 18:10
@ameynert
ameynert deployed to github-actions-snakemake-linting September 11, 2026 18:10 — with GitHub Actions Active
@ameynert
ameynert merged commit dbd47ed into main Sep 11, 2026
3 checks passed
@ameynert
ameynert deleted the feat/tsv-variant-validation branch September 11, 2026 20:01

This branch was successfully deployed

1 active deployment
github-actions-snakemake-linting — 513fc9c3 Deployed Sep 11, 2026 by ameynert via Tests #438
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants