Skip to content

data ingest: preflight parity gaps (semseg masks, DataValidator types) + stale pin — accept-then-reject #352

Description

@LukasWodka

Part of epic tracebloc/backend#1142 (Theme D).

MED. The CLI data ingest preflight (internal/push/preflight.go) under-mirrors the real in-cluster data-ingestors validators, so some datasets accept locally then reject after upload:

  • Semseg mask resolution not mirrored (preflight.go:1532) — data-ingestors adds a second ImageResolutionValidator (name "Mask Resolution Validator", subdir "masks", validators.py:168-173) the CLI never runs → corrupt/mis-sized masks pass preflight.
  • DataValidator per-value type checks not mirrored (preflight.go:1444) — for tabular/time-series/text-with-schema the ingestor runs DataValidator(schema) per value (validators.py:213…466); the CLI checks schema shape but not per-value types → schema/value mismatches pass preflight, reject in-cluster after table creation.
  • Stale pin (scripts/.data-ingestors-ref): pinned v0.7.0 while HEAD is v0.7.4; the comment falsely claims "develop HEAD"; goldens never regenerated. (The HEAD-drift canary fix(ci): PAT-reachability fallback + pin-ref shape guard for the data-ingestors jobs #326 is meant to catch this — verify it fires.)

Fix: mirror the two validators into preflight + extend the parity corpus; bump the pin + regenerate goldens.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions