feat(backfill): discover dataset tables directly, not just the runs journal

[divyasinghds]

Problem

The metadata-backfill runner (#393) enumerated the runs journal (Database.list_registered_runs), which only has entries for datasets ingested by the post-journal ingestor. But the whole point of the backfill is the pre-cutover backlog — datasets ingested before the run journal existed, which therefore have no journal entry.

Verified on the dev cluster: tracebloc_ingest_runs was empty, yet the DB had ~170 real dataset tables (cats_dogs_train, cancer_classification_*, claims_*, …). The runner ran end-to-end and authenticated to the backend, but swept 0 datasets — exactly the tables it was meant to fix were invisible to it.

Fix — discover dataset tables directly

• Database.list_dataset_ingestor_ids() — scans information_schema.columns for every table carrying the framework ingestor_id column (excluding the run journal + salt store), then reads each table's distinct, non-null ingestor_ids. This covers journalled and pre-journal datasets — the journal is now a strict subset.
• backfill_datasets() defaults to this discovery (explicit ingestor_ids still supported). Because a table's metadata is table-level, a table appended by several ingest runs (multiple ingestor_ids) is backfilled once — the first id the backend resolves wins; siblings are skipped. A not_found/error id does not mark the table done, so another id can still resolve the same table.

Per-dataset guarding (one bad table can't abort the sweep) and the exit-code/log hygiene from #393 are unchanged.

Why table-scan is safe

• Only tables with the framework ingestor_id column are considered; the run journal (which also has the column) and the salt store are excluded by name.
• Null/empty ingestor_ids are filtered out.
• The GET-by-ingestor_id is still owner-scoped on the backend, so a table whose id the backend doesn't recognise is skipped (not_found), not mis-backfilled.

Tests

test_metadata_backfill_runner.py: discovery is the default (journal not called); a multi-id table is backfilled once; a sibling id is retried after a not_found. test_database.py: list_dataset_ingestor_ids scans information_schema, excludes framework tables, filters null/empty ids. Full runner+database+backfill suites green (82 passed).

Rollout

Pairs with the client chart backfill hook (#380). No new endpoint or migration — a client-side enumeration change. After release, the hook's sweep will actually cover a client's pre-cutover tables.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes rollout enumeration and issues per-table SELECT DISTINCT scans across all dataset tables (~170+ on dev), which can be slow or load MySQL during install hooks; backfill behavior is otherwise scoped and owner-checked via existing GET-by-ingestor_id.

Overview
Fixes metadata backfill sweeps that processed zero datasets when the runs journal was empty but hundreds of legacy dataset tables still existed in MySQL.

Discovery: Adds Database.list_dataset_ingestor_ids() — finds tables via information_schema that have an ingestor_id column, skips framework tables (tracebloc_ingest_runs, salt store), and collects distinct non-null ids per table. Unreadable tables are logged and skipped so one failure does not stop the sweep.

Runner: backfill_datasets() with no explicit ingestor_ids now uses that discovery instead of list_registered_runs(). Tables with multiple ingestor ids are backfilled once (sibling ids skipped after a definitive outcome); not_found / error do not mark a table done so another id on the same table can still be tried.

Bumps package version to 0.7.7; tests cover discovery, dedup, and sibling retry.

^{Reviewed by Cursor Bugbot for commit 1fe3b9c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 16e3d1e. Configure here.}

[LukasWodka]

Functional review \u2014 passed\n\nBasis: the behavioural suites that ran on this PR at merge against real environments, not mocks:\n\n- \u2014 exercised against a real MySQL 8.0 service container, not a mock\n\nI could not reach the dev API (no dev credentials), so rather than rubber-stamp I used the strongest evidence available: these suites exercise the actual behaviour this change alters, on real infrastructure. Advancing \u2192 .\n\nIf the functional reviewer wants a manual pass on dev in addition, please move it back and say so.