Sync develop → master for v0.7.8 release - #400
Open
saadqbal wants to merge 7 commits into
Open
Conversation
…1) (#392) validator.name already ends in "Validator" (e.g. "Data Validator"), so appending the word again produced "Data Validator Validator successfully passed" (and the same doubling in the failed/error paths). Drop the redundant literal in all three spots so the lines read "Data Validator successfully passed" / "Data Validator failed: …" / "Data Validator error: …". Log-string only, no logic change; updated the one docstring that quoted the old doubled message. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…send to backend (#378 send side) (#393) * feat(backfill): metadata backfill runner — GET dataset, recompute, send to backend Wires data-ingestors #378 (build_dataset_metadata) to the backend metadata endpoints (#1166 POST, #1198 GET) into a one-time, per-client pre-cutover backfill sweep. For each registered dataset it: 1. GETs the backend record by ingestor_id (authoritative category + data_format, not recoverable from the table alone); 2. recomputes {schema, meta_data} from persisted rows via build_dataset_metadata, keyed on that category; 3. POSTs it back — the endpoint upserts GlobalMetaData and re-folds any competition built from the table. - APIClient.get_dataset_metadata / send_metadata_backfill — the two API calls, with local-mode mocks, 404→None (skip), and error surfacing. - Database.list_registered_runs — enumerate the client's REGISTERED ingest journal (the datasets to sweep). - metadata_backfill_runner — backfill_dataset (GET→build→POST, skips competitions/merged and already-new-shape datasets) + backfill_datasets (per-dataset guarded sweep) + a `tracebloc-backfill` console-script entrypoint reading DB/backend creds from env (no ingest.yaml). Idempotent and re-runnable; new ingests already emit new-shape metadata, so this is a rollout step, not always-on. Tests: runner orchestration/skip/guard paths, the two API methods (success/404/error/local/payload shape), and the DB enumeration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: retrigger checks after retargeting PR base to develop The FR gate (staging/prod-promotion guard) ran against the original master base and its stale failure lingered on the head commit after retargeting to develop, where the gate does not apply. Empty commit to reset the head SHA. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(backfill): don't leak raw exception text / response bodies in the sweep (Bugbot #393) Per the org rule "error messages and logs must never embed customer cell values": - Runner: on a per-dataset failure, record and log the exception TYPE only (`type(exc).__name__`) instead of `str(exc)[:500]`, and use logger.error rather than logger.exception (no traceback). This runs in a Helm-hook Job whose output lands in install logs. - APIClient.get_dataset_metadata / send_metadata_backfill: raise HTTPError("HTTP <status>") without response.text. The backfill POST body carries this table's categorical vocab (customer cell values), so a backend error echoing it back must not reach the exception string / logs. The response object is still attached for deliberate programmatic inspection. Test asserts the recorded error is the exception type ("RuntimeError"), proving the raw message ("boom") is not surfaced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(release): bump version to 0.7.6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-validator): auto-detect from first readable image, not files[0] test_too_small_result_still_reports_corrupt_files was order-dependent: auto-detection read only image_files[0], so when a corrupt file came first in glob order (deterministic on the Linux CI, not on macOS) it returned None, expected_resolution stayed unset, and _validate_image_resolutions bailed with "auto-detection failed" before the min-size floor check could run — masking the real verdict and making the outcome depend on filesystem ordering. Scan for the first *readable* image instead; if none are readable, expected_resolution stays unset and the honest "auto-detection failed" still surfaces. Adds an order-independence guard test that forces the corrupt file first (fails pre-fix, passes post-fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: shujaat <shujaat@tracebloc.io>
…ournal (#395) * feat(backfill): discover dataset tables directly, not just the runs journal The runner enumerated the runs journal (list_registered_runs), which only has entries for datasets ingested by the post-journal ingestor. The pre-cutover backlog the backfill exists for was ingested BEFORE the journal, so it had no entry and was silently swept 0 (verified on dev: empty journal, ~170 real dataset tables untouched). Enumerate dataset tables directly instead: - Database.list_dataset_ingestor_ids() — scan information_schema for tables carrying the framework `ingestor_id` column (excluding the run journal + salt store), then read each table's distinct non-null ingestor_ids. Covers journalled AND pre-journal datasets (the journal is a strict subset). - backfill_datasets() defaults to this discovery. A table appended by several ingest runs has multiple ingestor_ids; metadata is table-level, so a table is backfilled once — the first id the backend resolves wins, siblings are skipped (a not_found/error id doesn't mark the table done, so another id can resolve it). Explicit ingestor_ids still supported. Tests: table discovery is the default; framework tables excluded; multi-id table backfilled once; sibling id retried after a not_found. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(release): bump version to 0.7.7 Bundled with the table-discovery change so a single merge + v0.7.7 tag yields a deployable image carrying it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(backfill): guard per-table discovery so one bad table can't abort the sweep (Bugbot #395) list_dataset_ingestor_ids scanned every dataset table with no per-table guard, so a single failing SELECT DISTINCT (scan timeout, table dropped mid-sweep, perms) aborted enumeration and skipped ALL tables. Wrap the per-table query in try/except: log the table name + exception TYPE only (never str(exc) — a driver message can embed cell values and feeds install logs) and continue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Append "Validator" only when validator.name doesn't already end in it, so names like "Ingestable Records" keep the suffix and names like "Data Validator" don't double it. - tracebloc_ingestor/ingestors/base.py Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Ships the validator log-label normalization fix (#397) — the only code on develop/master ahead of the v0.7.7 release image. Tagging v0.7.8 after this merges triggers release-image.yml -> ghcr.io/tracebloc/ingestor:0.7.8. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ter-v0.7.8 # Conflicts: # tracebloc_ingestor/__init__.py
shujaatTracebloc
previously approved these changes
Jul 27, 2026
Collaborator
|
Heads-up: the fr-gate now enforces that promotions into master go through the release train (announced today) — this manual sync PR will fail the gate on its next run by design. It also looks superseded: v0.7.8 shipped 07-27 and data-ingestors rides the train since yesterday (its staging→master promotion is train-managed). Recommend closing this one; if there's content here the train hasn't carried, it'll go develop → staging hop → prod hop like everything else. |
tracebloc-release-train
Bot
dismissed
shujaatTracebloc’s stale review
July 29, 2026 15:36
The merge-base changed after approval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
developtomasterfor the v0.7.8 release. Closes #399.The develop-side release (bump #398, tag
v0.7.8, signed imageghcr.io/tracebloc/ingestor:0.7.8, GitHub Release) is already done. This PR ships the matching PyPI package so the pip-reported version stops lagging the image.Diff is a single line —
__version__0.7.7→0.7.8intracebloc_ingestor/__init__.py. All code (#392–#397) already reachedmastervia the #396 sync.On merge,
publish-master.ymlpublishes 0.7.8 to PyPI / GitHub Packages (schema-load smoke probe gates the upload). Per RELEASING.md, merge as a merge commit, not squash.Note
Low Risk
Single-line version bump with no logic, API, or dependency changes; publish is gated by the existing schema-load smoke probe.
Overview
Release alignment for v0.7.8 — the only code change is
__version__0.7.7→0.7.8intracebloc_ingestor/__init__.py, which remains the single source of truth (setup.pyreads it at build time).This develop → master sync does not introduce new features or fixes; it promotes the version bump so a merge to
mastertriggerspublish-master.ymland publishes 0.7.8 to PyPI/GitHub Packages, aligning pip with the existing container release. Merge as a merge commit (not squash), per release process.Reviewed by Cursor Bugbot for commit d37bb14. Bugbot is set up for automated code reviews on this repo. Configure here.