Release develop → master#396
Merged
Merged
Conversation
…1) (#392) validator.name already ends in "Validator" (e.g. "Data Validator"), so appending the word again produced "Data Validator Validator successfully passed" (and the same doubling in the failed/error paths). Drop the redundant literal in all three spots so the lines read "Data Validator successfully passed" / "Data Validator failed: …" / "Data Validator error: …". Log-string only, no logic change; updated the one docstring that quoted the old doubled message. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…send to backend (#378 send side) (#393) * feat(backfill): metadata backfill runner — GET dataset, recompute, send to backend Wires data-ingestors #378 (build_dataset_metadata) to the backend metadata endpoints (#1166 POST, #1198 GET) into a one-time, per-client pre-cutover backfill sweep. For each registered dataset it: 1. GETs the backend record by ingestor_id (authoritative category + data_format, not recoverable from the table alone); 2. recomputes {schema, meta_data} from persisted rows via build_dataset_metadata, keyed on that category; 3. POSTs it back — the endpoint upserts GlobalMetaData and re-folds any competition built from the table. - APIClient.get_dataset_metadata / send_metadata_backfill — the two API calls, with local-mode mocks, 404→None (skip), and error surfacing. - Database.list_registered_runs — enumerate the client's REGISTERED ingest journal (the datasets to sweep). - metadata_backfill_runner — backfill_dataset (GET→build→POST, skips competitions/merged and already-new-shape datasets) + backfill_datasets (per-dataset guarded sweep) + a `tracebloc-backfill` console-script entrypoint reading DB/backend creds from env (no ingest.yaml). Idempotent and re-runnable; new ingests already emit new-shape metadata, so this is a rollout step, not always-on. Tests: runner orchestration/skip/guard paths, the two API methods (success/404/error/local/payload shape), and the DB enumeration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: retrigger checks after retargeting PR base to develop The FR gate (staging/prod-promotion guard) ran against the original master base and its stale failure lingered on the head commit after retargeting to develop, where the gate does not apply. Empty commit to reset the head SHA. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(backfill): don't leak raw exception text / response bodies in the sweep (Bugbot #393) Per the org rule "error messages and logs must never embed customer cell values": - Runner: on a per-dataset failure, record and log the exception TYPE only (`type(exc).__name__`) instead of `str(exc)[:500]`, and use logger.error rather than logger.exception (no traceback). This runs in a Helm-hook Job whose output lands in install logs. - APIClient.get_dataset_metadata / send_metadata_backfill: raise HTTPError("HTTP <status>") without response.text. The backfill POST body carries this table's categorical vocab (customer cell values), so a backend error echoing it back must not reach the exception string / logs. The response object is still attached for deliberate programmatic inspection. Test asserts the recorded error is the exception type ("RuntimeError"), proving the raw message ("boom") is not surfaced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(release): bump version to 0.7.6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-validator): auto-detect from first readable image, not files[0] test_too_small_result_still_reports_corrupt_files was order-dependent: auto-detection read only image_files[0], so when a corrupt file came first in glob order (deterministic on the Linux CI, not on macOS) it returned None, expected_resolution stayed unset, and _validate_image_resolutions bailed with "auto-detection failed" before the min-size floor check could run — masking the real verdict and making the outcome depend on filesystem ordering. Scan for the first *readable* image instead; if none are readable, expected_resolution stays unset and the honest "auto-detection failed" still surfaces. Adds an order-independence guard test that forces the corrupt file first (fails pre-fix, passes post-fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: shujaat <shujaat@tracebloc.io>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9b0f439. Configure here.
…ournal (#395) * feat(backfill): discover dataset tables directly, not just the runs journal The runner enumerated the runs journal (list_registered_runs), which only has entries for datasets ingested by the post-journal ingestor. The pre-cutover backlog the backfill exists for was ingested BEFORE the journal, so it had no entry and was silently swept 0 (verified on dev: empty journal, ~170 real dataset tables untouched). Enumerate dataset tables directly instead: - Database.list_dataset_ingestor_ids() — scan information_schema for tables carrying the framework `ingestor_id` column (excluding the run journal + salt store), then read each table's distinct non-null ingestor_ids. Covers journalled AND pre-journal datasets (the journal is a strict subset). - backfill_datasets() defaults to this discovery. A table appended by several ingest runs has multiple ingestor_ids; metadata is table-level, so a table is backfilled once — the first id the backend resolves wins, siblings are skipped (a not_found/error id doesn't mark the table done, so another id can resolve it). Explicit ingestor_ids still supported. Tests: table discovery is the default; framework tables excluded; multi-id table backfilled once; sibling id retried after a not_found. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(release): bump version to 0.7.7 Bundled with the table-discovery change so a single merge + v0.7.7 tag yields a deployable image carrying it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(backfill): guard per-table discovery so one bad table can't abort the sweep (Bugbot #395) list_dataset_ingestor_ids scanned every dataset table with no per-table guard, so a single failing SELECT DISTINCT (scan timeout, table dropped mid-sweep, perms) aborted enumeration and skipped ALL tables. Wrap the per-table query in try/except: log the table name + exception TYPE only (never str(exc) — a driver message can embed cell values and feeds install logs) and continue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc
added a commit
that referenced
this pull request
Jul 24, 2026
Append "Validator" only when validator.name doesn't already end in it, so names like "Ingestable Records" keep the suffix and names like "Data Validator" don't double it. - tracebloc_ingestor/ingestors/base.py Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Append "Validator" only when validator.name doesn't already end in it, so names like "Ingestable Records" keep the suffix and names like "Data Validator" don't double it. - tracebloc_ingestor/ingestors/base.py Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
divyasinghds
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release: promote
develop→masterThis repo has no
stagingbranch, sodeveloppromotes straight to prod (master).Commits being promoted (3):
Release bumps the ingestor version to 0.7.6.
Note
Medium Risk
The backfill sweep issues authenticated writes to production metadata endpoints and scans all client dataset tables; failures are isolated per dataset but a mis-run could touch many tables—mitigated by idempotency, skips, and conservative logging (no response bodies in errors).
Overview
Adds a one-time pre-cutover metadata backfill path: new
tracebloc-backfillconsole script andmetadata_backfill_runnerthat discovers dataset tables (including pre–run-journal data vialist_dataset_ingestor_ids), GETs backend records byingestor_id, recomputes metadata withbuild_dataset_metadata, and POSTs to the metadata-backfill API—with per-table dedup, skip rules (competitions, already new-shape), and fault-tolerant sweeps.APIClientgainsget_dataset_metadataandsend_metadata_backfill;Databasegainslist_registered_runsand table discovery helpers.Validator / UX fixes: ingest preflight normalizes validator log labels via
_validator_label(no doubled or dropped “Validator”); image resolution auto-detection uses the first readable file so corrupt-first ordering no longer hides the min-size failure.Package version 0.7.7; broad test coverage for the new APIs, runner orchestration, DB enumeration, and the above fixes (plus minor test formatting).
Reviewed by Cursor Bugbot for commit 0a6bed2. Bugbot is set up for automated code reviews on this repo. Configure here.