feat(chart): one-time metadata backfill as a post-upgrade Helm hook

[divyasinghds]

Draft. The one-time, on-update trigger for the pre-cutover metadata backfill. Pairs with data-ingestors #393 (tracebloc-backfill entrypoint) + backend #1166/#1198 (endpoints).

What & why

Datasets ingested before the federated-alignment cutover carry no enriched schema / feature_stats on the global-metadata channel, so the backend's combine-time alignment checks and the edge metadata API can't reconcile them — old competitions/experiments that depend on that metadata degrade under the new backend code. data-ingestors #393 adds a tracebloc-backfill entrypoint that fixes each old table (GET record → recompute from persisted MySQL rows → POST back, which also re-folds competitions). This PR is the trigger: it runs that entrypoint once, automatically, when a cluster auto-updates to this release.

How — a post-upgrade Helm hook

client/templates/metadata-backfill-hook.yaml — a post-upgrade,post-install hook Job. The chart's auto-upgrade CronJob already runs helm upgrade when a new chart publishes; that fires this hook. So a client that auto-updates to this release runs the backfill as part of coming up — no operator step, no always-on cost.

Chosen over a jobs-manager-startup spawn (client-runtime) because the Helm hook is the idiomatic one-time-on-upgrade mechanism, it's declarative (no runtime code coupled to boot), and it fires exactly on the auto-update event the requirement describes. (The existing ingestor/templates/post-install-job.yaml is the same pattern.)

Wiring

• Auth: reuses the release's CLIENT_ID/CLIENT_PASSWORD Secret (the same creds jobs-manager uses); the ingestor's APIClient mints a backend token from them. (The chart holds no static backend token — jobs-manager mints its own at runtime — so the hook uses the client creds. Noted as a dependency on the ingestor's cred-auth path.)
• Store: MYSQL_HOST=mysql-client + CLIENT_ENV + tracebloc.proxyEnv, mirroring the ingestion Job so the ingestor's Config resolves the same MySQL + backend endpoint + corporate-proxy egress.
• Image: resolves the same images.ingestor repo/tag/digest the ingestion path spawns (floating tag + Always, or pinned digest + IfNotPresent).

Safety

• Best-effort: retries a few times in-pod, then exits 0 regardless, so a backfill problem can never fail the helm upgrade (which would strand the cluster on the old chart). Outcome is in kubectl logs.
• One-time / idempotent: before-hook-creation replaces the prior run each upgrade; the sweep skips datasets already carrying new-shape metadata and the backend upsert is a safe overwrite, so after the fleet backfills once it's a cheap no-op. Opt out per-cluster with metadataBackfill.enabled=false.
• Hardened pod: non-root (65534), no SA-token mount (never calls the k8s API), read-only rootfs, all caps dropped, seccomp RuntimeDefault.

Ordering dependency

The Job runs tracebloc-backfill, which only exists in the ingestor image once data-ingestors #393 is released and the image rebuilt. This chart change should ship in a release whose images.ingestor.tag points at an ingestor that carries the entrypoint. Until then the hook is best-effort (logs + exits 0) and re-runs cleanly on the next upgrade once the image has it.

Tests

client/tests/metadata_backfill_hook_test.yaml (helm-unittest, 7 cases): renders by default; hook + delete-policy annotations; image tag vs digest modes; cred + MySQL env; hardened pod; disabled-by-flag. Full chart suite green (274); helm lint --strict clean.

🤖 Generated with Claude Code

---

Note

Medium Risk
Runs on every upgrade with backend/MySQL access and can mutate global metadata, but failures are swallowed and the sweep is idempotent; manual upgrades on cold clusters need adequate Helm --timeout to avoid hook deadline failures.

Overview
Adds a post-upgrade Helm hook Job that runs the ingestor tracebloc-backfill entrypoint after chart upgrades so pre–federated-alignment datasets get enriched global metadata from MySQL and are pushed back to the backend. It is gated by metadataBackfill.enabled (default on), nil-safe for --reuse-values upgrades missing the new key, and not hooked on post-install.

The Job reuses images.ingestor (tag/digest), release CLIENT_ID/CLIENT_PASSWORD, in-cluster MySQL, and proxy env like ingestion jobs. It is best-effort: in-container timeout, always exit 0, backoffLimit: 0, and activeDeadlineSeconds derived from backfillTimeoutSeconds to cap stuck pulls without failing the upgrade. Pod hardening matches other one-off jobs (non-root, no SA token, read-only rootfs).

values.yaml, values.schema.json, and helm-unittest coverage (metadata_backfill_hook_test.yaml) document and lock in hook annotations, image modes, env, deadlines, and the disable flag.

^{Reviewed by Cursor Bugbot for commit 17ec7cb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4be4357. Configure here.}