Auto-upgrade per-dataset ingestor subchart releases

[saadqbal]

Context

Today's auto-update story (after #158 / v1.4.1):

• Parent client chart: customers' autoUpgrade CronJob polls gh-pages on a schedule, runs helm upgrade when a newer chart version publishes. Existing installs auto-update.
• Ingestor image (digest): image-refresh CronJob polls ghcr.io for new digests under a floating tag and applies them via kubectl set env. Existing deployments auto-update.
• Ingestor SUBCHART (tracebloc/ingestor, per-dataset releases): ❌ no auto-update. Customers who installed helm install <dataset> tracebloc/ingestor releases stay on whatever ingestor chart version they were installed with. New chart versions only apply to new installs.

Why this matters

Each helm install <dataset> tracebloc/ingestor ... is a separate Helm release in the customer's namespace. The ingestor subchart contains the ingest-config ConfigMap shape, the post-install hook job that POSTs to jobs-manager, and the SA. If we change any of those — say to fix a bug in the POST-body shape or to support a new ingest-config field — already-installed dataset releases keep running the old behaviour until someone manually helm upgrades each one.

Per the activity in this repo, there can be many dataset releases per cluster (~15 on the dev cluster). Touching each one by hand isn't realistic.

Proposed approach

Mirror the parent chart's autoUpgrade pattern, scoped per release.

Option A — embed an auto-upgrade CronJob inside each ingestor release

• Each helm install <dataset> tracebloc/ingestor adds a tiny CronJob in the dataset's namespace.
• CronJob polls the helm repo for newer ingestor chart versions.
• Runs helm upgrade <dataset> tracebloc/ingestor --version <new> --reuse-values when a newer version is available.
• Namespace-scoped RBAC, narrow trust boundary.

Cost: one extra CronJob per dataset release. At ~15 dataset releases per cluster that's 15 extra schedule slots. Probably fine; CronJobs are cheap.

Option B — central cron on the parent chart that upgrades all ingestor releases

• Parent client chart's autoUpgrade CronJob also handles ingestor releases.
• Iterates helm list -A | grep ingestor-, runs upgrade on each.
• One CronJob covers all dataset releases.

Cost: cluster-admin (or at least namespace-list + helm-upgrade-anywhere) RBAC. Wider trust boundary than (A).

Recommend (A) — same trust boundary as the existing per-release autoUpgrade.

Acceptance criteria

• New CronJob templated in the ingestor subchart, gated by an opt-in/opt-out value (consistent with parent chart's autoUpgrade.enabled).
• Polls https://tracebloc.github.io/client for ingestor chart versions.
• Uses --reset-then-reuse-values (preserves customer's ingest-config override).
• Namespace-scoped RBAC.
• Same :23 schedule + hardened pod (PSA restricted) pattern as parent autoUpgrade.
• Unit tests pinned: enabled/disabled, --reuse-values nil-guard, RBAC scope.
• Dev-cluster smoke test before release.

Notes

Not urgent — defer until either (a) we're about to ship a breaking ingestor chart change that needs to propagate, or (b) we accumulate enough small ingestor-chart fixes that the per-dataset manual upgrade story becomes painful.

Filed as follow-up to #158 / PR #159 / v1.4.1.

[github-actions]

👋 This issue has had no activity for 6 weeks.

If it's still relevant, please leave a comment with current context (or assign someone). Otherwise it will auto-close in 2 weeks.

To exempt permanently, add the keep-open label.