[bug]: tigera-operator-chart 3.32.2 requests calico-kube-controllers:v3.32.2, which doesn't seem to exist

[jemag]

Docker Hardened Image

tigera-operator-chart:3.32.2

Bug Description

After re-vendoring tigera-operator-chart 3.32.2, our Calico installation couldn't fully roll out. The chart now ships operator 1.42.6, and that operator version asks for Calico v3.32.2 component images. As far as I can tell, dhi.io/calico-kube-controllers:v3.32.2 doesn't exist, so the kube-controllers pod gets stuck in ImagePullBackOff.

The unprefixed tag dhi.io/calico-kube-controllers:3.32.2 does exist (sha256:22fddf87548d791907e398c1b3c6a1df648f19264e009fa6a98959a60fb6ef58). The other components we use have both the 3.32.2 and v3.32.2 tags (node, typha, apiserver, pod2daemon-flexvol, goldmane, whisker, whisker-backend, csi, node-driver-registrar). So it may just be a missing v alias for kube-controllers. calico-cni also seems to have no v3.32.2 tag, but I haven't checked whether 3.32 still pulls that image.

On the same rollout, dhi.io/calico-typha:v3.32.2 (sha256:19d7d2d36a0b6987284c010e6e5d6a4dce5524e551da4769dfa35903226d5e50) went into CrashLoopBackOff with a Go panic (log below). The upstream quay.io/calico/typha:v3.32.2 pods were running at the same time against the same cluster data without problems, so I suspect something in the DHI build. I may be wrong about that, though.

We went back to operator 1.42.5 (Calico v3.32.1 images), and everything is healthy again.

One more thing that might be useful: the 3.32.2 chart tag seems to have been republished with different contents. When we vendored it on 2026-09-17, it had appVersion: 1.42.5 and tigera-operator:1.42.5-debian13@sha256:df907e958eb6df17532736232f1f6e2cd42360e6652c20ff985ae2efbd17ac6d. Today the tag points to manifest sha256:582b22eadaf0ed525f71de72176f499ea33b006398c36e7bd1992f132590ed2b (created 2026-09-23T10:13:14Z) with appVersion: 1.42.6. If republishing under the same version is expected, that's fine, but it caught us off guard because the version stayed the same while the operator version changed.

Steps to Reproduce

1. helm pull oci://dhi.io/tigera-operator-chart --version 3.32.2 --untar
2. Check Chart.yaml: appVersion: 1.42.6, operator image dhi/tigera-operator:1.42.6-debian13@sha256:4b1655578ed0e6b201ae86f65e6bacbab58c96ac2abecee1ff80e0fc291788be.
3. Install it on a cluster. With default values the Installation uses registry: dhi.io/ and imagePrefix: calico-.
4. The operator creates calico-kube-controllers with image dhi.io/calico-kube-controllers:v3.32.2.
5. Checking the registry directly:

   $ regctl image digest dhi.io/calico-kube-controllers:v3.32.2
   failed to request manifest head dhi.io/calico-kube-controllers:v3.32.2: request failed: not found
   $ regctl image digest dhi.io/calico-kube-controllers:3.32.2
   sha256:22fddf87548d791907e398c1b3c6a1df648f19264e009fa6a98959a60fb6ef58
   

Expected Behavior

Every image the operator bundled in the chart requests exists on dhi.io, so the Calico components start normally.

Actual Behavior

• calico-kube-controllers stays in ImagePullBackOff (not found).
• calico-typha (v3.32.2) crashes on startup with the panic below.
• The other components roll out.

Environment

• Orchestration: Kubernetes v1.35.6 (AKS)
• Node OS: Microsoft Azure Linux 3.0
• Container runtime: containerd 2.2.4
• Helm 3.20.2 (rendered through Grafana Tanka 0.38.0)

Relevant Logs

Failed to pull image "<acr>/aks/dhi/calico-kube-controllers:v3.32.2": rpc error: code = NotFound desc = failed to pull and unpack image "<acr>/aks/dhi/calico-kube-controllers:v3.32.2": failed to resolve image: <acr>/aks/dhi/calico-kube-controllers:v3.32.2: not found

# calico-typha v3.32.2
panic: runtime error: makeslice: len out of range [recovered, repanicked]

goroutine 106 [running]:
encoding/json.(*encodeState).marshal.func1()
	encoding/json/encode.go:339 +0x68
...
github.com/projectcalico/calico/libcalico-go/lib/json.Marshal(...)
	github.com/projectcalico/calico/libcalico-go/lib/json/json.go:21
github.com/projectcalico/calico/libcalico-go/lib/backend/model.SerializeValue(0x2ba8b5335990)
	github.com/projectcalico/calico/libcalico-go/lib/backend/model/keys.go:686 +0x11a
github.com/projectcalico/calico/typha/pkg/syncproto.SerializeUpdate(...)
	github.com/projectcalico/calico/typha/pkg/syncproto/sync_proto.go:370 +0x44e
github.com/projectcalico/calico/typha/pkg/snapcache.(*Cache).publishBreadcrumb(0x2ba8b4f93d40)
	github.com/projectcalico/calico/typha/pkg/snapcache/cache.go:417 +0x378
github.com/projectcalico/calico/typha/pkg/snapcache.(*Cache).loop(0x2ba8b4f93d40, {0x2aa65c0, 0x4530600}, 0x0?)
	github.com/projectcalico/calico/typha/pkg/snapcache/cache.go:289 +0x59

Additional Context

Upstream operator v1.42.6 pins these tags in pkg/components/calico.go (Version: "v3.32.2" for kube-controllers, node, typha, etc.), which is why the v-prefixed tag is being requested.

For now we're pinning the operator to 1.42.5-debian13@sha256:df907e958eb6df17532736232f1f6e2cd42360e6652c20ff985ae2efbd17ac6d as a workaround.

Pre-submission Checklist

• I have searched existing issues to ensure this bug hasn't been reported before
• I have provided all the requested information above
• I have tested this with the latest available version of the hardened image

[jemag]

Another issue I noticed while looking into this: the chart's Chart.yaml has version: v0.0.

$ helm pull oci://dhi.io/tigera-operator-chart --version 3.32.2
$ tar -xzOf tigera-operator-chart-3.32.2.tgz tigera-operator-chart/Chart.yaml | grep -E
'^(name|version|appVersion):'
appVersion: 1.42.6
name: tigera-operator-chart                                                                              
version: v0.0                                                                                            

I think this comes from the upstream source tree. charts/tigera-operator/Chart.yaml in
projectcalico/calico has version: v0.0 with a comment saying it gets overridden when the chart is
packaged, and the chart Tigera publishes at https://docs.tigera.io/calico/charts has version: v3.32.2.
The other DHI charts we use all have their real version in Chart.yaml, so this looks specific to the
tigera-operator chart.

It's not a big problem on its own, but tools that compare the vendored Chart.yaml version with the
requested version (Tanka in our case) see a mismatch every time and re-pull the chart. That's how we
ended up picking up the republished 3.32.2 without realizing it. It would be nice if the chart version
matched the tag.