CI pulls container images from Docker Hub without credentials. Docker Hub
applies an anonymous pull rate limit per source address. When several E2E runs
start together, Docker Hub answers 429 Too Many Requests, and the runs fail
for a reason that has no relation to the code under test.
What happened
On 2026-09-10, nine E2E runs started between 09:28:48 and 09:29:53 UTC. Six of
the nine failed. Two different pulls got a 429 answer.
1. The local-path provisioner helper pod
E2E run 34460953651, job k8s [1]. All 13 tests error in fixture setup with
TimeoutError: timed out after 120s: operator deployment not ready. The
artifact e2e-results-k8s, file cluster-dump/events.txt, shows the cause:
local-path-storage Warning Failed pod/helper-pod-create-pvc-2045a95a-... Failed to pull image "busybox": failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve image: unexpected status from HEAD request to https://registry-1.docker.io/v2/library/busybox/manifests/latest: 429 Too Many Requests
The helper pod never started, so the volume for spurctld-0 was never created.
spurctld-0 stayed Pending, and the operator readiness probe returned 503.
E2E run 34460937591, job k8s [2], failed in the same way in the same minute.
2. spur image import in the native-host suite
E2E run 34460864809, job native-host (non-gpu 11/12) [3]:
E RuntimeError: Command failed on 10.44.10.32 (exit 1): SPUR_IMAGE_DIR='...' '/tmp/spur-ci-bin-34460864809/spur' image import docker.io/library/python:3.11-slim
E Error: registry returned 429 Too Many Requests for manifest of library/python:3.11-slim
What pull request 863 covers, and what it does not
Pull request 863 moved the images that the test suites pull to
mirror.gcr.io. This covers tests/native_host/e2e/cluster.py,
tests/native_host/e2e/test_wg_k0s.py, tests/k8s/e2e/k8s_cluster.py and the
postgres service container in ci.yml.
It does not cover the pulls that start inside the CI virtual machine image. The
local-path provisioner on the CI Kubernetes nodes uses the default helper
image busybox from docker.io. That provisioner comes from the runner image,
not from this repository, so no change in this repository stops that pull.
Expected behavior
CI authenticates to Docker Hub with an account. An authenticated account has a
much higher pull quota than an anonymous client. Parallel E2E runs then do not
exhaust the limit, and a run fails only for a fault in the code under test.
References
CI pulls container images from Docker Hub without credentials. Docker Hub
applies an anonymous pull rate limit per source address. When several E2E runs
start together, Docker Hub answers
429 Too Many Requests, and the runs failfor a reason that has no relation to the code under test.
What happened
On 2026-09-10, nine E2E runs started between 09:28:48 and 09:29:53 UTC. Six of
the nine failed. Two different pulls got a 429 answer.
1. The
local-pathprovisioner helper podE2E run 34460953651, job
k8s[1]. All 13 tests error in fixture setup withTimeoutError: timed out after 120s: operator deployment not ready. Theartifact
e2e-results-k8s, filecluster-dump/events.txt, shows the cause:The helper pod never started, so the volume for
spurctld-0was never created.spurctld-0stayedPending, and the operator readiness probe returned 503.E2E run 34460937591, job
k8s[2], failed in the same way in the same minute.2.
spur image importin the native-host suiteE2E run 34460864809, job
native-host (non-gpu 11/12)[3]:What pull request 863 covers, and what it does not
Pull request 863 moved the images that the test suites pull to
mirror.gcr.io. This coverstests/native_host/e2e/cluster.py,tests/native_host/e2e/test_wg_k0s.py,tests/k8s/e2e/k8s_cluster.pyand thepostgresservice container inci.yml.It does not cover the pulls that start inside the CI virtual machine image. The
local-pathprovisioner on the CI Kubernetes nodes uses the default helperimage
busyboxfromdocker.io. That provisioner comes from the runner image,not from this repository, so no change in this repository stops that pull.
Expected behavior
CI authenticates to Docker Hub with an account. An authenticated account has a
much higher pull quota than an anonymous client. Parallel E2E runs then do not
exhaust the limit, and a run fails only for a fault in the code under test.
References