Skip to content

CI pulls images from Docker Hub without credentials and gets 429 Too Many Requests #866

Description

@pre

CI pulls container images from Docker Hub without credentials. Docker Hub
applies an anonymous pull rate limit per source address. When several E2E runs
start together, Docker Hub answers 429 Too Many Requests, and the runs fail
for a reason that has no relation to the code under test.

What happened

On 2026-09-10, nine E2E runs started between 09:28:48 and 09:29:53 UTC. Six of
the nine failed. Two different pulls got a 429 answer.

1. The local-path provisioner helper pod

E2E run 34460953651, job k8s [1]. All 13 tests error in fixture setup with
TimeoutError: timed out after 120s: operator deployment not ready. The
artifact e2e-results-k8s, file cluster-dump/events.txt, shows the cause:

local-path-storage  Warning  Failed  pod/helper-pod-create-pvc-2045a95a-...  Failed to pull image "busybox": failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve image: unexpected status from HEAD request to https://registry-1.docker.io/v2/library/busybox/manifests/latest: 429 Too Many Requests

The helper pod never started, so the volume for spurctld-0 was never created.
spurctld-0 stayed Pending, and the operator readiness probe returned 503.
E2E run 34460937591, job k8s [2], failed in the same way in the same minute.

2. spur image import in the native-host suite

E2E run 34460864809, job native-host (non-gpu 11/12) [3]:

E   RuntimeError: Command failed on 10.44.10.32 (exit 1): SPUR_IMAGE_DIR='...' '/tmp/spur-ci-bin-34460864809/spur' image import docker.io/library/python:3.11-slim
E   Error: registry returned 429 Too Many Requests for manifest of library/python:3.11-slim

What pull request 863 covers, and what it does not

Pull request 863 moved the images that the test suites pull to
mirror.gcr.io. This covers tests/native_host/e2e/cluster.py,
tests/native_host/e2e/test_wg_k0s.py, tests/k8s/e2e/k8s_cluster.py and the
postgres service container in ci.yml.

It does not cover the pulls that start inside the CI virtual machine image. The
local-path provisioner on the CI Kubernetes nodes uses the default helper
image busybox from docker.io. That provisioner comes from the runner image,
not from this repository, so no change in this repository stops that pull.

Expected behavior

CI authenticates to Docker Hub with an account. An authenticated account has a
much higher pull quota than an anonymous client. Parallel E2E runs then do not
exhaust the limit, and a run fails only for a fault in the code under test.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions