The native-host E2E test
tests/native_host/e2e/test_wg_k0s.py::TestServiceCidrOverMesh::test_clusterip_service_reachable_cross_node
fails from time to time. The failures occur on branches that do not change the
mesh, the k0s integration or the test itself. A later run of the same code
passes.
What happened
E2E run 34460879775, job native-host (non-gpu 0/12), 2026-09-10 [1]:
tests/native_host/e2e/test_wg_k0s.py::TestServiceCidrOverMesh::test_clusterip_service_reachable_cross_node FAILED [ 96%]
E TimeoutError: ClusterIP 10.48.100.60 reachable from a pod on ci-server2-nogpu-10-34460879775-0-3 not met within 90s
/tmp/spur-e2e-34460879775-non-gpu-0/e2e-assets/tests/native_host/e2e/test_wg_k0s.py:447:
/tmp/spur-e2e-34460879775-non-gpu-0/e2e-assets/tests/native_host/e2e/wg_cluster.py:302: TimeoutError
===== 1 failed, 24 passed, 2 skipped, 366 deselected in 524.87s (0:08:44) ======
All steps before the last check passed. The k0s cluster started, the server pod
got an address, the ClusterIP was inside the service CIDR, and the client pod
got an address. Only the request from the client pod to the ClusterIP got no
answer in 90 s. The other 13 native-host shards of the same run passed.
Expected behavior
The test gives the same result for the same code.
How often it occurs
A scan of the 100 most recent failed E2E runs, from 2026-08-20 to 2026-09-10,
shows four failures of this test. Each failure is in the shard
native-host (non-gpu 0/12). The failures show two different messages:
- ClusterIP not reachable in 90 s: run 34460879775 [1].
calico-node DaemonSet Running cluster-wide not met within 480s: runs
33081087905 [2], 33069852677 [3] and 33044385015 [4]. All three ran on main
on 2026-08-27.
In 12 other failed E2E runs the same shard failed for a different reason, and
this test reported PASSED or SKIPPED in the same job.
Run 34460879775 started in a burst of nine E2E runs that started between
09:28:48 and 09:29:53 UTC on 2026-09-10.
This is not the same symptom as issue 747, which describes the fixed Postgres
host port and a 60 s state deadline.
How to reproduce
Run the E2E workflow. The test runs in the job native-host (non-gpu 0/12).
There is no known deterministic trigger. The failure occurs on branches that do
not touch tests/native_host/ or the mesh code.
References
The native-host E2E test
tests/native_host/e2e/test_wg_k0s.py::TestServiceCidrOverMesh::test_clusterip_service_reachable_cross_nodefails from time to time. The failures occur on branches that do not change the
mesh, the k0s integration or the test itself. A later run of the same code
passes.
What happened
E2E run 34460879775, job
native-host (non-gpu 0/12), 2026-09-10 [1]:All steps before the last check passed. The k0s cluster started, the server pod
got an address, the ClusterIP was inside the service CIDR, and the client pod
got an address. Only the request from the client pod to the ClusterIP got no
answer in 90 s. The other 13
native-hostshards of the same run passed.Expected behavior
The test gives the same result for the same code.
How often it occurs
A scan of the 100 most recent failed E2E runs, from 2026-08-20 to 2026-09-10,
shows four failures of this test. Each failure is in the shard
native-host (non-gpu 0/12). The failures show two different messages:calico-node DaemonSet Running cluster-wide not met within 480s: runs33081087905 [2], 33069852677 [3] and 33044385015 [4]. All three ran on
mainon 2026-08-27.
In 12 other failed E2E runs the same shard failed for a different reason, and
this test reported PASSED or SKIPPED in the same job.
Run 34460879775 started in a burst of nine E2E runs that started between
09:28:48 and 09:29:53 UTC on 2026-09-10.
This is not the same symptom as issue 747, which describes the fixed Postgres
host port and a 60 s state deadline.
How to reproduce
Run the
E2Eworkflow. The test runs in the jobnative-host (non-gpu 0/12).There is no known deterministic trigger. The failure occurs on branches that do
not touch
tests/native_host/or the mesh code.References