What happened
spur k8s up with the default cni = "kuberouter" on three cloud VMs (OCI) in one subnet. Pod-to-pod traffic across nodes fails completely. CoreDNS answers only when the client Pod runs on the same node, so name resolution inside the cluster looks intermittent, and no log names the cause.
kube-router runs in its default overlay-type=subnet mode and routes pod packets directly between nodes of the same subnet. Those packets leave the node with a pod address as the source. An OCI VNIC drops a packet whose source is not one of its own addresses unless skip-source-dest-check is set on the VNIC.
Reproduced with ping from node 2 to a Pod on node 3:
100% packet loss
route: 10.244.0.5 via 10.0.255.204 dev enp0s9 src 10.0.255.146
Workaround used for the test: add --overlay-type=full --enable-overlay=true to the kube-router container arguments in /var/lib/k0s/manifests/kuberouter/kube-router.yaml. Every pod packet then travels inside the IPIP tunnel with the node address as the outer source, the same ping has 0 % loss, and every later step of the test passed. Setting skip-source-dest-check on the VNICs would work too; it was not done, because it is a network security control.
Related observation, same code path
With cni = "kuberouter", k0s_controller_config_yaml returns None, so k0s runs with its own defaults: podCIDR 10.244.0.0/16 and serviceCIDR 10.96.0.0/12. The [cluster] defaults pod_cidr = "10.42.0.0/16" and service_cidr = "10.43.0.0/16" are written only for Calico. The documented CIDRs and the running CIDRs differ, and any firewall rule written from the config is wrong on a kube-router cluster.
What was expected
A cluster built by spur k8s up with the default CNI passes pod traffic between nodes on OCI, or docs/deployment/managed-kubernetes.rst says what OCI needs.
Environment
- k0s v1.36.2, three VMs, one subnet, control plane on node 1 and two workers
spur at 2dfe3bb, default [cluster] settings apart from enabled = true
What happened
spur k8s upwith the defaultcni = "kuberouter"on three cloud VMs (OCI) in one subnet. Pod-to-pod traffic across nodes fails completely. CoreDNS answers only when the client Pod runs on the same node, so name resolution inside the cluster looks intermittent, and no log names the cause.kube-router runs in its default
overlay-type=subnetmode and routes pod packets directly between nodes of the same subnet. Those packets leave the node with a pod address as the source. An OCI VNIC drops a packet whose source is not one of its own addresses unless skip-source-dest-check is set on the VNIC.Reproduced with
pingfrom node 2 to a Pod on node 3:Workaround used for the test: add
--overlay-type=full --enable-overlay=trueto the kube-router container arguments in/var/lib/k0s/manifests/kuberouter/kube-router.yaml. Every pod packet then travels inside the IPIP tunnel with the node address as the outer source, the same ping has 0 % loss, and every later step of the test passed. Setting skip-source-dest-check on the VNICs would work too; it was not done, because it is a network security control.Related observation, same code path
With
cni = "kuberouter",k0s_controller_config_yamlreturnsNone, so k0s runs with its own defaults: podCIDR10.244.0.0/16and serviceCIDR10.96.0.0/12. The[cluster]defaultspod_cidr = "10.42.0.0/16"andservice_cidr = "10.43.0.0/16"are written only for Calico. The documented CIDRs and the running CIDRs differ, and any firewall rule written from the config is wrong on a kube-router cluster.What was expected
A cluster built by
spur k8s upwith the default CNI passes pod traffic between nodes on OCI, ordocs/deployment/managed-kubernetes.rstsays what OCI needs.Environment
spurat2dfe3bb, default[cluster]settings apart fromenabled = true