Cluster creation fails when using kindest/node:v1.35.0 on macOS arm64

[danielmorozoff]

What happened:

Cluster creation fails with kubelet crash loop when using kindest/node:v1.35.0 on macOS arm64. The kubelet inside the control-plane node repeatedly fails with:

E1219 17:30:48.770126 207 run.go:72] "command failed" err="failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
The kubelet cannot connect to containerd's CRI v1 runtime API. kubeadm init times out after 4 minutes waiting for a healthy kubelet at http://127.0.0.1:10248/healthz.

Note: Cluster creation works fine with kindest/node:v1.32.11 and earlier versions.

What you expected to happen:

Cluster should create successfully with the v1.35.0 node image.

How to reproduce it (as minimally and precisely as possible):

# Minimal kind config
cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
  kind: ClusterConfiguration
  apiVersion: kubeadm.k8s.io/v1beta3
nodes:
  - role: control-plane
    image: kindest/node:v1.35.0@sha256:452d707d4862f52530247495d180205e029056831160e22870e37e3f6c1ac31f
EOF
kind create cluster --config kind-config.yaml --retain
# Wait for failure, then:
kind export logs ./kind-logs
kind delete cluster

Anything else we need to know?:

The containerd inside the kind node appears to have issues with the CRI v1 RuntimeService
Docker Desktop's containerd version is 1c4457e00facac03ce1d75f7b6777a7a851e5c41
cgroup version is v2 (not the cgroup v1 issue mentioned in release notes)
Issue does NOT occur with v1.32.11, v1.33.7, or v1.34.3 node images
Environment:

kind version: kind v0.31.0 go1.25.5 darwin/arm64
Runtime info:
Server Version: 29.1.3
containerd version: 1c4457e00facac03ce1d75f7b6777a7a851e5c41
runc version: v1.3.4-0-gd6d73eb8
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
Cgroup Driver: cgroupfs
Cgroup Version: 2
OS: macOS 26.2 (Build 25C56) - Apple Silicon (arm64)
Kubernetes version: Client v1.34.1 (attempting to create v1.35.0 cluster)
Any proxies or other special environment settings?: No

[stmcginnis]

Have you tried with just kind create cluster and no extra configuration?

[danielmorozoff]

kind@0.30.0 works fine...

Creating cluster "*****" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-*****"
You can now use your cluster with:

kubectl cluster-info --context kind-****

Have a nice day! 👋
configmap/local-registry-hosting created

@stmcginnis I have not will do so a bit later and get back to you

[mikhailswift]

Also encountering this same error on x86 using the newest versions of the 1.33, 1.34, and 1.35 kindest images.

[BenTheElder]

Can we get kind export logs (There should be a note in the bug template about this for when cluster creation fails, you need to use --retain when creating the cluster then kind delete cluster can manually clean it up after the logs are dumped)

[BenTheElder]

On my work mac (arm64) with limactl and docker v29.1.2 kind create cluster works @ v0.31.0, unfortunately I cannot use docker desktop (licensing/policy) to replicate so we'll need your cluster logs to investigate further.

re-reading the opening issue, it looks like you already ran kind export logs, Github will let you upload a zip/tarball to a comment / issue.

[BenTheElder]

My best guess is related to runc version which was blocked by kubernetes/kubernetes#132791

The partial docker info output in the OP is missing the runc version, locally I have:

$ docker info
Client: Docker Engine - Community
 Version:    29.1.2
 Context:    lima-default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /opt/homebrew/lib/docker/cli-plugins/docker-buildx

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 27.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-87-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 11.65GiB
 Name: lima-default
 ID: 84667c2f-7cfc-4bae-bee1-2273ed0d5da0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: bentheelder
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

could also be the containerd version, I see some minor errors locally:

Dec 19 18:56:42 kind-control-plane containerd[109]: time="2025-12-19T18:56:42.499117966Z" level=error msg="unable to parse "max 0" as a uint from Cgroup file "/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod67ab49be_36a8_4616_ae7c_f0d7c749b28b.slice/cri-containerd-689d2c4896bac579378b5b9a8a40110ecb865f8a854e33b3568b97619ce3a7d5.scope/hugetlb.32MB.events""

That one we can pickup a fix via containerd/containerd#12579 I think which was just released in 2.2.1

But that's not causing any hard failures locally

I'm inclined to go ahead with upgrading containerd and runc now that we're unblocked again.

We could try kind @ HEAD for e6b3778 which has an updated runc already.

[BenTheElder]

To try that, since there's no key CLI changes you could just try that image directly with 0.31.0, to install from HEAD you can use make install in a clone of the repo.

[mikhailswift]

As far as my issue goes, this is the kind config in use

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
  apiServerAddress: "0.0.0.0"
  apiServerPort: 6443
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://172.16.0.1:35000"]
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /dev
        containerPath: /dev
      - hostPath: /var/run/containerd.sock
        containerPath: /var/run/containerd.sock
      - hostPath: /var/run/falco
        containerPath: /var/run/falco
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          certSANs:
          - "redacted"

Removing containerdConfigPatches allows the cluster to start. When this is included, I see the same error described in the original issue in the kubelet logs

run.go:72] "command failed" err="failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

I'll work on exporting the logs

[BenTheElder]

 - hostPath: /var/run/containerd.sock
       containerPath: /var/run/containerd.sock

Uhhh ... that's not supported, mounting the containerd socket from the host into the node container. That shouldn't have ever worked ... kubelet and containerd need a consistent view of the pods ... dev mount may also cause problems.

Those host path mounts have a high likelihood of breaking things ...

It's interesting that it seems to work without certSAN but ...

[mikhailswift]

It works with the certSAN, but not with

containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://172.16.0.1:35000"]

Here's the exported logs from a test run on my macbook.

I'm still trying to understand the intent behind those mounts, myself.

kind-logs.tar.gz

[mikhailswift]

A minimal config that causes this issue for me is

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://172.16.0.1:35000"]

[mikhailswift]

Looks like my case could be due to containerd changes

https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

NOTE: registry.mirrors and registry.configs as previously described in this document have been DEPRECATED. As described in the cri config you should now use the following configuration

[BenTheElder]

Ah yeah, We actually had some release notes in kind about this, that config was deprecated by containerd a long time ago and shouldn't have specifically broken in this release but ... maybe there was a change in containerd we missed.

https://github.com/kubernetes-sigs/kind/releases?q=Mirror&expanded=true

v0.20 had the warning and a subsequent release switched to the containerd config version we still use today.

#3946 will help enable kind configs to target multiple containerd versions but it's not landed yet

So is kind create cluster no options working for you on x86 @ v0.31.0 then? Would you happen to be using docker desktop on mac?

[mikhailswift]

Yes. Without a config, or a config without that specific config value, on a mac and x86 machine they both boot. Using docker desktop on the mac.