Docker 29 in Lima shows containers that it cannot delete

[rfay]

Description

In the latest versions of Lima our DDEV test runners are ending up with containers that cannot be deleted. (macOS 26.2, ARM64, VZ)

Right now on tb-macos-arm64-7 we have a ddev ssh-agent container that cannot be deleted, and this has happened a number of times lately. The only fix I know of is to completely destroy and recreate the lima instance.

testbot@tb-macos-arm64-7 workspace % docker ps -a
CONTAINER ID   IMAGE                                                       COMMAND                 CREATED             STATUS                    PORTS     NAMES
497e249b4d59   ddev/ddev-ssh-agent:20251222_stasadev_tzdata_legacy-built   "/entry.sh ssh-agent"   About an hour ago   Created                             dbcc431b5cef_ddev-ssh-agent
dbcc431b5cef   43a94aa3f90f                                                "/entry.sh ssh-agent"   2 days ago          Exited (137) 2 days ago

% docker rm -f $(docker ps -aq)
497e249b4d59
Error response from daemon: No such container: dbcc431b5cef

% docker rm -f $(docker ps -aq)
497e249b4d59
Error response from daemon: No such container: dbcc431b5cef

Here's how we build lima test instances: https://docs.ddev.com/en/stable/developers/buildkite-testmachine-setup/#additional-lima-macos-setup

We've been using that technique for years without trouble; I think this only popped up in v2.0.2 perhaps.

These lima instances have docker engine 29.1.3

The container remains after docker system prune --volumes --force

Inside limactl ssh lima-vz there is nothing in /var/lib/docker/containers

broken-limactl-info.json.txt

failure context

limactl factory-reset lima-vz does solve the problem, as we'd expect. It's easier than what I've done before.

[rfay]

I understand that this looks clearly like a docker error... but we test extensively in lots of contexts, and we're seeing this only on lima, and only recently.

[rfay]

This is happening so often that it's failing almost every PR test. We may have to turn off testing on lima. https://buildkite.com/ddev/macos-lima/builds/5296#019b6108-71bf-4913-a873-447c6a91e396

[AkihiroSuda]

Can you try downgrading docker to identify whether this is a regression in docker or lima?

[rfay]

Thanks, yes, I'm on my way to downgrading docker in Lima. I'm working on reproducing consistently first. However, several other environments are in fact using Docker 29.1.3 (engine), including Docker Desktop on ARM64.

[rfay]

• The full DDEV test run leaves containers "behind" although they don't show in /var/lib/docker/containers
• Then the next run fails because of undeletable containers.
• Downgrading docker-ce to 28.5.2 gets rid of the extra containers, but only after stoping and starting the lima instance sudo apt install docker-ce=5:28.5.2-1~ubuntu.24.04~noble docker-ce-cli=5:28.5.2-1~ubuntu.24.04~noble
• Downgrading to 29.1.0 or 29.0.4 does not solve the problem (containers persist, even after lima stop/start)

[rfay]

This is truly odd. Why did it suddenly start happening (on all 3 DDEV macos/arm64 runners)? Why can it be fixed by going down to docker (rootless) 28? If it were a new docker release, why couldn't it be fixed by like 29.0? How could lima be responsible? It's possible that it's a docker rootless issue...

We have a number of docker rootful tests that use docker 29.1.3

But I see that our own docker rootless test is only on 28.0.4, so the lima docker rootless is the only docker rootless we have on docker 29

[afbjorklund]

Did you compare the versions of the other components involved (with docker), such as containerd and runc?

[afbjorklund]

You could also compare with "docker-rootful", to see if the behaviour is isolated to "docker"(-rootless) or not

[rfay]

@AkihiroSuda I see your fingerprints all over moby/moby docker rootless lately, interested if this seems meaningful to you.

You could also compare with "docker-rootful", to see if the behaviour is isolated to "docker"(-rootless) or not

@afbjorklund it's a fair bit of work to test so far, since the only path I know at this point is to run the full set of DDEV tests (3 hours) and then find that one container is un-deletable. But coach me on how I'd use docker rootful in Lima, assuming that's what you're saying.

[afbjorklund]

But coach me on how I'd use docker rootful in Lima, assuming that's what you're saying.

There are two different templates, one "docker" and one "docker-rootful"

Was just wondering if the issue (with Docker v29) was specific to rootless?

limactl start --name docker template:docker-rootful

But I only know about the containerd image store change.

[rfay]

All this time and it was only just the other day that I even knew that the lima template we suggest to people was rootless. @stasadev discovered that in recent work on supporting docker rootless directly, had to change some tests :)

[afbjorklund]

Actually the default template that we suggest is running containerd rootless, but that's a different story I presume

[rfay]

I ran a significant DDEV test run with Docker rootless 29.1.3 on linux and didn't see this behavior. That's not a full report, but after test completion I was able to remove any existing containers.