Skip to content

fix(core): mount a fresh tmpfs over /run in docker mode (#214) - #282

Merged
Lupus merged 5 commits into
mainfrom
fix/docker-stale-pidfiles
Sep 3, 2026
Merged

fix(core): mount a fresh tmpfs over /run in docker mode (#214)#282
Lupus merged 5 commits into
mainfrom
fix/docker-stale-pidfiles

Conversation

@Lupus

@Lupus Lupus commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Closes #214

What

A docker-mode sandbox that was stopped uncleanly (VMM killed, host reboot) booted fine but its nested Docker Engine never came back: docker.pid / containerd.pid lived on the persistent rw disk (the overlay upper), the same low PIDs recur on every boot (pause = 1, dockerd = 2), and dockerd's pidfile check refused with process with PID 2 is still running. With no auto-restart by contract, the sandbox stayed permanently docker-less until someone deleted the files by hand.

The fix mounts a fresh tmpfs over the container's /run in the docker-mode OCI spec only (add_docker_run_tmpfs, options nosuid,nodev,mode=755 — what a real host's /run looks like), ordered before the /run/izba/* bind mounts (crun mounts in array order; the VNC secrets bind lives beneath /run). Every mainstream base image symlinks /var/run → /run, so dockerd's default pidfile paths land on the tmpfs and structurally cannot outlive a boot.

How it was verified

TDD against real KVM microVMs:

  • Reproduce first. docker_publish_reaches_inner_container gained phases [5]–[7]: in-guest sync, SIGKILL the VMM, plain izba start, the engine must come back, the nested container from before the kill must still be in docker ps -a, and /var/run → /run must sit on tmpfs. RED on the unfixed tree with the exact refusal line in the engine log; GREEN with the fix.
  • Two findings along the way, both now in the test's comments: izbad never wait(2)s its VMMs, so a SIGKILLed VMM lingers as a zombie (the wait helper treats state Z as dead, mirroring procmgr::unix::pid_alive); and a kill seconds after boot races ext4 writeback so the bug does not reproduce without an in-guest sync first (three real-VM runs across dockerd 28.5.2 and 29.7.2 came back healthy without it; with it, the refusal is deterministic).
  • vnc_docker_e2e (docker + VNC, the shape with a bind beneath /run) also GREEN on a real VM.
  • Three unit guards: exactly one tmpfs at /run with the exact options; the tmpfs precedes every mount whose destination is beneath /run; a non-docker spec (with and without VNC) has no /run mount.
  • All six workspace gates + the app gate green locally.

Acceptance criteria (#214)

Criterion Where
Unclean stop → izba start → engine starts with no manual intervention e2e phases [5]/[6]
Same for the second-layer containerd.pid tmpfs covers both paths; docker info only answers once containerd is up; the log scan catches either layer's refusal
Images that symlink /var/run → /run still work e2e phase [7] on the Alpine-based dind image
Non-pid state on persistent storage is preserved e2e phase [6b] (docker ps -a still lists the pre-kill container)
An e2e covers the unclean-stop → restart → engine-up sequence this PR's Task 1

Out of scope, untouched: no auto-restart; nothing is deleted from the rw disk; #207 liveness work.

🤖 Generated with Claude Code

Greptile Summary

The PR gives docker-mode sandboxes a fresh /run tmpfs on every boot so stale Docker and containerd pidfiles cannot survive an unclean shutdown.

  • Adds the docker-only OCI mount before all /run descendant binds.
  • Adds unit coverage for mount options, ordering, and non-docker isolation.
  • Extends the real-VM Docker lifecycle test through an unclean VMM stop and engine recovery.
  • Documents the runtime invariant and implementation plan.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
crates/izba-core/src/image/runtime_config.rs Adds the docker-only /run tmpfs before descendant binds and guards its exact options, ordering, and mode isolation.
crates/izba-cli/tests/daemon_e2e.rs Extends the Docker real-VM test to verify recovery after an unclean VMM stop while preserving nested Docker state.
CLAUDE.md Records the fresh-/run invariant, mount-order dependency, and compatibility boundary.
docs/superpowers/plans/2026-09-03-docker-run-tmpfs-stale-pidfiles.md Documents the implementation and verification plan for preventing stale Docker pidfiles.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Stop[Unclean VMM stop] --> Disk[Persistent overlay retains data]
  Restart[Next sandbox boot] --> Tmpfs[Mount fresh tmpfs at /run]
  Tmpfs --> Binds[Apply later /run/izba binds]
  Binds --> Engine[Start dockerd and containerd]
  Disk --> State[Preserve /var/lib/docker state]
  Tmpfs --> Pids[Create fresh runtime pidfiles]
  State --> Engine
  Pids --> Engine
Loading

Reviews (2): Last reviewed commit: "test(cli): dump host VMM/virtiofsd logs ..." | Re-trigger Greptile

Extend docker_publish_reaches_inner_container with an unclean-stop phase:
SIGKILL the VMM, plain `izba start`, and require the nested engine to come
back, the earlier nested container to survive in `docker ps -a`, and the
workload's /var/run → /run to sit on a fresh tmpfs. Fails on this tree with
dockerd's 'process with PID N is still running' refusal.

Refs #214
The workload's /run lived on the persistent rw disk (overlay upper), so
docker.pid/containerd.pid outlived an unclean stop; on the next boot the
same low PIDs existed again and dockerd refused to start, leaving the
sandbox permanently docker-less (no auto-restart). Mount a tmpfs over the
container's /run in the docker-mode OCI spec — ordered before the
/run/izba/* binds, which crun would otherwise see shadowed — so the pid
files structurally cannot persist. /var/run → /run is a symlink in every
mainstream base image and crun resolves it inside the rootfs. Non-docker
specs are unchanged (guard-tested).

Closes #214
… cover usb in the /run ordering test

The plan still showed the pre-ruling wait_pid_gone helper and no pre-kill
sync; record both as-shipped amendments. The /var/run → /run symlink is
followed by the guest kernel when dockerd opens the pidfile, not resolved
by crun (the mount destination is the real /run). The ordering guard now
also enables USB so every /run/izba-sourced bind is in the spec.

Refs #214
@Lupus Lupus added this to the v0.1.0 (MVP) milestone Sep 3, 2026
…ut crashed sidecars

A CI attempt failed the #214 unclean-stop restart with "virtiofsd:izba-trust
did not create <run>/fs-izba-trust.sock within 3s" while the guest-only
diagnostics could not say whether the fresh sidecar was slow or died. Dump
the last lines of logs/vmm.log and logs/virtiofsd-*.log alongside the
console, and wait for the crashed run's sidecars to be dead before the
restart so the phase models a reboot rather than racing the old teardown.

Refs #214
@sonarqubecloud

sonarqubecloud Bot commented Sep 3, 2026

Copy link
Copy Markdown

@Lupus
Lupus merged commit 592b027 into main Sep 3, 2026
56 checks passed
@Lupus
Lupus deleted the fix/docker-stale-pidfiles branch September 3, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker mode: stale docker.pid/containerd.pid on the persistent rw disk blocks the engine after an unclean stop

1 participant