fix(core): mount a fresh tmpfs over /run in docker mode (#214) - #282
Merged
Conversation
Extend docker_publish_reaches_inner_container with an unclean-stop phase: SIGKILL the VMM, plain `izba start`, and require the nested engine to come back, the earlier nested container to survive in `docker ps -a`, and the workload's /var/run → /run to sit on a fresh tmpfs. Fails on this tree with dockerd's 'process with PID N is still running' refusal. Refs #214
The workload's /run lived on the persistent rw disk (overlay upper), so docker.pid/containerd.pid outlived an unclean stop; on the next boot the same low PIDs existed again and dockerd refused to start, leaving the sandbox permanently docker-less (no auto-restart). Mount a tmpfs over the container's /run in the docker-mode OCI spec — ordered before the /run/izba/* binds, which crun would otherwise see shadowed — so the pid files structurally cannot persist. /var/run → /run is a symlink in every mainstream base image and crun resolves it inside the rootfs. Non-docker specs are unchanged (guard-tested). Closes #214
… cover usb in the /run ordering test The plan still showed the pre-ruling wait_pid_gone helper and no pre-kill sync; record both as-shipped amendments. The /var/run → /run symlink is followed by the guest kernel when dockerd opens the pidfile, not resolved by crun (the mount destination is the real /run). The ordering guard now also enables USB so every /run/izba-sourced bind is in the spec. Refs #214
…ut crashed sidecars A CI attempt failed the #214 unclean-stop restart with "virtiofsd:izba-trust did not create <run>/fs-izba-trust.sock within 3s" while the guest-only diagnostics could not say whether the fresh sidecar was slow or died. Dump the last lines of logs/vmm.log and logs/virtiofsd-*.log alongside the console, and wait for the crashed run's sidecars to be dead before the restart so the phase models a reboot rather than racing the old teardown. Refs #214
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #214
What
A docker-mode sandbox that was stopped uncleanly (VMM killed, host reboot) booted fine but its nested Docker Engine never came back:
docker.pid/containerd.pidlived on the persistent rw disk (the overlay upper), the same low PIDs recur on every boot (pause = 1, dockerd = 2), and dockerd's pidfile check refused withprocess with PID 2 is still running. With no auto-restart by contract, the sandbox stayed permanently docker-less until someone deleted the files by hand.The fix mounts a fresh
tmpfsover the container's/runin the docker-mode OCI spec only (add_docker_run_tmpfs, optionsnosuid,nodev,mode=755— what a real host's/runlooks like), ordered before the/run/izba/*bind mounts (crun mounts in array order; the VNC secrets bind lives beneath/run). Every mainstream base image symlinks/var/run → /run, so dockerd's default pidfile paths land on the tmpfs and structurally cannot outlive a boot.How it was verified
TDD against real KVM microVMs:
docker_publish_reaches_inner_containergained phases [5]–[7]: in-guestsync, SIGKILL the VMM, plainizba start, the engine must come back, the nested container from before the kill must still be indocker ps -a, and/var/run → /runmust sit ontmpfs. RED on the unfixed tree with the exact refusal line in the engine log; GREEN with the fix.wait(2)s its VMMs, so a SIGKILLed VMM lingers as a zombie (the wait helper treats stateZas dead, mirroringprocmgr::unix::pid_alive); and a kill seconds after boot races ext4 writeback so the bug does not reproduce without an in-guestsyncfirst (three real-VM runs across dockerd 28.5.2 and 29.7.2 came back healthy without it; with it, the refusal is deterministic).vnc_docker_e2e(docker + VNC, the shape with a bind beneath/run) also GREEN on a real VM.tmpfsat/runwith the exact options; the tmpfs precedes every mount whose destination is beneath/run; a non-docker spec (with and without VNC) has no/runmount.Acceptance criteria (#214)
izba start→ engine starts with no manual interventioncontainerd.piddocker infoonly answers once containerd is up; the log scan catches either layer's refusal/var/run → /runstill workdocker ps -astill lists the pre-kill container)Out of scope, untouched: no auto-restart; nothing is deleted from the rw disk; #207 liveness work.
🤖 Generated with Claude Code
Greptile Summary
The PR gives docker-mode sandboxes a fresh
/runtmpfs on every boot so stale Docker and containerd pidfiles cannot survive an unclean shutdown./rundescendant binds.Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains.
Important Files Changed
/runtmpfs before descendant binds and guards its exact options, ordering, and mode isolation./runinvariant, mount-order dependency, and compatibility boundary.Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart LR Stop[Unclean VMM stop] --> Disk[Persistent overlay retains data] Restart[Next sandbox boot] --> Tmpfs[Mount fresh tmpfs at /run] Tmpfs --> Binds[Apply later /run/izba binds] Binds --> Engine[Start dockerd and containerd] Disk --> State[Preserve /var/lib/docker state] Tmpfs --> Pids[Create fresh runtime pidfiles] State --> Engine Pids --> EngineReviews (2): Last reviewed commit: "test(cli): dump host VMM/virtiofsd logs ..." | Re-trigger Greptile