Evaluate Apple `container` machine as a macOS backend (alongside or instead of Lima)

[hbrodin]

Summary

Evaluate replacing — or supplementing — the macOS Lima backend with Apple's container tool and its container machine abstraction. Both run guest VMs on Apple's Virtualization.framework, so this is a question of which orchestration layer coop delegates to on macOS, not a change to the virtualization technology itself.

Empirical testing on a real macOS 26.5.1 / Apple-Silicon host (notes below) shows an Apple container backend is viable as an opt-in third backend on macOS 26+. This issue lays out why we'd consider the shift, what we'd gain, and what it would cost.

A note on scope. The relevant comparison is Apple container vs Lima, since that's what it would replace or supplement on macOS — not vs Firecracker (the Linux backend). Lima already runs a full Ubuntu kernel with CONFIG_NF_TABLES and CONFIG_IP_NF_RAW, so the two kernel workarounds coop maintains (iptables-legacy, DOCKER_INSECURE_NO_IPTABLES_RAW) are Firecracker-only and already absent from the Lima path (src/lima.rs:1373). Apple container matches Lima here — it does not improve on it. The honest gains over Lima are first-party tooling and architecture fit; the kernel-workaround story is irrelevant to this decision and is called out below only to forestall the apples-to-oranges framing.

Assumed floor: macOS 26+ (Apple Silicon). Older-macOS support would have to be dropped for an Apple-container-only backend; keeping Lima as a fallback avoids that.

Background

Today coop selects a backend at compile time: Lima on macOS (wrapping Virtualization.framework via limactl), Firecracker on Linux. The shared SshTarget/Backend abstraction in src/backend.rs means all SSH-based operations — config injection, workspace sync, VS Code, port forwarding — are backend-agnostic. A new macOS backend only has to satisfy the same contract: create named VMs, give each a host-reachable SSH endpoint, support live host-directory mounts, and tear down cleanly.

Apple container reached 1.0.0 on 2026-06-09. container machine boots a Linux VM (Apple's Containerization kernel) into which we can bake an Ubuntu+systemd guest image — the same golden-image pattern we already use for Lima.

Why consider the shift — what we'd gain

1. First-party, Apple-maintained tool. container is Apple's own project tracking Virtualization.framework directly, versus Lima as a third-party wrapper. Fewer layers between coop and the platform, and a tighter coupling to the OS it targets.

2. Live mounts at full virtiofs parity. container run --volume /any/abs/path:/dst (or --mount source=…,target=…) live-mounts any absolute host directory — bidirectional virtiofs, changes visible immediately on both sides, matching Lima's live-mount semantics. No $HOME restriction (verified in Parser.swift/validateMount). Only constraint: the host path must exist and be absolute (no auto-create).

3. The SshTarget/Backend abstraction holds. Each machine gets a host-reachable private IP (192.168.64.x via vmnet). Baking sshd + a pubkey into the guest image (the golden-image pattern we already use) and reaching it over the private IP keeps the entire shared SSH layer intact — no coop-side rework.

4. Concurrent multi-machine boot works. coop runs several named VMs at once. Created coop-a and coop-b simultaneously; both showed running with distinct IPs. set-default only governs whether -n can be omitted, not exclusivity.

What it would cost — drawbacks and risks

1. macOS 26 floor. An Apple-container-only macOS backend drops pre-26 support. Mitigation: keep Lima as the macOS fallback and add container as an opt-in third backend, paying the cost of maintaining two macOS paths instead of one.

2. 1.0.0 maturity / lifecycle flakiness. container is freshly 1.0.0. In testing, stop/delete were clean and instant, and of the lifecycle bugs originally flagged only #861 remains open (the rest are closed; #977 was a fixed old-version regression, not a live --rm data-loss bug). Still, this is young software and the lifecycle surface deserves caution.

3. Machine IPs are not stable across reboots. Observed drift (.6→.7, .8→.9) on restart. coop must re-resolve the IP via container machine inspect on each start rather than caching it. This is a real change to how coop tracks an instance's SSH endpoint.

4. container machine run exec is flaky and quirky — intermittent Operation not supported by device during boot races; it word-splits the command (sh -c "echo AAA" runs echo with AAA as $0); and it does not forward piped host stdin. coop already drives guests over SSH, so this is fine for runtime — but provisioning steps cannot rely on machine run to inject files. All provisioning must be baked into the golden image or pushed over SSH.

5. No disk-size knob — coop resize has no analog. machine create/machine set expose only cpus, memory, home-mount. Disk is grow-on-demand EXT4 (sparse on host APFS; the DISK column grew 75M→700M+ as content unpacked). coop's resize command can't be implemented against container. Likely moot given grow-on-demand, but it's a feature-parity gap to acknowledge.

6. --publish loopback forwarding is buggy on macOS 26.1–26.3 (#919, #1321). Use the direct private IP, not -p. A host-side macOS "Local Network" permission gate can apply (not hit in the CLI context tested).

Empirical confirmation (macOS 26.5.1, Apple Silicon, container 1.0.0)

All five originally-open questions now have verdicts from a real host:

Question	Verdict
Concurrent multi-machine boot	✅ Two machines running with distinct IPs
dockerd in Ubuntu+systemd, end-to-end	✅ docker run hello-world pulled+ran; raw table present (full kernel, like Lima — no FC-style workarounds needed)
SSH over the private IP	✅ Ping + TCP/22 + key auth as host-matching user, end-to-end
Disk sizing / resize	✅ No disk knob; grow-on-demand EXT4 (resize moot)
Lifecycle robustness on 1.0.x	✅ Clean stop/delete; only #861 still open

Two corrections to earlier desk research: the disk is EXT4 grow-on-demand images sparse on host APFS (not "APFS thin-provisioned" per-machine volumes); and #977 is fixed, not a live data-loss bug.

Recommendation

The case here is weaker than the Firecracker comparison would suggest, because vs Lima there is no kernel-workaround win — both run full kernels. The genuine upside over Lima is first-party tooling and tighter OS coupling, with mount/SSH parity (not improvement). Against that sit real costs: a macOS 26 floor, 1.0.0 lifecycle maturity, per-reboot IP churn, provisioning that can't use machine run, and no coop resize analog.

Given that the upside over Lima is modest and the costs are concrete, the recommendation is to treat this as exploratory / opt-in only — worth a prototype third backend behind the existing compile-time selection (Lima stays the macOS default), but not a reason to drop Lima. Revisit if Apple container matures past 1.0.x or if a future need (e.g. dropping the third-party Lima dependency, or a capability Lima lacks) tips the balance.

Implementation notes for whoever picks this up

• New backend module mirroring src/lima.rs, wired into the Backend enum in src/backend.rs.
• Golden image: ubuntu:24.04 + systemd + docker.io + openssh-server, pubkey baked into /etc/skel.
• Re-resolve the guest IP via container machine inspect on every start; never cache it.
• Drive all guest interaction over SSH; never depend on container machine run for provisioning or stdin.
• No resize implementation; document the grow-on-demand behavior instead.
• Reuse direct private-IP connectivity; avoid --publish.

[hbrodin]

Research: feasibility of an Apple container backend

Assuming macOS 26+ is the floor (dropping older-macOS support), I dug into Apple's container repo (docs + Swift source + issue tracker) to verify whether it could serve as a coop backend alongside Lima/Firecracker. Summary: viable as an opt-in third backend, and notably simpler than Firecracker on the Docker-networking front. Details below, then a checklist of what still needs confirming on a real macOS 26 box.

Verified ✅

Nested Docker works — and cleaner than our Firecracker path. Apple's Containerization kernel (config-arm64) compiles in everything Docker needs — overlayfs, bridge, veth, conntrack, NAT — including both CONFIG_NF_TABLES=y and CONFIG_IP_NF_RAW=y. Those are the exact two modules the Firecracker CI kernel lacks and that we work around (iptables-legacy + DOCKER_INSECURE_NO_IPTABLES_RAW). On Apple's kernel those workarounds would not be needed. Nested Podman is publicly demonstrated (podman#27278); dockerd is a strong inference from the same kernel facilities plus the machine's systemd-service model.

Arbitrary live mounts — full virtiofs parity (confirmed in source). container run --volume /any/abs/path:/dst (or --mount source=...,target=...) live-mounts any absolute host directory — no $HOME restriction (verified in Parser.swift/validateMount; a unit test even covers ..). Mounts are bidirectional virtiofs, changes visible immediately on both sides — matching Lima's live-mount semantics. Only gotcha: the host path must exist and be absolute (no auto-create). The $HOME → /Users/<user> machine mount is a separate convenience, not a constraint.

SSH is solvable and preserves our abstraction. Each container/machine gets a host-reachable private IP (192.168.64.x via vmnet on macOS 26), and Apple's own machine example image already installs openssh-server. Baking sshd into the guest image (the same pattern as the Lima golden image) and reaching it over the private IP keeps the entire SshTarget/Backend layer intact. Note: --publish loopback forwarding is buggy on macOS 26.1–26.3 (#919, #1321) — use the direct IP, not -p. A host-side macOS "Local Network" permission gate applies.

Updated verdict

With old-macOS support dropped, this moves from "not worth it" to a genuinely viable opt-in backend — and simpler than Firecracker (no kernel workarounds). Mounts and SSH are no longer blockers. The remaining unknowns all require a real macOS 26 Apple-Silicon host to settle.

Remaining gaps to confirm

• Concurrent multi-machine boot. coop runs several named VMs at once. Apple supports many named machines with a default-selection model, but the docs never state multiple can be booted simultaneously. Confirm: container machine create ... --name coop-a and --name coop-b, then verify both are running concurrently.
• dockerd inside an Ubuntu+systemd machine, end-to-end. Confirm dockerd starts as a systemd service inside a container machine and docker run hello-world works with bridge networking (no iptables workarounds needed, per the kernel config above).
• SSH in over the private IP. Start sshd inside the machine and confirm the host can ssh ubuntu@<machine private IP> — validating the SshTarget integration path.
• Disk sizing / resize. container machine set exposes only cpus, memory, home-mount — no disk knob at create or after. Confirm whether disk is APFS thin-provisioned (making our resize moot) or whether there's a hidden setting. coop's resize command currently has no analog.
• Lifecycle robustness on 1.0.x. container hit 1.0.0 on 2026-06-09 with breaking changes and a steady drip of stop/delete/boot bugs (#1058, #861, #946, #130, --rm-ignored data loss #977). Confirm create → start → stop → destroy is reliable enough to depend on.

Quick spike to cover the first three:

container machine create <ubuntu-systemd-img> --name coop-a --cpus 2 --memory 4G
container machine create <ubuntu-systemd-img> --name coop-b --cpus 2 --memory 4G   # both running?
container machine run -n coop-a -- bash   # install+start dockerd; docker run hello-world; start sshd
ssh ubuntu@<coop-a private IP>            # reachable?

[hbrodin]

Empirical confirmation on macOS 26.5.1 (Apple Silicon)

Installed container 1.0.0 (brew install container) on a real macOS 26.5.1 / arm64 host and ran the spike. All five gaps now have verdicts. A parallel source/issue-tracker fact-check backs the desk-research claims.

Gap 1 — Concurrent multi-machine boot: ✅ CONFIRMED

Created coop-a and coop-b at the same time; container machine list showed both running with distinct IPs (192.168.64.2 / .3). No single-running constraint exists in the CLI or source — set-default only controls whether you can drop -n, not exclusivity.

Gap 2 — dockerd in Ubuntu+systemd, end-to-end: ✅ CONFIRMED, no workarounds needed

Built an ubuntu:24.04 + systemd + docker.io machine image (the documented recipe). PID 1 = systemd; dockerd, containerd, sshd all came up as services. docker run hello-world pulled and ran (Docker 29.1.3 — exercises egress NAT + DNS). Crucially, iptables -t raw -L returned a populated PREROUTING chain — the raw table the Firecracker CI kernel lacks. So neither coop Firecracker workaround is needed (no iptables-legacy, no DOCKER_INSECURE_NO_IPTABLES_RAW). Guest kernel: 6.18.15.

Gap 3 — SSH over the private IP: ✅ CONFIRMED end-to-end

Host pinged the guest, TCP/22 connected, and after baking a pubkey into the image (/etc/skel, the golden-image pattern), ssh hbrodin@192.168.64.9 authenticated and ran commands as the host-matching user. No macOS "Local Network" permission block was hit in this CLI context. The SshTarget/Backend integration path holds.

Gap 4 — Disk sizing / resize: ✅ CONFIRMED, no knob

machine create flags are only --cpus / --memory / --home-mount (plus name/default/no-boot); machine set accepts only cpus / memory / home-mount. No disk flag anywhere. Disk is grow-on-demand EXT4 (the list DISK column grew 75M → 700M+ as content was unpacked). coop's resize has no analog — but it's likely moot.

Gap 5 — Lifecycle robustness on 1.0.x: ✅ mostly resolved

stop / delete were clean and instant in this run. Against the live tracker, of the cited issues only #861 is still OPEN; the rest are CLOSED. Note #977 was mischaracterized above — it's a CLOSED/fixed old-version regression, not a live --rm data-loss bug.

---

New caveats found during the spike (worth folding into the design)

• Machine IPs are not stable across reboots. Observed .6→.7 and .8→.9 drift. coop must re-resolve the IP via container machine inspect on each start, not cache it.
• container machine run exec is flaky and quirky. Intermittent Operation not supported by device during boot races; it word-splits the command (sh -c "echo AAA" runs echo with AAA as $0); and it does not forward piped host stdin. coop should drive the guest over SSH, not machine run (which it does anyway) — but provisioning steps can't rely on machine run to inject files.
• Two corrections to the desk research: "APFS thin-provisioned" → it's EXT4 grow-on-demand images sparse on host APFS (no per-machine APFS volume); and the Gap-5 issue framing (most closed; #977 fixed).

Net

Confirmed viable as an opt-in third backend on macOS 26+, and genuinely simpler than Firecracker — the two kernel workarounds disappear. Remaining real risks are the per-reboot IP churn and general 1.0.0 exec/lifecycle flakiness, both manageable (re-resolve IP on start, drive via SSH, bake provisioning into the image).