-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathnextbsd-apple-virtualization-plan.html
More file actions
319 lines (255 loc) · 43 KB
/
Copy pathnextbsd-apple-virtualization-plan.html
File metadata and controls
319 lines (255 loc) · 43 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>NextBSD under Apple Virtualization.framework — boot, console, and the fbdev gap</title>
<style>
:root { --fg:#1a1a1a; --fg-muted:#555; --bg:#fafaf7; --accent:#b8472a; --accent-soft:#f3e7df; --border:#d8d4c8; --code-bg:#f0ece2; --table-stripe:#f4efe5; --warn:#8a5a00; --good:#2d6f3b; --bad:#a23030; }
* { box-sizing:border-box; }
body { font-family:-apple-system,BlinkMacSystemFont,"Helvetica Neue",Helvetica,sans-serif; color:var(--fg); background:var(--bg); line-height:1.55; margin:0; padding:0; }
.wrap { max-width:880px; margin:0 auto; padding:48px 32px 96px; }
h1 { font-size:2.05rem; line-height:1.2; margin:0 0 8px; letter-spacing:-0.01em; }
h2 { font-size:1.4rem; margin:54px 0 12px; padding-top:18px; border-top:2px solid var(--border); }
h3 { font-size:1.12rem; margin:30px 0 10px; color:var(--accent); }
h4 { font-size:0.98rem; margin:20px 0 8px; }
p { margin:0 0 14px; }
ul,ol { margin:0 0 14px 22px; padding:0; } li { margin:0 0 6px; }
code { font-family:"SF Mono",Menlo,Consolas,monospace; font-size:0.9em; background:var(--code-bg); padding:1px 5px; border-radius:3px; }
pre { font-family:"SF Mono",Menlo,Consolas,monospace; font-size:0.84em; line-height:1.5; background:var(--code-bg); border:1px solid var(--border); border-radius:4px; padding:14px 16px; overflow-x:auto; margin:0 0 14px; }
pre code { background:none; padding:0; }
.lede { font-size:1rem; color:var(--fg-muted); margin:0 0 28px; }
.meta { font-size:0.85rem; color:var(--fg-muted); margin:0 0 24px; }
table { border-collapse:collapse; width:100%; margin:12px 0 22px; font-size:0.92rem; }
th,td { text-align:left; padding:9px 12px; border:1px solid var(--border); vertical-align:top; }
th { background:var(--accent-soft); font-weight:600; }
tr:nth-child(even) td { background:var(--table-stripe); }
.scroll { overflow-x:auto; }
.callout { border-left:3px solid var(--accent); background:var(--accent-soft); padding:14px 18px; margin:18px 0 22px; border-radius:0 4px 4px 0; }
.callout p:last-child { margin-bottom:0; }
.callout-warn { border-left-color:var(--warn); background:#fbf3df; }
.callout-good { border-left-color:var(--good); background:#e8f1e3; }
.callout-bad { border-left-color:var(--bad); background:#f7e6e6; }
.pill { display:inline-block; font-size:0.78rem; font-weight:600; text-transform:uppercase; letter-spacing:0.04em; padding:2px 8px; border-radius:10px; margin-right:8px; }
.pill-good { background:#d6ead0; color:var(--good); } .pill-warn { background:#f4dfbf; color:var(--warn); } .pill-bad { background:#f0c8c8; color:var(--bad); } .pill-neutral { background:#ddd; color:#333; }
.toc { background:white; border:1px solid var(--border); border-radius:4px; padding:18px 24px 14px 36px; margin:0 0 36px; font-size:0.95rem; }
.toc h2 { margin:0 0 8px; padding-top:0; border-top:none; font-size:1rem; text-transform:uppercase; letter-spacing:0.04em; color:var(--fg-muted); margin-left:-14px; }
.toc ol { margin:0 0 0 6px; } .toc li { margin-bottom:4px; }
.toc a { color:var(--fg); text-decoration:none; } .toc a:hover { text-decoration:underline; }
.back { font-size:0.9rem; margin-bottom:18px; } .back a { color:var(--accent); text-decoration:none; }
.footnote { font-size:0.85rem; color:var(--fg-muted); border-top:1px solid var(--border); margin-top:48px; padding-top:16px; }
.cite { font-size:0.82em; color:var(--fg-muted); }
.opt { border:1px solid var(--border); border-radius:4px; padding:2px 22px 6px; margin:0 0 26px; background:white; }
.opt h3 { margin-top:22px; }
.phase { border-left:4px solid var(--accent); padding-left:18px; margin:0 0 30px; }
.verdict { font-weight:600; }
</style>
</head>
<body>
<div class="wrap">
<p class="back"><a href="index.html">← Back</a> · part of <a href="nextbsd-research.html">NextBSD Research</a> · sibling of the <a href="nextbsd-virtio-gpu-plan.html">virtio-gpu DRM plan</a> · decision record: <a href="nextbsd-arm64-kms-options.html">arm64 KMS options</a> · console policy: <a href="nextbsd-quiet-console-plan.html">quiet console plan</a></p>
<h1>NextBSD under Apple Virtualization.framework — boot, console, and the fbdev gap</h1>
<p class="lede">Apple’s VM stack is the one arm64 target where NextBSD has never booted, and the reason turned out not to be ACPI, GIC, or graphics. It was USB. This page records what was measured on real hardware, what the remaining blockers are, who else is already working on them, and the order to fix them in.</p>
<p class="meta">Investigated 2026-08-18, updated 2026-08-19 on an Apple Silicon Mac, macOS 26.5.2. Every claim tagged <em>measured</em> was observed directly on that machine against a NextBSD arm64 install (kernel <code>20260818-174943</code>) and the <code>NextBSD-arm64-20260817-065847</code> ISO. Upstream claims carry source links. Claims from binary analysis of Apple’s framework are tagged as such and are the least durable thing here.</p>
<div class="callout callout-good">
<p><span class="pill pill-good">Root-caused 2026-08-18</span> <strong>The “Internal Virtualization error” is xHCI.</strong> Apple’s virtual xHCI controller aborts the entire VMM — <code>Security Assertion Hit</code> → <code>__os_crash</code> → <code>brk #0x1</code> on the vCPU thread — when a guest writes a register it does not model, and FreeBSD’s <code>xhci(4)</code> does exactly that. Any <code>VZUSBKeyboardConfiguration</code>, <code>VZUSBScreenCoordinatePointingDeviceConfiguration</code>, or <code>VZUSBMassStorageDevice</code> instantiates that controller. Remove them and the crash goes away: <strong>6–8 s crash → 121 s and still running, guest idle at 0.1 % CPU</strong>.</p>
</div>
<div class="callout callout-warn">
<p><span class="pill pill-warn">Still blocked</span> <strong>Booting is not fixed, only unblocked.</strong> With USB removed the guest reaches the kernel and then sits idle without networking. It is mute — no GOP, no UART, no SPCR — so the next blocker cannot be identified without first giving the guest a console. That, not ACPI, is the critical path.</p>
</div>
<div class="callout callout-good">
<p><span class="pill pill-good">Updated 2026-08-19</span> <strong>Two of the three blockers on this page are now closed in tree, and neither of them was the boot hang.</strong> The <code>vtgpu</code> attach spin (§4 #4) is fixed by kernel patch <code>0011</code>, which bounds the control-queue wait and calls <code>virtio_stop()</code> on timeout instead of spinning a vCPU forever. The fbdev gap (§6) is closed outright: <code>drm_fbdev_shmem</code> is built and shipping, and a stock arm64 boot now reaches <code>/dev/dri/card0</code> with a drmfb console unattended. Neither depends on a firmware framebuffer, so both carry over to VZ, where there is no GOP at all.</p>
<p><strong>What is left for VZ is exactly one thing: the early hang.</strong> With USB removed the guest reaches the kernel and goes idle before the virtio bus is probed — earlier than any console this project can register, which is why the consdev (§8 Phase 1) is resident, correct, and silent. The next move is the <strong>PSCI tracer</strong> (§8 Phase 2): it needs no console at all. Everything else on this page is either measured-and-eliminated or now shipped.</p>
</div>
<div class="toc">
<h2>Contents</h2>
<ol>
<li><a href="#measured">What was measured</a></li>
<li><a href="#xhci">The xHCI fatality</a></li>
<li><a href="#mute">Why the guest is mute — three routes to a console</a></li>
<li><a href="#blockers">Remaining boot blockers, ranked</a></li>
<li><a href="#prior">Prior art — we are not first</a></li>
<li><a href="#fbdev">The fbdev gap — smaller than we thought</a></li>
<li><a href="#reconcile">Reconciling with V7 and Option D</a></li>
<li><a href="#phases">Plan</a></li>
<li><a href="#calls">Open calls</a></li>
</ol>
</div>
<h2 id="measured">1. What was measured</h2>
<p>A NextBSD arm64 install was copied out of its UTM qcow2 to a sparse raw image and booted under Virtualization.framework through a ~90-line Swift harness, ad-hoc signed with <code>com.apple.security.virtualization</code>. Everything below is first-hand.</p>
<div class="scroll"><table>
<tr><th>Question</th><th>Result</th><th>How it was established</th></tr>
<tr><td>Does Apple’s EFI read a FreeBSD disk?</td><td><strong>Yes</strong></td><td><code>lsdev</code> at the loader prompt: <code>disk0: ISO9660</code> / GPT+ESP raw image; kernel and <code>mfsroot</code> load normally</td></tr>
<tr><td>Device tree?</td><td><strong>None</strong></td><td>Loader: <code>No valid device tree blob found!</code> — a warning, not a failure; <code>bootinfo.c</code> prints it whenever <code>acpi.revision</code> is unset</td></tr>
<tr><td>ACPI?</td><td><strong>Yes</strong></td><td><code>show acpi.rsdp</code> → <code>0x00000000efbf0018</code>; <code>show acpi.oem</code> → <code>APPLE</code></td></tr>
<tr><td>GOP / framebuffer?</td><td><strong>None at all</strong></td><td><code>gop list</code> → <code>gop: Graphics Output Protocol not present</code>. Not Blt-only — absent</td></tr>
<tr><td>Kernel output after ExitBootServices?</td><td><strong>Zero, ever</strong></td><td>Screen freezes mid-word on the loader’s last line; nothing follows on any configuration</td></tr>
<tr><td>With USB devices attached</td><td><strong>VMM aborts in 6–8 s</strong></td><td><code>VZErrorDomain Code=1</code>; <code>EXC_BREAKPOINT</code>/<code>SIGTRAP</code> on <code>com.apple.virtualization.thread.cpu-0</code></td></tr>
<tr><td>With no USB devices</td><td><strong>Survives; idle</strong></td><td>121 s+ alive, VMM at 0.1 % CPU, no DHCP lease</td></tr>
<tr><td>Stock kernel (with <code>virtio_gpu</code>)</td><td><strong>Hangs, spinning</strong></td><td>VM alive but pegged ~199 % across 2 vCPUs — a different failure from the idle case</td></tr>
<tr><td>Does <code>virtio_gpu(4)</code> drive Apple’s virtio-gpu?</td><td><strong>No evidence it does</strong></td><td>A kernel containing it produced no console under VZ</td></tr>
<tr><td><strong>Linux under <code>VZEFIBootLoader</code></strong> (GRUB on the guest ESP)</td><td><strong>Boots, console works</strong></td><td>ACPI path confirmed (<code>rtc-efi</code>, not <code>pl031</code>). Tables present: <code>APIC DSDT FACP GTDT MCFG</code> — <strong>no SPCR</strong></td></tr>
<tr><td>MADT GIC Distributor version</td><td><strong><code>GicVersion = 3</code></strong></td><td>Not 0. The carried GIC-version fallback patch is inert here and was dropped. GICD base <code>0x10000000</code>, GICv2m MSI frame <code>0x1fff0000</code></td></tr>
<tr><td>GTDT / FADT</td><td><strong>Well-formed</strong></td><td>VirtualTimer <strong>INTID 27</strong> (what Linux uses), NonSecureEL1 30, EL2 26; FADT rev 6 with <code>HW_REDUCED_ACPI</code>, <code>PSCI_COMPLIANT</code> and <code>PSCI_USE_HVC</code> all set</td></tr>
<tr><td>Linux via <code>VZLinuxBootLoader</code> (direct kernel)</td><td><strong>Boots, DT path</strong></td><td>VZ’s own DTB captured: <code>arm,gic-v3</code>, <code>psci</code> method <code>hvc</code>, RAM at <code>0x70000000</code>, PL031 <code>0x20050000</code>, PL061 <code>0x20060000</code>, <strong>no serial node</strong></td></tr>
<tr><td><strong>NextBSD + VZ graphics device</strong></td><td><strong>199 % spin</strong></td><td>Every kernel containing base <code>virtio_gpu(4)</code> spins; the one built <code>nodevice virtio_gpu</code> does not</td></tr>
<tr><td><strong>NextBSD, no graphics device</strong></td><td><strong>Idle 0.1 %</strong></td><td>No console output even with the consdev patch resident — so it stops <em>before</em> virtio device probe. Cause unknown</td></tr>
<tr><td><strong>Stock FreeBSD 16.0-CURRENT VM image</strong> (<code>-ufs.raw</code>, 2026-08-04 snapshot)</td><td><strong>Same 199 % spin, no lease</strong></td><td>Loader menu renders normally under Apple’s EFI, <code>Loading kernel…</code>, then two vCPUs pegged for 151 s+ with no DHCP. <strong>Not a NextBSD bug</strong></td></tr>
<tr><td>Stock FreeBSD <code>disc1.iso</code> as VZ media</td><td><strong>Not bootable</strong></td><td>No loader output, then a clean power-off. macOS also refuses that ESP (<code>mount -t msdos</code> → error 71) while NextBSD’s mounts as FAT12 <code>EFISYS</code>. Two independent readers failing on the same image points at the media, not the kernel</td></tr>
</table></div>
<div class="callout callout-warn">
<p><span class="pill pill-warn">Media caveat</span> <strong>Test FreeBSD releases with a VM image or memstick, never <code>disc1.iso</code>.</strong> An install ISO that Apple’s EFI cannot read produces exactly the same outward signature as a kernel that fails to boot — black screen, no output — so it is an easy way to draw a wrong conclusion. This may be what <a href="https://github.com/crc-org/vfkit/issues/446">vfkit #446</a> actually is, reported there as “missing device tree”.</p>
</div>
<p>The crash/hang split is diagnostic, not cosmetic: <strong>a crash means the guest touched a device model Apple refuses; an idle guest means the kernel stopped scheduling; a spin means it is looping.</strong> The triggered thread name in <code>~/Library/Logs/DiagnosticReports/com.apple.Virtualization.VirtualMachine-*.ips</code> discriminates further — <code>thread.cpu-N</code> is a guest-triggered device assertion, <code>virtual-machine-service</code> is a configuration the VMM refused at start.</p>
<h2 id="xhci">2. The xHCI fatality</h2>
<p>The crash report’s faulting function, disassembled, references only xHCI MMIO-decode strings:</p>
<pre><code>"Security Assertion Hit: "
"Guest attempted to write into RsvdP in the Host Controller Runtime Registers's
Interrupter Register Set \"%u\". "
"Guest attempted to write into a Host Controller Operational Registers' port
register at offset \"0x%x\". The offset is past MaxPorts of this device."
"Unhandled endpoint type %d" "Non-zero doorbell target %u is unexpected"</code></pre>
<p>The call site is <code>adrp x1, "Report an error"; bl …</code>, and that callee ends in <code>adrp x0, "assertion failed"; bl __os_crash; brk #0x1</code>. This is a deliberate design choice: an unmodelled guest write kills the VMM rather than being ignored.</p>
<p>It also matches the only known-good FreeBSD-on-VZ writeup independently, whose blocker #4 reads verbatim <em>“USB/XHCI is fatal — any USB device makes VZ present an XHCI controller and FreeBSD’s USB stack dies”</em>.</p>
<div class="callout">
<p><strong>Two independent belts.</strong> Host side: never configure keyboards, pointing devices, USB mass storage, or an explicit <code>VZXHCIControllerConfiguration</code>. Guest side: <code>nodevice xhci</code> in KERNCONF, or <code>hint.xhci.0.disabled="1"</code>. The guest-side belt is the one that matters, because it protects users running third-party VZ front-ends we do not control — and every published VZ tool configures a keyboard by default.</p>
</div>
<p class="cite">Sources: local crash reports 2026-08-18; <a href="https://github.com/networkextension/FreeBSD-vz">networkextension/FreeBSD-vz</a>.</p>
<h2 id="mute">3. Why the guest is mute — three routes to a console</h2>
<p>VZ gives an arm64 guest no console path that FreeBSD can currently use. This is a platform fact, not a bug in our configuration:</p>
<ul>
<li><strong>No GOP</strong> — measured. So <code>vt_efifb</code> can never attach, and there is no <code>MODINFOMD_EFI_FB</code> to hand over. The <a href="nextbsd-fallback-graphics-plan.html">fallback graphics</a> route does not exist on this platform.</li>
<li><strong>No SPCR, no DBG2, no UART in the DSDT</strong> — a real VZ Linux guest shows exactly five tables (<code>RSDP/XSDT/FACP/DSDT/GTDT/APIC/MCFG</code>, OEM <code>APPLE</code>, creator <code>Apple Vz</code>). SPCR is generated per configured device, and the public API configures none that qualify. Note FreeBSD ignores DBG2 anyway unless <code>hw.acpi.enable_dbg2=yes</code>, and SPCR structurally cannot describe a virtio console — its Interface Type byte defers to DBG2 serial subtypes, all of which are UART variants.</li>
<li><strong>virtio-console is tty-only in FreeBSD</strong> — <code>sys/dev/virtio/console/virtio_console.c</code> creates <code>/dev/ttyV0.x</code> but registers no <code>consdev</code>/<code>cnputc</code>, so it cannot carry <code>printf</code> from early boot. Linux uses the same device as <code>hvc0</code>, which is why every VZ tool’s “serial output” works for Linux guests and not for us — including <a href="https://github.com/pkgdemon/swift-vm-cli">swift-vm-cli</a>, which wires <code>VZVirtioConsoleDeviceSerialPortConfiguration</code> to stdio in raw mode.</li>
</ul>
<div class="opt">
<h3>Route A — port the virtio-console <code>consdev</code></h3>
<p><span class="pill pill-good">Recommended</span> <span class="verdict">~250 LOC, already written by someone else, no entitlements, ships to users.</span></p>
<p>The FreeBSD-vz fork carries a low-level console for <code>virtio_console(4)</code> (<code>cf50f191e</code>, +252 lines). This is the only console VZ hands an <em>unentitled</em> harness, it needs no private API, and it is the piece that makes NextBSD debuggable on this platform for good. It also benefits <code>std.virt</code> generally — the one-line <code>device virtio_console</code> is worth adding regardless of what else lands.</p>
<p>Reconcile with <a href="nextbsd-driver-delivery-plan.html">driver delivery</a> §3: <code>console (vt)</code> sits in the compiled-in tier. A console-capable <code>virtio_console</code> must join it there, or take an explicit documented exception.</p>
</div>
<div class="opt">
<h3>Route B — Apple’s private PL011</h3>
<p><span class="pill pill-warn">Debug harness only</span> <span class="verdict">Zero kernel LOC — but entitlement-gated, and unusable by end users.</span></p>
<p>Binary analysis of <code>Virtualization.framework</code> shows a private <code>_VZPL011SerialPortConfiguration</code> (serial port type 1) beside the public virtio console (type 2), plus a device table giving four slots at <code>0x20010000 + n*0x10000</code>, GSIV <code>32+n</code>. QEMU’s clean-room <code>vmapple</code> machine model places a PL011 at exactly <code>0x20010000</code>, which corroborates it. FreeBSD would then need only:</p>
<pre><code>hw.uart.console="mm:0x20010000,br:115200,rs:2,rw:4,xo:0"</code></pre>
<p><strong>Measured caveat:</strong> the class passes <code>validate()</code> but <code>start()</code> fails with <code>Code=1</code> under an ad-hoc signature, trapping on <code>virtual-machine-service</code> — the <code>com.apple.private.virtualization</code> gate. Unlocking it means weakening SIP/AMFI. Fine for a lab, never for a shipped tool. Treat as a debugging accelerator, not a solution.</p>
</div>
<div class="opt">
<h3>Route C — the virtio-gpu KMS console</h3>
<p><span class="pill pill-neutral">Downstream</span> <span class="verdict">The eventual product, not a debugging tool.</span></p>
<p>Since VZ has no GOP, the <em>only</em> way a VZ guest ever gets a graphical console is a virtio-gpu driver that publishes one — which is the <a href="nextbsd-virtio-gpu-plan.html">virtio-gpu plan</a> plus the fbdev work in §6. It cannot help diagnose early boot (it needs a fully running kernel), so it is sequenced last.</p>
</div>
<h2 id="blockers">4. Remaining boot blockers, ranked</h2>
<div class="callout callout-good">
<p><span class="pill pill-good">Settled 2026-08-18</span> <strong>Four of these six are now eliminated by measurement</strong> (§1): the MADT reports <code>GicVersion = 3</code>, there is no SPCR to be malformed, the GTDT virtual timer is INTID 27, and the FADT sets both PSCI flags. VZ’s firmware description is well formed — the blocker is not in the tables.</p>
</div>
<div class="callout callout-warn">
<p><span class="pill pill-warn">Two failures, not one</span> <strong>#4 was real and is now fixed.</strong> Base <code>virtio_gpu(4)</code> attaching to Apple’s virtio-gpu pegged both vCPUs, because <code>vtgpu_attach()</code> waited on the control queue with an unbounded <code>virtqueue_poll(9)</code>. Kernel patch <code>0011</code> replaces that with a 5 s bounded dequeue loop that reports and calls <code>virtio_stop()</code> rather than spinning. The spin is gone without removing the graphics device from the VM.</p>
<p>What remains underneath is the <em>second</em>, earlier failure, and it is now the only one: the guest goes idle at 0.1 % with no console output at all, which places it <strong>before virtio device probe</strong> — early enough that no device-attach-time console, the consdev included, can ever report it.</p>
</div>
<p>With xHCI removed, the guest reaches the kernel and goes idle. These are the candidates, ranked by how well they fit that signature, each with the cheapest test. Status reflects what has since been measured.</p>
<div class="scroll"><table>
<tr><th>#</th><th>Hypothesis</th><th>Fit</th><th>Cheapest test / known fix</th></tr>
<tr><td>1</td><td><strong>MADT GICD <code>Version == 0</code></strong> → <code>gic_v3_acpi_identify()</code> bails → <code>panic: No usable event timer found</code></td><td>High — explains an idle guest with no output</td><td>Dump <code>APIC</code> from a Linux VZ guest and read the version byte. Fix exists: read <code>GICD_PIDR2</code> as Linux does — <code>aabce0c83</code> on <code>networkextension:apple-vz-gic-version-none</code>, not upstream</td></tr>
<tr><td>2</td><td><strong>Zeroed/bad SPCR geometry</strong> (<code>rs:0,rw:1</code> vs <code>rs:2,rw:4</code>)</td><td>High</td><td>FreeBSD bug 282936 is the same signature; fixed upstream by <code>dea3eef94caf</code> (D47946), <strong>not in releng/14.3</strong>. Kernel-side <code>SerialPort.Address == 0</code> guard analogous to <code>d82698ac68c2</code> is still unfiled</td></tr>
<tr><td>3</td><td><strong>virtio feature negotiation</strong> — VZ is strict/all-or-nothing</td><td>High</td><td>Probe with <code>hw.vtnet.{csum,tso,lro,mq}_disable="1"</code>; then cherry-pick <code>f9a4dd60b0cf</code>, <code>8bdc312dac55</code> (<code>VIRTIO_F_RING_RESET</code>), <code>d4ddf8eaee93</code>, <code>5481c2d3ac1f</code> from <code>main</code> — <strong>none are MFC’d to stable/15</strong></td></tr>
<tr><td>4</td><td><strong><code>vtgpu</code> hangs at attach</strong> <span class="pill pill-good">fixed</span></td><td>Was medium-high — it was the 199 % spin</td><td><strong>Fixed in tree by patch <code>0011</code></strong>: bounded control-queue wait (5 s, 10 µs interval) then <code>device_printf</code> + <code>virtio_stop()</code> + <code>ETIMEDOUT</code>. No <code>nodevice</code> needed, and the fix helps qemu guests too</td></tr>
<tr><td>5</td><td><strong>GTDT virtual-timer interrupt wrong</strong> — an EL1 guest uses <code>GT_VIRT</code> only; vmapple wires only INTID 27</td><td>Medium</td><td>Same ACPI dump, <code>iasl -d GTDT</code></td></tr>
<tr><td>6</td><td><strong>PSCI conduit resolved to SMC</strong> — <code>psci_acpi_get_callfn()</code> reads FADT <code>ArmBootFlags</code> with no probe or fallback; an SMC faults on a guest with no EL3</td><td>Low — VZ’s DT says <code>hvc</code></td><td><code>iasl -d FACP</code>, check bits 0/1</td></tr>
</table></div>
<div class="callout callout-warn">
<p><span class="pill pill-warn">Do this first</span> <strong>One Linux guest settles four hypotheses at once.</strong> Boot arm64 Linux under <code>VZEFIBootLoader</code> on the same host and copy out <code>/sys/firmware/acpi/tables/{APIC,GTDT,FACP,SPCR,MCFG,DSDT}</code>, then <code>iasl -d</code> them. That answers #1, #2, #5 and #6 with no NextBSD changes at all. A second, direct-kernel boot (<code>VZLinuxBootLoader</code>) yields VZ’s own DTB via <code>dtc -I fs -O dts /sys/firmware/devicetree/base</code> — which could be handed to <code>loader.efi</code> in <code>/boot/dtb/</code> and selected with <code>kern.cfg.order=fdt</code>, bypassing the ACPI path entirely as a bring-up shortcut.</p>
</div>
<p>Note also that <code>bus_probe()</code> in <code>sys/arm64/arm64/machdep.c</code> has <strong>no health check and no fallback</strong> — the presence of an RSDP alone wins, even if the tables that follow are unusable. And FreeBSD detects hypervisors only via SMBIOS, so under VZ <code>vm_guest</code> is unset and no guest quirks apply (bug 292082 / D54595 proposes <code>boot_el</code>-based detection).</p>
<div class="callout callout-bad">
<p><span class="pill pill-bad">Measured 2026-08-18</span> <strong>Upstream <code>main</code> does not boot under VZ either.</strong> The stock FreeBSD 16.0-CURRENT arm64 VM image — which already carries the four merged virtio fixes and ACPI-by-default — loads its kernel and then spins two vCPUs indefinitely with no networking, exactly as NextBSD’s stock kernel does. This removes NextBSD from the suspect list entirely and makes the out-of-tree patch set in §5 a <strong>hard dependency</strong>, not an optimisation. It also means a VZ boot cannot be fixed by rebasing.</p>
</div>
<h2 id="prior">5. Prior art — we are not first</h2>
<div class="scroll"><table>
<tr><th>Who</th><th>Status</th><th>What it gives us</th></tr>
<tr><td><a href="https://github.com/networkextension/FreeBSD-vz">networkextension/FreeBSD-vz</a></td><td><strong>FreeBSD 16.0-CURRENT boots to multiuser under VZ</strong>, headless, 2026-07-06</td><td>The proof it is possible, a <code>VZ</code> KERNCONF, the GIC-version patch, and the virtio-console <code>consdev</code> port. Their dmesg shows <code>acpi0: <APPLE Apple Vz></code>, <code>gic0 … v3.0</code>, <code>vtnet0/vtcon0/vtblk0/vtrnd0</code></td></tr>
<tr><td><a href="https://github.com/tjfontaine/freebsd-apple-virtualization">tjfontaine/freebsd-apple-virtualization</a></td><td>Publishes dated evidence runs</td><td>Raw ACPI blobs — potentially skips our §4 dump entirely</td></tr>
<tr><td>Faraz Vahedi, freebsd-src PRs 2319 / 2322</td><td><strong>Merged to <code>main</code></strong>, 2026-07</td><td>The four virtio commits VZ needs. Not MFC’d — if NextBSD tracks releng/15.0 we must carry them</td></tr>
<tr><td>OpenBSD 7.9, 2026-05-19</td><td><em>“Made OpenBSD work on Apple Virtualization.”</em></td><td>Every blocker they hit was <strong>virtio feature negotiation</strong>, not boot: <code>VIRTIO_NET_F_MTU</code> (VZ refuses to work without it), TSO-without-<code>MRG_RXBUF</code>, and a <code>viogpu</code> mmap returning a KVA where a physical address was required. Also the architectural idea worth stealing: their bootloader <strong>synthesizes an FDT from ACPI</strong> (<code>efiacpi.c</code>) so the kernel only ever sees FDT</td></tr>
<tr><td>Apple, <a href="https://github.com/apple/containerization">containerization</a></td><td>Reference guest config</td><td>The authoritative device list: GICv3, PSCI, arch timer, PCI-generic, virtio-PCI, PL011, PL061, PL031, EFI stub. Console is <code>hvc0</code>; shutdown is PL061 GPIO → <code>gpio-keys</code>, not ACPI PWRBTN</td></tr>
<tr><td>UTM / vfkit</td><td>Explicitly unsupported</td><td>UTM documents macOS and Linux only; vfkit issue 446 has FreeBSD 15.0 failing with “missing device tree”. Nobody ships this — being early is the point</td></tr>
</table></div>
<h3>The platform, as measured by others</h3>
<p>Worth recording because it differs from QEMU’s <code>virt</code> in ways that break assumptions: <strong>RAM is at <code>0x7000_0000</code>, not <code>0x4000_0000</code></strong> (PCIe ECAM sits there instead); GICv3 only, <strong>no ITS and no MSI/MSI-X</strong> — PCIe is legacy INTx on SPIs; PSCI 0.2 with HVC conduit and an EL1-only guest, so SMC faults; virtio-PCI modern only, no virtio-mmio; 4K or 16K granule, never 64K. Our own <code>acpi.rsdp = 0xefbf0018</code> is consistent with the <code>0x70000000</code> RAM base and inconsistent with a QEMU-virt layout — a useful independent confirmation.</p>
<h2 id="fbdev">6. The fbdev gap — closed</h2>
<div class="callout callout-good">
<p><span class="pill pill-good">Shipped 2026-08-19</span> <strong>This section is now history rather than plan.</strong> <code>drm_fbdev_shmem.c</code> is vendored into <code>graphics/drivers/gpu/drm/</code> with four <code>#ifdef __linux__</code> guards, built by <code>drm_shmem_helpers</code>, and called by default from the virtio-gpu kext (<code>hw.virtio_gpu_drm.fbdev=0</code> opts out) — kernel-modules#38. The prediction below held: <strong>no deferred I/O was required</strong>. One further fix was needed that this section did not anticipate — <code>lkpinew_pci_dev()</code> created a <code>pci_dev</code> with no DMA tags, so the first <code>dma_map_sgtable()</code> page-faulted; kernel patch <code>0012</code> gives it tags via <code>linux_pdev_dma_init()</code>. With both in place: <code>card0</code> + <code>renderD128</code>, <code>VT: Replacing driver "virtio_gpu" with new "drmfb"</code>, and zero alignment warnings or <code>[drm] *ERROR*</code> lines.</p>
</div>
<p>This closes the open question left dangling in the <a href="nextbsd-graphics-plan.html">graphics plan</a> §10 (<em>“Did drm-kmod’s core build with <code>CONFIG_DRM_FBDEV_EMULATION</code>?”</em>) and corrects the rationale currently written into <code>graphics/drm_shmem_helpers/Makefile</code>.</p>
<p>Enabling <code>drm_fbdev_shmem.c</code> in CI produced exactly <strong>8 errors, no missing headers, no incomplete types</strong>: <code>fb_deferred_io_init</code>/<code>_cleanup</code> undeclared, <code>fb_WARN_ON_ONCE</code> undeclared, and <code>fbdefio</code> missing from <code>struct drm_fb_helper</code> and <code>struct linux_fb_info</code>.</p>
<div class="callout callout-good">
<p><span class="pill pill-good">Corrected</span> <strong>Deferred I/O is not needed at all.</strong> The Makefile’s note that “wiring deferred I/O into drm-kmod’s fb_helper is its own piece of work” is wrong. <code>drm_fbdev_ttm.c</code> — which drm-kmod <em>does</em> build, and which is why BochsGraphics has a console — makes the identical calls and simply wraps them in <code>#ifdef __linux__</code>. Everything in Linux’s <code>fb_defio.c</code> exists to service a userspace <code>mmap()</code> of <code>/dev/fbN</code>, and drm-kmod creates no such cdev.</p>
</div>
<p>The console’s damage path is separate and already implemented and exported: <code>vt_drmfb</code> draws only through <code>info->fbops->fb_imageblit</code> → <code>drm_fb_helper_damage_area()</code> → <code>fb_dirty</code> → our <code>drm_atomic_helper_dirtyfb</code> → <code>virtio_gpu_cmd_transfer_to_host_2d()</code> + <code>resource_flush</code>. vt(4) already batches at 25 Hz (<code>VT_TIMERFREQ</code>) — the same order as Linux’s <code>HZ/20</code> defio delay, with exact damage clips rather than full-screen blasts.</p>
<p>So the fix is <strong>~20–40 LOC of stubs</strong>, with upstream precedent: drm-kmod commit <code>2ac98d2787</code> (wulf@, 2026-06-14, on <code>master</code> but not yet <code>6.12-lts</code>) adds the empty <code>fb_deferred_io_cleanup()</code> and drops the <code>__linux__</code> guard; Phabricator D57488 moves this code into base <em>keeping</em> the stubs. Implementing real deferred I/O would fight the direction of travel and needs VM plumbing FreeBSD lacks (<code>vm_page</code> has no <code>lru</code>).</p>
<p>Three runtime traps the compiler will not catch: <code>drm_fb_helper_deferred_io</code> is declared but <code>__linux__</code>-guarded out (undefined symbol at load — <code>tools/check-kext-symbols.py</code> catches it); <code>fb_deferred_io_mmap()</code> is a live <code>panic()</code>; and shmem’s <code>fix.smem_start == 0</code> means <code>vt_fb_mmap()</code> would hand out pages nothing ever flushes, so set <code>FB_FLAG_NOMMAP</code>.</p>
<p><strong>Scope note:</strong> this unlocks shmem-backed fbdev only — i.e. virtio-gpu, plus the other shmem consumers (<code>ast</code>, <code>mgag200</code>, <code>cirrus</code>, <code>hyperv</code>) our <code>drm_shmem_helpers</code> module already pays for. Bochs and VBox are unaffected: they are TTM-backed and already work.</p>
<p class="cite">OpenBSD’s <code>viogpu.c</code> is the counter-example worth knowing: it transfers and flushes the <em>whole screen every 10 ms</em> with no damage tracking, from a non-DRM wsdisplay driver. It proves a crude periodic flush is viable, and simultaneously that FreeBSD’s <code>vt_drmfb</code> path is the better of the two.</p>
<h2 id="reconcile">7. Reconciling with V7 and Option D</h2>
<p>The <a href="nextbsd-virtio-gpu-plan.html">virtio-gpu plan</a> carries standing constraint <strong>V7: never add <code>device virtio_gpu</code> to the NEXTBSD kernel config</strong>, and <a href="nextbsd-arm64-kms-options.html">arm64 KMS options</a> §Option D recommends <code>nodevice virtio_gpu</code>, because base’s driver registers at <code>VD_PRIORITY_GENERIC+10</code> against efifb’s <code>+1</code> and displaces the console that works.</p>
<div class="callout">
<p><strong>Both remain correct, with one qualification neither page states: they assume a firmware framebuffer exists.</strong> That holds when <code>QemuRamfbDxe</code> produces a GOP with a real linear framebuffer. It does <em>not</em> hold on a VM whose only display device is the virtio-gpu — there OVMF’s GOP is Blt-only, and under Apple’s EFI there is no GOP at all. On such a machine <code>nodevice virtio_gpu</code> leaves <code>VT: init without driver</code> and a black screen from kernel start until a kext loads. Measured on a UTM guest configured <code>-vga none -device virtio-gpu-pci</code>.</p>
</div>
<div class="callout callout-good">
<p><span class="pill pill-good">Settled 2026-08-19 — the other way</span> <strong>V7 and Option D are withdrawn for virtio-gpu. Base <code>virtio_gpu(4)</code> stays compiled in, and the kext takes the device from it.</strong> The reasoning above is why: on a machine whose only display is the virtio-gpu — which is every VZ guest — base is not a nuisance competing with efifb, it is the <em>only</em> console there is. <code>nodevice</code> buys KMS at the price of a blind boot from the first kernel message until userland loads a kext, panics included. That is the wrong trade on the one platform we cannot currently debug.</p>
</div>
<p>Restated constraint: <em>“never rely on base <code>virtio_gpu(4)</code> as the <strong>final</strong> console, but do keep it as the <strong>first</strong> one, and hand the device over explicitly once the KMS kext is resident.”</em></p>
<p>The handover mechanism, since newbus will not do it for us. It remains true that there is <strong>no re-bid</strong> — <code>bus_generic_driver_added()</code> re-probes only <code>DS_NOTPRESENT</code> children, and <code>releng/15.1</code> has no <code>DF_REBID</code> — so an attached <code>vtgpu0</code> is never displaced by a better driver arriving later. Two pieces close that:</p>
<ul>
<li><strong>kernel#76</strong> teaches the in-kernel IOKit present-scan to look <em>through</em> <code>virtio_pci</code> to a base-shadowed virtio-gpu, exactly as it already looks through <code>vgapci</code> to an unbound <code>drmn</code>. Without it nothing ever asks kextd for the bundle: <code>device_nomatch</code> does not fire for a device base matched perfectly well.</li>
<li><strong>kernel-modules#39</strong> makes the kext take the device itself on load — <code>device_detach()</code> then <code>device_probe_and_attach()</code>, inside one <code>bus_topo_lock()</code> section, on a deferred task. That is <code>DEV_SET_DRIVER</code>’s sequence written in exported functions, and <code>BUS_PROBE_VENDOR</code> (kernel-modules#37) is what wins the resulting bid.</li>
</ul>
<div class="callout callout-bad">
<p><span class="pill pill-bad">Measured 2026-08-19</span> <strong>Atomicity is the whole design, not a detail.</strong> Detaching whichever driver currently owns the console panics the machine seconds later, when the next console write reaches a driver that is gone — <code>vtgpu_detach()</code> calls <code>vt_deallocate()</code> and nothing repaints until a replacement registers. Both <code>devctl detach vtgpu0</code> and <code>devctl clear driver -f virtio_gpu_drm0</code> did exactly that, the latter twice; <code>clear driver</code> is just as fatal because it detaches first and re-attaches only if <code>device_probe()</code> succeeds. The atomic form, <code>devctl set driver -f vtgpu0 virtio_gpu_drm</code>, hands over cleanly. Any in-kernel trigger must do the same and never a bare detach.</p>
</div>
<p>Result, on a stock arm64 boot with nothing typed: base paints the console from <code>VT: initialize with new VT driver "virtio_gpu"</code>, kextd loads the bundle the scan asked for, the kext takes the device, and vt switches to <code>drmfb</code> — console the whole way through, plus <code>card0</code>. This is the shape VZ will inherit for free once it boots at all.</p>
<h2 id="phases">8. Plan</h2>
<div class="phase">
<h3>Phase 0 — stop killing the VM <span class="pill pill-good">done 2026-08-18</span></h3>
<p>Remove USB from the harness; add <code>nodevice xhci</code> / <code>hint.xhci.0.disabled="1"</code> to the guest side so third-party front-ends cannot trip it. Exit criterion met: guest survives >120 s.</p>
</div>
<div class="phase">
<h3>Phase 1 — give the guest a voice <span class="pill pill-good">merged</span></h3>
<p>Done: the consdev landed as patch <code>0010</code> in nextbsd-kernel (arm64 <code>GENERIC</code> already includes <code>std.virt</code>, so no config change was needed), and VZ’s ACPI tables were dumped from a Linux guest booted on the same host. A carried GIC-version patch was landed alongside it and then dropped once the MADT proved it inert.</p>
<p><strong>Exit criterion not yet met.</strong> The consdev is resident and correct (<code>cn_pri = CN_NORMAL</code>, <code>cnadd(9)</code> at attach) but produces nothing, because the guest stops before the virtio bus is probed. A console that registers at device-attach time cannot report a pre-attach failure — a limitation worth recording, since it is the whole premise of Route A.</p>
</div>
<div class="phase">
<h3>Phase 2 — make it boot</h3>
<p><span class="pill pill-warn">the only thing left</span> The table-driven theories are spent and the <code>vtgpu</code> spin is fixed (patch <code>0011</code>), so this is now the whole remaining task. Since the guest is mute before device probe, the one technique that needs no console is a <strong>PSCI tracer</strong>: patch an <code>HVC SYSTEM_OFF</code> (<code>0x84000008</code>) in at a chosen early point and watch for a clean power-off on the host — <code>guestDidStop</code> is observable from the harness — then binary-search the boot path. Roughly log₂(N) CI builds, and it reports through the hypervisor rather than through any guest device, which is precisely why it survives a failure this early.</p>
<p>Note what this no longer needs: no console work, no graphics work, no ACPI work. Those were the plausible-looking prerequisites, and every one has since been either measured away (§1) or shipped (§6, §7). <strong>Exit criterion:</strong> multiuser, DHCP lease, ssh.</p>
</div>
<div class="phase">
<h3>Phase 3 — the console people actually see <span class="pill pill-good">done on qemu 2026-08-19</span></h3>
<p>Landed as kernel-modules#37 (probe priority), #38 (<code>drm_fbdev_shmem</code>), #39 (atomic takeover) and kernel#76 (present-scan look-through), with kernel patch <code>0012</code> supplying the DMA tags the fbdev path needs. Exit criterion met on arm64 qemu, unattended, on a stock boot:</p>
<pre><code>vtgpu0: <VirtIO GPU> on virtio_pci1
VT: initialize with new VT driver "virtio_gpu".
vtgpu0: detached
virtio_gpu_drm0: <VirtIO GPU (DRM/KMS)> on virtio_pci1
[drm] Initialized virtio_gpu 0.1.0 for virtio_pci1 on minor 0
VT: Replacing driver "virtio_gpu" with new "drmfb".
/dev/dri/card0 /dev/dri/renderD128</code></pre>
<p><strong>The VZ half remains untested, for the obvious reason</strong> — it needs Phase 2 first. Nothing in this path depends on a GOP or a firmware framebuffer, so it should carry over unchanged; that expectation is reasoned, not measured, and is flagged as such deliberately.</p>
</div>
<div class="phase">
<h3>Phase 4 — upstream and CI</h3>
<p>Offer the virtio-console <code>consdev</code> and the GIC-version fallback to FreeBSD; add a VZ boot lane once a headless boot is reproducible. Note the boot-test guard in kernel-modules greps <code>kldstat</code>, which cannot see a compiled-in driver — it should read <code>kern.conftxt</code>.</p>
</div>
<h2 id="calls">9. Open calls</h2>
<ol>
<li>Do we carry the four virtio commits locally, or wait for an MFC to stable/15? They are merged to <code>main</code> only.</li>
<li>Does <code>virtio_console</code> join the compiled-in tier in <a href="nextbsd-driver-delivery-plan.html">driver delivery</a> §3, or take a documented exception?</li>
<li>Contact networkextension and tjfontaine before writing code — two parties are already on this platform, and one has it booting. Given that stock <code>main</code> spins (§5), their patch set is the dependency; the question is whether we carry it, or help push it upstream.</li>
<li>Is the OpenBSD trick (synthesize FDT from ACPI in the loader) worth adopting as a bring-up shortcut, given <code>bus_probe()</code> has no fallback when ACPI is present but unusable?</li>
<li>Does the harness belong in <a href="https://github.com/pkgdemon/swift-vm-cli">swift-vm-cli</a> (which already does the VZ + virtio-console half) rather than living as a scratch tool?</li>
<li><s>Restate V7 / Option D per §7, or leave them and add the qualification only here?</s> <strong>Answered 2026-08-19:</strong> both are withdrawn for virtio-gpu — base stays compiled in and the kext takes the device (§7). The peer pages still need editing to match.</li>
</ol>
<p class="footnote">Sub-plan of the <a href="nextbsd-graphics-plan.html">NextBSD graphics plan</a> where it touches the console; a peer of the <a href="nextbsd-virtio-gpu-plan.html">virtio-gpu DRM plan</a> and the <a href="nextbsd-arm64-kms-options.html">arm64 KMS decision record</a>, which it amends in §7. Everything tagged <em>measured</em> was observed on 2026-08-18 (VZ, macOS 26.5.2) or 2026-08-19 (arm64 qemu guest, kernel <code>20260819-184420</code>) against NextBSD arm64; upstream claims are linked to source. Claims derived from disassembling Apple’s framework are marked and are the least durable material here — they describe private, undocumented API and may change without notice.</p>
</div>
</body>
</html>