0.7.7: guests no longer see vmx / /dev/kvm (nested virtualization regression from 0.7.6) #1778

Summary

With the same host, image and sandbox config, a guest started by the 0.7.6 runtime has vmx in /proc/cpuinfo and a working /dev/kvm. A guest started by the 0.7.7 runtime has neither. Nested KVM that worked in 0.7.6 no longer works.

Environment

• Host: Fedora, kernel 7.2.7-200.fc44.x86_64, Intel CPU (11th Gen Intel(R) Core(TM) i7-11700KF @ 3.60GHz)
• Host has vmx in /proc/cpuinfo and /sys/module/kvm_intel/parameters/nested is Y
• msb 0.7.6 and 0.7.7, linux-x86_64 release tarballs, checksum-verified
• Each version used its own throwaway MSB_HOME

Reproduction

for v in 0.7.6 0.7.7; do
  mkdir -p /tmp/r/$v/bin /tmp/r/$v/lib /tmp/r/home-$v
  curl -fsSL https://github.com/superradcompany/microsandbox/releases/download/v$v/microsandbox-linux-x86_64.tar.gz |
    tar -xz -C /tmp/r/$v/bin msb
  curl -fsSL https://github.com/superradcompany/microsandbox/releases/download/v$v/microsandbox-linux-x86_64.tar.gz |
    tar -xz -C /tmp/r/$v/lib --wildcards 'libkrunfw*'
  ( cd /tmp/r/$v/lib && ln -sf libkrunfw.so.5.6.1 libkrunfw.so.5 && ln -sf libkrunfw.so.5 libkrunfw.so )
  MSB_HOME=/tmp/r/home-$v /tmp/r/$v/bin/msb run --name kvmtest --cpus 2 alpine -- sh -c \
    'uname -r; grep -c -w -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
  MSB_HOME=/tmp/r/home-$v /tmp/r/$v/bin/msb rm -f kvmtest
done

Result

What I narrowed down

The third row is the useful one. I ran the 0.7.7 msb with MSB_LIBKRUNFW_PATH pointing at the 0.7.6 libkrunfw.so.5.6.1. The guest booted the 0.7.6 kernel (6.12.109) and still had no vmx. So the "update libkrunfw to Linux 6.12.111" change (#1740) is not the cause. Something in the 0.7.7 VMM changes what the guest sees.

The embedded VMM crates went from msb_krun_* 0.1.39 to 0.1.40, including msb_krun_cpuid, which configures guest CPUID. I have not confirmed that this is where it changes. The 0.7.7 release notes do not mention a nested-virtualization or CPUID change.

Side note: the two libkrunfw.so.5.6.1 files have the same name and size but different sha256 hashes, so the version string did not change when the kernel build did.

Questions

1. Was hiding vmx from guests intentional in 0.7.7, for example as a hardening change?
2. If not, can it be restored?
3. If it was intentional, is there an option to enable nested virtualization? I found nothing in the 0.7.7 msb create --help.

Impact

Anything that runs a hypervisor or microVM inside a sandbox stops working, for example running msb itself for testing, or Firecracker or QEMU/KVM guests. The only workaround I know of is pinning the host to 0.7.6.