Takeaways utilising VFIO in a KVM virtual guest

Having a seamless experience with a host OS running a KVM-guest that gets full access to a GPU by PCI passthrough does come with some hurdles. These are some takeaways from setting it all up.

VFIO is a modern way of passing through PCI-devices from the host to a virtual machine. IOMMU must be enabled for this to work. Both in BIOS and in the kernel. For adding the kernel flags I found it easiest to use grubby for this.

Do note that grubby modifies the current grub.cfg which will be overwritten when grub2-mkconfig is run. That will occur when upgrading a kernel. So this will have to be re-run after a kernel upgrade.

grubby --args="intel_iommu=on" --update-kernel=/boot/vmlinuz-$(uname -r)

KVM guest

• • Install the KVM guest as usual but change bootmode to UEFI. This requires the package edk2-ovmf installed.
  • Then remove every virtual device having to do with display and spice
  • Change USB controller to accomodate usb3
  • Change storage and NIC to use virtio drivers
  • Add PCI hostdevice
    • Select the graphics card, usually 01:00.0 along with its HD audio device 01:00.1. Then by using an HDMI cable you get both display & sound!

/etc/dracut.conf.d/vfio.conf

add_drivers+="vfio vfio_iommu_type1 vfio_pci"

The GPU being passed through must be disabled from being used by the host OS.

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1401,10de:0fba
options vfio-pci disable_vga=1
options snd-hda-intel enable_msi=1

grubby --args="rd.driver.pre=vfio-pci" --update-kernel=/boot/vmlinuz-$(uname -r)
grubby --args="video=efifb:off" --update-kernel=/boot/vmlinuz-$(uname -r)
dracut -f --kver $(uname -r)

Without enabling MSI the sound can begin to stutter when there’s too much data going through the HDMI-cable. To verify if it’s enabled, check for a plus sign in the lspci -v -s 01:00.1|grep MSI: output.

	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+

Keyboard & mouse

Found a software-KVM solution called Barrier (https://github.com/debauchee/barrier) to be most viable.

• • Installed as a client on the VM. It will autostart when having made a connection.
  • Installed as a server on the host OS. Went for letting the window manager autostart it upon login.
    • Tick use relative mouse moves
    • For some games lock-mode will have to be used (scroll-lock)
    • Non-ascii characters can be troublesome. Set meta = altgr and altgr = shift in the screens section.

USB-devices

As for handling USB-devices inside the VM there are some options. I went for udev and wrote a script to handle that. It’s available on https://github.com/mstrahlert/vfio-usb.