Create a gaming virtual machine using VFIO PCI passthrough for KVM

This how-to will show you how to configure:

  • Install and configure the KVM hypervisor
  • Patch the kernel and QEMU for better compatibility with graphics card / VGA VFIO passthrough
  • Create and configure a new virtual machine (VM) with real hardware attached to it
  • Configure CPU pinning on the VM for better gaming performance

Build considerations & preparation

QEMU has several PCI passthrough techniques, the newest of which is VFIO. QEMU's normal PCI passthrough leaves much to be desired whereas VFIO takes full advantage of IOMMU, has better device support and prevents multiple access to the same device (you can read more about it in Alex Williamson's presentation here).

That said, VFIO it is relatively new and experimental technology for the purposes of passing through entire VGA cards to virtual machines. While myself and many others have had tremendous success, different hardware can produce different results and getting there may not always be straightforward.

Likely, until Fedora 21 is released you will need to patch and rebuild both the Linux kernel and QEMU; instructions for doing so will be provided in this tutorial. If you are purchasing hardware, it is also strongly recommended that you read over the KVM VGA-passthrough thread on Arch Forums to confirm that your intended hardware configuration has been reported to work by another user.

Personally, the following hardware has worked wonderfully for me:

  • Motherboard: Supermicro X10SAE
  • CPU: Xeon E3-1225v3
  • Audio: Onboard (Intel C220 HD audio) and AMD R9 270X HDMI
  • Video: AMD Radeon R9 270X
  • Network: Intel I210 Gigabit

Anecdotal evidence suggests that for graphics passthrough, nVidia cards seem to fare better than AMD ones. However, success has been had on both sides on a variety of device models dating back several years. I have found that generally, problems are not inherent to the hardware but more a matter of adjusting you software stack (i.e. applying certain patches) to get a compatible passthrough. From my reading nVidia's GeForce 6xx/7xx and AMD's Radeon R9 series seem to work fairly painlessly.

For network cards, always pass through an Intel ethernet controller over a Realtek one if possible.

Common problems with VGA VFIO passthrough

Before getting to the fun part, there are several key pieces to getting a functioning VGA passthrough that need further description. Understanding these issues will be key in creating a functioning host environment for VFIO VGA passthrough.

PCIe device reset

In order to re-initialize a PCIe device, it needs to be reset. Normally the host controls this, however now that we are passing through the device to a VM, some additional work is required to get reset functioning correctly. Without this extra PCIe reset support, the machine typically freezes when starting your VM for a second time.

Fortunately for us, kernel >= 3.12 has this support and simply upgrading the kernel fixes the issue.

VGA arbiter and multiple GPUs

Passing through generic PCI devices with VFIO works pretty well. Graphics card passthrough gets put into its own category called "VGA passthrough" because of the technical challenges involved in presenting a functioning GPU for the virtual machine to initialize without things going awry.

Most computers today come with a GPU built-in to the CPU. This will be a major headache when trying to setup VGA passthrough, as VGA is a really old standard. Back when it was created, having multiple graphics cards on a single system was not a configuration they had foreseen or designed for. VGA calls can only be directed to a single device at a time, so the kernel has to use a VGA arbiter that switches the active device and directs VGA accesses to the appropriate card. I am oversimplifying this a bit, but Alex Williamson has a detailed post explaining the technical issues. In short, the x-vga=on flag passed to VFIO indicates to the VGA arbiter that the VFIO driver will need to participate in VGA arbitration, so everyone stays happy.

The problem is that the Xorg i915 driver for Intel's integrated GPUs does not participate in VGA arbitration, even though the devices claim the VGA address space. This means VGA calls get directed to the wrong card, (a) messing up your display on the host and (b) preventing the graphics card on the VM from functioning correctly. Ugh.

Fortunately, Alex has also written kernel patches to fix this, however be warned that they cripple 3D performance on the Intel GPU. Since we're building a high-performance VM for gaming, I am assuming that will not be an issue for you.

Kernel patches:

NoSnoop

NoSnoop is a feature flag on a PCIe device that allows it to issue transitions that to bypass cache. This can cause consistency problems when passing through the card to a virtual machine.

You can check if you card has NoSnoop enabled by running lspci -vvvv as root and seeing if your graphics card lists NoSnoop+ (enabled) or NoSnoop- (disabled) under the Capabilities > DevCtl section.

Previously this required patches to the kernel, but with kernel 3.15.5 in Fedora, these patches are no longer required.

Rebuilding packages

The first step is to setup a packaging environment and download the upstream source RPMS:

yum install fedora-packager yum-utils
rpmdev-setuptree
cd ~/rpmbuild/SRPMS
yumdownloader --source kernel
rpm -i kernel*.src.rpm

It should have downloaded kernel-[version].fc20.src.rpm in your directory.

Download any of the patches listed above that may be required for your hardware configuration (as plaintext patch/diff files) and save them to ~/rpmbuild/SOURCES.

Rebuilding QEMU

Update 2014-06-09: the newest versions of QEMU in virt-preview (>= 2.0.0) have the NoSnoop patches included! No rebuilding necessary. If you previously followed this guide, remove any QEMU exclusions from yum.conf and update to the latest available version.

Rebuilding Kernel

Download any required patches and save them as plaintext files in your ~/rpmbuild/SOURCES folder. Next, add your PatchXYZ: filename.patch lines where you see other patches being declared. For example:
Next, edit ~/rpmbuild/SPECS/qemu.spec and find the lines where existing patches are listed. You should see something like this:

[...]
# patches headed upstream
Patch12016: disable-i8042-check-on-apple-mac.patch

Patch14000: hibernate-freeze-filesystems.patch

Patch14010: lis3-improve-handling-of-null-rate.patch

Patch15000: nowatchdog-on-virt.patch

Add additional lines corresponding to your patchsets. The result could be something like:

[...]
# patches headed upstream
Patch12016: disable-i8042-check-on-apple-mac.patch

Patch14000: hibernate-freeze-filesystems.patch

Patch14010: lis3-improve-handling-of-null-rate.patch

Patch15000: nowatchdog-on-virt.patch

Patch16000: aw-i915-v3-add-vga-arbiter-module-option.patch
Patch16001: aw-vgaarb-non-decoded-resources.patch

I added in the Patch16000 and Patch16000, specifying the filenames of the patch files I saved into my SOURCES folder.

Now, scroll down to the %prep section where you see other patches being applied. That would look something like:

[...]
#rhbz 1051668
ApplyPatch Input-elantech-add-support-for-newer-elantech-touchpads.patch

# CVE-2014-3917 rhbz 1102571 1102715
ApplyPatch auditsc-audit_krule-mask-accesses-need-bounds-checking.patch

# END OF PATCH APPLICATIONS

Above the line # END OF PATCH APPLICATIONS, add lines for your patch sets, for example:
[...]
# CVE-2014-3917 rhbz 1102571 1102715
ApplyPatch auditsc-audit_krule-mask-accesses-need-bounds-checking.patch

ApplyPatch aw-i915-v3-add-vga-arbiter-module-option.patch
ApplyPatch aw-vgaarb-non-decoded-resources.patch

# END OF PATCH APPLICATIONS

Now rebuild the kernel with our patches and install it:
<code>rpmbuild -ba ~/rpmbuild/SPECS/kernel.spec --without=perf --without=tools --without=debug --without=debuginfo

It may list some build dependencies. If so, install them with yum install foo and then run the rpmbuild command again. When the build is complete, it output a list of filenames that you can install. Here's a quick command to do so:
yum reinstall ~/rpmbuild/RPMS/$(uname -m)/kernel-{,headers,devel}*.rpm

Reboot and your system will be fully patched! I suggest you add a line exclude=kernel* to /etc/yum.conf to prevent for patched packages from being upgraded.

Installing KVM

Because this is all experimental stuff, install fedora-virt-preview to get the latest and greatest software set:

wget http://fedorapeople.org/groups/virt/virt-preview/fedora-virt-preview.repo -O /etc/yum.repos.d/fedora-virt-preview.repo
yum install @virtualization

Next, let's be nice to the VMs and give them some time to perform a graceful shutdown before the host powers off:

sed -i 's/#ON_SHUTDOWN=.*/ON_SHUTDOWN=shutdown/' /etc/sysconfig/libvirt-guests
systemctl enable libvirt-guests
systemctl enable libvirtd

Edit the default kernel boot arguments (specified in GRUB_CMDLINE_LINUX of /etc/default/grub) and add intel_iommu=on for Intel CPUs or iommu=pt iommu=1 for AMD CPUs to turn on IOMMU functionality.

As well, the host initializes certain devices on init (e.g. graphic cards, USB controller, audio chipsets) so these devices need to be manually assigned to the PCI stub driver to prevent the host from using the device during host boot. It allows the VFIO driver to later bind to the devices and pass them to a VM. Add pci-stub.ids=PCI_IDs where PCI_IDs is a comma-separated list of PCI IDs as given by lspci -nn. For example, my GRUB_CMDLINE_LINUX looks like:

GRUB_CMDLINE_LINUX="vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet pci-stub.ids=1002:6810,1002:aab0,8086:8c20,8086:153a,8086:8c31 intel_iommu=on systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0"

With the configuration changes in place, regenerate the GRUB configuration:

grub2-mkconfig -o /boot/grub2/grub.cfg

Lastly, we will now need to have VFIO bind to any device you want passed to a VM. This will include devices stubbed on boot and possibly others:

cat << EOF > /etc/sysconfig/vfio-bind
DEVICES="FULL_PCI_IDs"
EOF

cat << EOF > /usr/local/bin/vfio-bind
#!/bin/sh
modprobe vfio-pci
for dev in "\$@"; do
        vendor=\$(cat /sys/bus/pci/devices/\$dev/vendor)
        device=\$(cat /sys/bus/pci/devices/\$dev/device)
        if [ -e /sys/bus/pci/devices/\$dev/driver ]; then
                echo \$dev > /sys/bus/pci/devices/\$dev/driver/unbind
        fi
        echo \$vendor \$device > /sys/bus/pci/drivers/vfio-pci/new_id
done
EOF
chmod +x /usr/local/bin/vfio-bind

cat << EOF > /etc/systemd/system/vfio-bind.service
[Unit]
Description=Binds devices to vfio-pci
After=syslog.target

[Service]
EnvironmentFile=-/etc/sysconfig/vfio-bind
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/local/bin/vfio-bind \$DEVICES

[Install]
WantedBy=multi-user.target
EOF
systemctl enable vfio-bind.service
systemctl start vfio-bind.service

The system will now automatically attempt to bind to the devices indicated in /etc/sysconfig/vfio-bind to VFIO at bootup. The format of FULL_PCI_IDs is a little different than earlier, as it is space separated and requires a full bus address prefix as per ls /sys/bus/pci/devices. You can use lspci -nn to identify a device, and then the output from the file listing to identify its full prefix. Here's an example of my configuration:
DEVICES="0000:01:00.0 0000:01:00.1 0000:00:1b.0 0000:00:19.0 0000:06:00.0 0000:00:14.0"

Because QEMU normally runs sandboxed, we need to 'unhinge' it and give it root privileges so it can control hardware. In /etc/libvirt/qemu.conf, add:

user = "root"
group = "root"
clear_emulator_capabilities = 0

As well, we need to provide QEMU with access to the VFIO device files. List all available VFIO groups like this:
ls -1 /dev/vfio

For ever number that appears, ensure its full path appears in the cgroup_device_acl configuration parameter. For example, mine looks like so:

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1", "/dev/vfio/4", "/dev/vfio/5",
    "/dev/vfio/6", "/dev/vfio/7", "/dev/vfio/8",
    "/dev/vfio/9"
]

That's it! The last step is to actually create your virtual machine. The following snippet creates a sample reference file in your home directory:

cat << EOF > ~/gaming-vm-sample.xml
<domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>YOUR_VM_NAME</name>
  <uuid>07478ac0-6a99-11e3-8266-00259086c7d9</uuid>
  <memory unit='KiB'>MEMORY_KB</memory>
  <currentMemory unit='KiB'>MEMORY_KB</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>NUM_CORES</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='NUM_CPUS' cores='NUM_CORES' threads='NUM_THREADS'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='DEV_PARTITION_PATH'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='PATH_TO_LOCAL_FILE'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb0'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb0'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb0'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:e3:f9:47'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=GPU_PCI_ID,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=GPU_PCI_AUDIO,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=OTHER_DEVICE,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=ANOTHER_DEVICE,bus=pcie.0'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
  </qemu:commandline>
</domain>
EOF

Thanks to kaeptnb on Arch Linux forums for providing the libvirt-xml file structure the above is based off of.

Edit the variables YOUR_VM_NAME, MEMORY_KB, NUM_CPUS (physical CPUs), NUM_CORES (cores per CPU), NUM_THEADS (threads per core; 1=normal and 2=each virtual core gets a corresponding HyperThreading CPU) to your liking.

On my host, I have a 4 core CPU with hyperthreading (8 logical cores) so assigning the virtual machine 1 CPU, 3 cores and 2 threads (resulting in 6 logical virtual CPUs visible, and reserving 1 physical core + 1 HT core for the host) has worked very well.

The section where the variables DEV_PARTITION_PATH and PATH_TO_LOCAL_FILE appear can be used in conjunction or one at a time, depending on your configuration. A plain image file on your disk can be created with qemu-img or just point it to an unformatted partition (preferred for better performance - it can be a physical partition or a logical RAID/LVM one).

As well, be sure to specify the correct GPU_PCI_ID and other devices IDs for your setup. If you only want to pass through a GPU, then remove the -device and vfio-pci parameters for the other devices.

When you have fully customized the file, import it:

virsh define ~/gaming-vm-sample.xml

You may wish to open virt-manager and copy the host CPU features if you're not running a Haswell CPU. Another tip is for Windows, you will need to download the latest VirtIO driver image and attach it to the machine in order for Windows to detect a disk. Use the Have Disk... option and Browse to the WIN7/AMD64 folder during installation and install the device drivers listed there.

Now set the VM auto-start on boot and get gaming!:

virsh autostart YOUR_VM_NAME

Troubleshooting

If your VM isn't booting, try editing the VM configuration (virsh edit YOUR_VM_NAME) and on the line where you input GPU_PCI_ID, change bus=pcie.0 to bus=root.1,addr=00.1. This exposes the graphics card on a different PCIe port in the VM which may sometimes help.

For nVidia users, recent driver packages apparently are broken without the kvm=off flag to the QEMU's -cpu parameter. Apparently, nVidia checks for the KVM hypervisor's signature and disables their driver if it detects it. It is not clear if this was an intentional change or not, but this is the reality of it.

Be sure to read through the KVM thread on Arch Linux forums that's been linked throughout this howto, as it contains tons of valuable (albeit sparse) information. Another tip would be to get in touch with the fedora-virt mailing list and describe the issue.

Resources

Rating: 

Comments

Works great for me but don't forget to run "chmod +x /usr/local/bin/vfio-bind"

Using a radeon R7 240 as the vfio device on a gigabyte 990XA-UD3 mobo.

Cheers!

Thanks! Fixed.

Hi,

Still in the process of setting up my gaming virtual machine but here are my updates for your howto:

No snoop patches are no longer required (kernel-3.15.5-200 already has them)
The line that says "Next, edit ~/rpmbuild/SPECS/qemu.spec and find the lines where existing patches are listed" should read "Next, edit ~/rpmbuild/SPECS/kernel.spec and find the lines where existing patches are listed".

Thanks for the great guide :)

Nick

"rpmbuild -ba ~/rpmbuild/kernel.spec --without=perf --without=tools --without=debug --without=debuginfo" should be "rpmbuild -ba ~/rpmbuild/SPECS/kernel.spec --without=perf --without=tools --without=debug --without=debuginfo"

Thanks! Fixed.

I just bought a brand new gaming machine, with the full intention of running linux with a windows VM.
Unfortunately I found out a bit too late my mobo does not support VT-d, so I'm returning it and I'll have a new one sometime before the end of the week.
I will be using this guide and I'll let you know how it goes, wish me luck!

And thank you in advance for writing this, it looks way simpler than any of the other guides on using KVM like this.

Glad to hear you like the style - I do my best to keep things simple and straightforward. Feel free to comment here if you have any issues!

Alright, I've finally got my virtual machine up and running and its amazing!
First off, there are some issues with the guide. I ended up using virt-manager to create and run the vm because i got a bit lost trying to create and run it with the xml configuration.
Secondly, you should/need to add a new section to the guide, as I ran into an interesting roadblock while patching the kernel:
The second patch posted (6e1b4fdad5157bb9e88777d525704aba24389bee), in particular the very last change in it, doesn't apply properly with the latest kernel. I needed to manually open the header file, and edit the patch file so that it added in the 1-line change somewhere near the end, luckily it was pretty simple so it didn't take too long and I was able to build the kernel.

Anyways, after a lot of fiddling with virt-manager (it has its issues as well, particularly in regards to virtual HDDs) I finally got it up and running, with the usually snapshots and pausing and usb redirection and all working great.
Also I am using synergy (hosted on the windows machine, since my keyboard (Corsair K70) has some driver issues in linux, and so that UAC prompts don't mess up your input which they do when synergy is ran as client on windows)
Here are my specs:
CPU: i7-4790k (One of the very few -K cpus to have VT-d
GPU: R9 280X 3GB
MOBO: ASRock Z87 Extreme6 (Fantastic motherboard btw, and a great price point for VT-d support, I got it $130 at a MicroCenter)
RAM: 2x4GB GSkill Sniper 2133
+ 2x8GB Crucial Sport 1600
(24GB total, 16gb to VM, 8gb to host)
HDD: 2x1TB WD Blue in linux softraid 1 (mirror)
SSD: 1x256GB Crucial SSD

So, if you want to contact me (email) about fixing up this guide and updating it with the new patches, please do so, I would love to help improve it. I might even make my own guide considering how well this works.

Anyways, I just want to thank you a bunch for guide, running windows like this is awesome!

Glad to hear it's working! Please reach out to me at s.adam [at] diffingo.com with the kernel version you used and the patchsets and I'll run some tests and update the guide.

I'm curious to see how you setup your virt-manager VM as well, as when I setup my machine VFIO passthrough was not available from the GUI. Has this changed, or are you using regular PCI assignment?

I seem to be running into the same problem with this patch. Could you tell me specifically which lines you changed in order to get it to work? Without it, I am getting this possibly related error:

error: internal error: early end of file from monitor: possible problem:
2014-08-14T00:36:38.445181Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga
2014-08-14T00:36:39.449059Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get device 0000:01:00.0
2014-08-14T00:36:39.684085Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
2014-08-14T00:36:39.684115Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Thanks in advance.

How much does Alex's patches 'cripple' the Intel GPU?

If I remember correctly, Alex had quoted that 3D acceleration will no longer work. It will function fine for basics (web browsing or email, etc) but I'm not sure how it fairs for video decoding, youtube, etc.

Hello,
running FC20 on a Phenom 3Gh but seems to be very cpu intensive. I try different configuration for CPU.
The result is the same ... Windows 7 VM start and VM cpu is fixed on 100%. :(

I had this same problem, the solution was to make sure that you configure the virtual cpu to have the _exact_ same format (cpus, cores, threads) as your physical cpu. I used the virt-manager gui, which lets you configure CPUs, cores, and threads (per core). I have an i7-4790k which is a quad core hyperthreaded to 8 cores, so its 1 cpu, 4 cores, and 2 threads (PER CORE), this got the cpu usage down to almost nothing at idle and the cpu graph looked way more normal.

hi Firewing,
thanks for the guide. I use a very similar setup with kUbuntu. I was able to set libvirt / virt-manager up BUT libvirt kept changing the xml - adds "video" clause, changes time parameter and changes the last few "qemu:arg" values at the end of the file - to add slashes and thereby prevent the VM from starting.
I could change the xml back fairly easily but it becomes painful after a while and I notice others on the Arch forum having similar problems (and also unable to resolve them)

Have you had any trouble with libvirt unilaterly / magically changing the xml (making it unworkable).

PS I used the latest libivrt from the Ubuntu virtualisation team ppa, so should be running the latest stable code ... and still have trouble

any advice is much appreciated :)

Hi Redger,

I haven't run into any issues using libvirt XML on Fedora 20. Attached below are the versions of my package set:

libvirt-client-1.2.6-2.fc20.x86_64
libvirt-daemon-1.2.6-2.fc20.x86_64
libvirt-daemon-config-network-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-interface-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-network-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-nodedev-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-nwfilter-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-qemu-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-secret-1.2.6-2.fc20.x86_64
libvirt-daemon-driver-storage-1.2.6-2.fc20.x86_64
libvirt-daemon-kvm-1.2.6-2.fc20.x86_64
libvirt-glib-0.1.7-2.fc20.x86_64
libvirt-python-1.2.6-1.fc20.x86_64
qemu-common-2.1.0-0.5.rc3.fc20.x86_64
qemu-guest-agent-2.1.0-0.5.rc3.fc20.x86_64
qemu-img-2.1.0-0.5.rc3.fc20.x86_64
qemu-kvm-2.1.0-0.5.rc3.fc20.x86_64
qemu-sanity-check-nodeps-1.1.5-1.fc20.x86_64
qemu-system-x86-2.1.0-0.5.rc3.fc20.x86_64

I configured my XML and it's been running smoothly since. To be fair though, after the initial setup I have not been playing with it much in virt-manager.

Cheers,
Stewart

thanks Stuart
it seems I have the same versions of libvirt as you, with a slightly older qemu (2.0)

libvirt-bin 1.2.6-0ubuntu1~ppa
libvirt-doc 1.2.6-0ubuntu1~ppa
libvirt-glib-1.0-0 0.1.6-1ubuntu2
libvirt0 1.2.6-0ubuntu1~ppa
libvirtodbc0 6.1.6+repack-0ubun

qemu 2.0.0+dfsg-2ubuntu
qemu-keymaps 2.0.0+dfsg-2ubuntu
qemu-kvm 2.0.0+dfsg-2ubuntu
qemu-launcher 1.7.4-1ubuntu2
qemu-system 2.0.0+dfsg-2ubuntu
qemu-system-common 2.0.0+dfsg-2ubuntu
qemu-system-x86 2.0.0+dfsg-2ubuntu
qemu-user 2.0.0+dfsg-2ubuntu
qemu-utils 2.0.0+dfsg-2ubuntu
qemuctl 0.3.1-1

It seems unlikely that this is qemu related (mind you, anything's possible ...). Perhaps it's related to the Debian implementation (I use Ubuntu 14.04). All very odd

thanks again
R

ok, solved it ...
Ubuntu uses Apparmor which is a bit sensitive to some things. In particular it will not fail when it parses /usr/share/qemu/bios.bin ... and will therefore not create a security profile for the vm. Therefore -
(b) copy the file /usr/share/seabios/bios.bin (the /usr/share/qemu/bios.bin file is actuall a link to the seabios entry) to a non-system location. You can copy it to your home directory or a mounted drive
(c) Update your VM to point to the newly copied bios.bin. Better still do all this before you create the VM definition and avoid this apparmor problem in the first place
(d) if the VM is already created you'll need to restrospectively construct (or update) the apparmor profile using - If profile does not exist: export VM=foo ; virsh dumpxml $VM | sudo /usr/lib/libvirt/virt-aa-helper -c -u libvirt-`virsh domuuid $VM`
If profile already does exist: export VM=foo ; virsh dumpxml $VM | sudo /usr/lib/libvirt/virt-aa-helper -r -u libvirt-`virsh domuuid $VM` all described in this bug report https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/799997
(e) Also add vfio definitions to the apparmor profile, this is a bit messy because you need to find the generated profile and manually update it. Go to /etc/apparmor.d/libvirt and find the definitions for the VM, there should be 2 files with name format "libvirt-[uuid]" and "libvirt-[uuid].files". We need to copy the second one, delete the current content, add a new line allowing eg. "/dev/vfio/*" rw, and then add the name of this new file to "libvirt-[uuid]" so it will be read and included (use the existing entry for "libvirt-[uuid].files" as a starting point).
(f) Reload libvirt (not sure if this is necessary, but better to be safe) "sudo service libvirt-bin restart"
(g) now the VM should start .,,, assuming there are no other issues (I had a few created by the translation process, but they weren't security related)

Hello. Thank you for the guide. I have one question, with only one video card you can run both the host system and the guest system?

You can, but only one at a time. You can boot your host system normally, but when you want to switch over to the virtual machine you would need switch to the command line (i.e. "init 3") and then unbind your graphics card from its currently assigned driver to free it for use with the virtual machine. When you're done with the virtual machine, you'll need to do the inverse and bind your card to the OS driver and then start X again.

G'day, I'm wondering, will this howto allow me to build on top of a system with only a single discrete video card?
I have an asus g74 laptop, and I want to run a windows vm and osx vm. happy to manage the underlying host using ssh, if I need graphics, I can forward to an x11 server on the windows or osx vm.

I did have a go at doing the same via the arch linux howto, but couldn't get it to work, wondering how successful I'll be with this walk through before I have a go.

Hope you have the time to respond.

Hi all,

Just a few corrections to the above guide around patching with the latest kernel.

Firstly, this line "Next, edit ~/rpmbuild/SPECS/qemu.spec" should really be "Next, edit ~/rpmbuild/SPECS/kernel.spec".

Secondly, the second patch that we need to apply to the kernel (from here: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?i...) does not apply cleanly to kernel 3.16 (or 3.15 for that matter). We can fix this by removing lines 125 through 135 from the patch file, then adding the following line directly to /drivers/gpu/drm/i915/intel_drv.h

extern void i915_disable_vga_mem(struct drm_device *dev);

Add this line direclty above the last line in the file, which is:

#endif /* __INTEL_DRV_H__ */

save this then run the compile step again (and follow the rest of the guide), the patching should succeed and your kernel should work for vfio.