Xen VGA Passthrough

Xen VGA graphics adapter passthrough

Xen 4.0.0 is the first version to support VGA graphics adapter passthrough to Xen HVM (fully virtualized) guests. This means you can give HVM guest full and direct control of the graphics adapter, making it possible to have high performance full 3D and video acceleration in a virtual machine.

Xen VGA passthrough requires IOMMU (Intel VT-d) support from the motherboard chipset, from the motherboard BIOS and from Xen. Please see the VTdHowTo wiki page for more information about Xen VT-d support and supported chipsets with IOMMU.

Xen 4.0.0 supports VGA passthrough of the primary graphics adapter (the one that gets initialized and is used when you power-on the computer) only. VGA passthrough of secondary graphics adapters currently requires additional extra patches to Xen.

Initial Xen VGA graphics passthru support was committed to xen-unstable development repository in September 2009, and more support for Intel IGD passthru was added in February 2010.

Why can't I do VGA passthrough to Xen PV (paravirtual) guest?

Xen PV domUs are lightweight by design so they don't have legacy x86 features like BIOS and various other features. This means complicated x86-centric features like VGA emulation are not easy for Xen PV guests. Note that some graphics cards might work in the PV domU just by using the normal Xen PCI passthru.

Can I share the graphics card between multiple VMs with Xen VGA passthru?

In short, no. Xen VGA graphics passthru is a special form of PCI passthru, and PCI passthru dedicates the PCI device (graphics card) to exactly one single VM.

Why is VGA passthrough different from normal PCI passthrough?

VGA adapters are not simple PCI devices like NICs or disk controllers. VGA adapters need to support many legacy x86 features like VGA BIOS, text mode, legacy IO ports, legacy memory ranges, VESA video modes, etc for correct and expected operation. This means VGA passthrough requires more code than normal Xen VT-d PCI passthrough.

Qemu-dm emulator used in the Xen HVM guest needs to disable the internal (emulated) graphics adapter, copy and map the real graphics adapter VGA BIOS to the virtual machine memory, emulate and execute it there to reset and initialize the graphics card properly, map and passthru the VGA adapter real memory to the HVM guest, passthru all the legacy IO-port and legacy memory ranges from the HVM guest to the real graphics adapter etc.

For more information about normal Xen PCI passthrough see XenPCIpassthrough wiki page.

Does Xen VGA graphics passthru work with computers with AMD CPUs/motherboards?

AMD chipsets having an IOMMU currently are the SR5690/SR5670 server chipsets, AMD 890FX, 990FX, 990X and 970 desktop chipsets. Note that other (non-FX) 890 chipsets don't have an IOMMU! AMD Server chipset SR5690 / SR5670 have been tested successfully with Xen and VGA passthrough (Tyan S8212 motherboard). If you know more AMD chipsets with IOMMU, or you have tested some chipset/motherboard, please send an email to xen-devel with details about the chipset/motherboard and we'll update this wiki page.

Also see VTdHowTo wiki page, it has the latest info about chipsets having a working IOMMU.

AMD posted a patch to enable VGA passthrough on their graphics adapters on 2010-10-08: http://lists.xensource.com/archives/html/xen-devel/2010-10/msg00284.html , also see this thread for some discussion about the patch: http://lists.xensource.com/archives/html/xen-devel/2010-12/msg00809.html .

Xen VGA passthru requirements from dom0 kernel

New kernels (v3.0 and following)

dom0 Linux kernel needs "XEN_PCIDEV_BACKEND" option enabled.

Old kernels (up to v2.6.39)

Make sure your dom0 Linux kernel has "CONFIG_XEN_PCIDEV_BACKEND_PASS" option enabled. "CONFIG_XEN_PCIDEV_BACKEND_VPCI" should be disabled. For example Debian Squeeze 2.6.32 kernel has the wrong PCIDEV_BACKEND mode as a default.. so in that case you might need to recompile the kernel with custom .config to get the right backend mode in use. BACKEND_PASS mode means the PCI IDs of the PCI device (graphics card) will be the same in the VM as in dom0/host.

Using Xen VGA graphics passthru

You need these settings for your Xen HVM guest cfgfile in /etc/xen/<guest> to enable VGA passthrough:

gfx_passthru=X 
pci=['yy:zz.n']

Where X is 1 to enable graphics passthru. The default value for X is 0, aka no graphics passthru with the traditional Qemu device model, but according to the xl.cfg man page, the default is to enable passthru and autodetect the type of graphics card being passed through with the upstream Qemu device model. "yy:zz.n" is the PCI ID of the VGA graphics adapter, you can check it with "lspci -v" in dom0.

Remember to install the graphics drivers in the DomU first, before expecting secondary passthrough to work.

The effect of gfx_passthru= option

When you specify "gfx_passthru=1" the passthru graphics card will be made the primary graphics card in the VM, and the Xen Qemu-dm emulated Cirrus graphics card is disabled when device_model = 'qemu-xen-traditional'. With the device_model = 'qemu-xen' (the upstream Qemu device model), specify vga = 'none' to disable the Qemu upstream emulated graphics card. It may be possible to make a Qemu upstream emulated graphics card secondary in the VM by setting vga = 'cirrus' or vga = 'stdvga' with the upstream Qemu device model.

If you use "gfx_passthru=0", or don't have the gfx_passthru= option at all, then the Xen Qemu-dm emulated Cirrus graphics card will be the primary in the VM, and the passthru graphics card will be secondary when device_model = 'qemu-xen-traditional'. With the device_model = 'qemu-xen' (the upstream Qemu device model) and gfx_passthru = 0, use the vga = ... option to set the type of primary emulated graphics card in the VM, and the passthru card will be secondary. According to the xl.cfg man page, with the device_model = 'qemu-xen' (the upstream Qemu device model) and no gfx_passthru option at all, PCI graphics passthru is enabled by default if a PCI graphics card is passed through and the type of device being used is autodetected. Also, the type of PCI graphics card can be specified as an Intel integrated graphics device with gfx_passthru = 'igd'.

How can I passthru my physical keyboard and mouse to the guest?

There are multiple options for that:

• Use Xen USB passthrough to passthru keyboard and mouse devices: XenUSBPassthrough
• Use Xen PCI passthrough to passthru the whole USB controller having the keyboard/mouse connected to it: XenPCIpassthrough
• Use the guest console over the network with VNC or RDP.
• Use a tool like Synergy that allows you to share the keyboard/mouse over a network.

Later there will be a patch that allows sharing the keyboard/mouse with dom0. This sharing functionality currently exists in Xen XCI.

Status of VGA graphics passthru in Xen

Status for Xen 4.0.0:

• VGA passthru is known to work with Intel Integrated Graphics Devices (IGDs)
• VGA passthru is known to work for primary graphics adapters (the graphics adapter that is used when you power-on the physical computer).
• Some Nvidia/ATI adapters require additional/extra patches for FLR (Function Level Reset) to work.
• Some Nvidia adapters require additional/extra vBAR=pBAR patches to make them work.
• Some graphics adapters modify their VGA BIOS when the computer starts, so the exact VGA BIOS cannot be read from memory after that.. this makes it impossible for Xen to copy the VGA BIOS to the HVM guest memory and re-execute it there to initialize the graphics card. In this case you need to use vendor-specific tools to extract the VGA BIOS from the graphics adapter, and load the VGA BIOS from a file to the HVM guest. Some vendors may have VGA BIOS images available from their website as a part of product (driver) updates.

These problems are being worked on and general methods to work-around the vendor-specific problems and issues are being developed.

Status for Xen 4.1.1:

• Passing thru AMD/ATI Radeon/FirePro/FireGL adapters as secondary to the VM should work out-of-the-box (you need latest ATI gfx drivers in the VM). Secondary means the Xen Qemu-dm virtual Cirrus adapter is the primary where you see the VM BIOS etc when powering on the VM, and the passthru adapter is secondary, so you can enable/use it after the OS in the VM has started and you've installed drivers for the adapter.

Status for Xen 4.2:

• Many patches to be merged in.. needs someone to collect all the patches from xen-devel mailinglist archives, clean them up, and submit for inclusion.
• AMD/ATI primary passthru with dynamic detection of BARs (both MMIO and PIO).
• Some support for Nvidia gfx passthru.

Status for Xen 4.4.x:

• VGA passthru is known to work with Intel Integrated Graphics Devices (IGDs) using the qemu-xen-traditonal device model.

Status for Xen 4.8.x:

• VGA passthru does not work with Intel Integrated Graphics Devices (IGDs) because a critical patch to support Intel IGD passthru was never backported to Xen 4.8. Applying the following patch to Xen 4.8 enables Intel IGD VGA passthru on Xen 4.8: https://www.mail-archive.com/xen-devel@lists.xen.org/msg96096.html.

Status for Xen 4.11.x:

• VGA passthru is known to work with Intel Integrated Graphics Devices (IGDs) using either the qemu-xen-traditonal device model or >= version 3.1 of Qemu upstream and Windows HVM guests.

Status for Xen 4.14.x and Xen 4.16.x:

• VGA passthru is known to work with Intel Integrated Graphics Devices (IGDs) using version 5.2 of Qemu upstream and Windows HVM guests.
• See the Xen VGA Passthrough Tested Adapters#Intel_display_adapters wiki page for more information, hints, and workarounds for various configurations with Intel IGDs.

Tested and working VGA graphics adapters

See XenVGAPassthroughTestedAdapters wiki page for more information about tested and working graphics adapters for Xen VGA passthrough. When you test a graphics adapter with Xen VGA passthru please write about your experiences to xen-users or xen-devel mailinglists so we can add the info about tested configurations here!

Are there patches for Xen secondary graphics card passthru?

Yes, please check this email for more information: http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00441.html and also check the links in the end of this wiki page.
These Patches are outdated and not supported.
These Patches are only useful for NVIDIA Cards, and will only fully work for WindowsXP.
Vista and Windows7 DomU will report a "not fully working" GPU within a DomU with these patches.

Hints

After a DomU Reboot the passed through GPU may have reduced 3D Performance. Reasons are unknown so far, but a workaround exists: [1]

Tutorials

Tutorial for Passing through a NVIDIA GPU as Primary GPU within the DomU:

• File:Xen VGA Passthrough.pdf

Tutorial for Passing through a (ATI) GPU as Secondary GPU within the DomU (much easier than as primary):

• SecondaryGPUPassthrough

Links to more information about Xen VGA passthrough

Xen VGA passthrough patches and discussion on xen-devel mailinglist:

• Setting Up A Xen Graphics Card Pass-Through
• Ubisoft Is Playing With Linux & Xen Virtualization
• Add support for Intel IGD graphics passthru: http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00240.html
• Initial Xen graphics passthru support patch (Aug 2009): http://lists.xensource.com/archives/html/xen-devel/2009-08/msg01050.html and http://xenbits.xensource.com/xen-unstable.hg?rev/05f3ca68f62d
• Intel Calpella / SandyBridge graphics passthru support patches: http://lists.xensource.com/archives/html/xen-devel/2010-06/msg00595.html
• AMD/ATI Radeon Xen VGA passthru patches (first version): http://lists.xensource.com/archives/html/xen-devel/2010-10/msg00284.html
• Some discussion about the AMD/ATI Xen VGA passthru patches: http://lists.xensource.com/archives/html/xen-devel/2010-12/msg00809.html
• AMD/ATI Xen VGA passthru patch with dynamic detection of BARs, both MMIO and PIO (second version): http://lists.xensource.com/archives/html/xen-devel/2010-12/msg00705.html
• Experimental xen-vBAR-pBAR patches etc to enable Nvidia Geforce 8400GS VGA passthru and secondary graphics adapter passthru: http://markmail.org/message/7gb7djbmlaxruaai and also here: http://lists.xensource.com/archives/html/xen-devel/2009-08/msg01176.html
• Nvidia graphics card extra requirements for VGA passthru: http://lists.xensource.com/archives/html/xen-devel/2009-08/msg00486.html
• Nvidia Xen VGA passthru patches (2010/05): http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00441.html
• Nvidia Xen primary VGA passthru patches (2011/05): http://lists.xensource.com/archives/html/xen-devel/2011-05/msg01426.html
• Description of some of the required changes when hacking Xen VGA passthru code: http://lists.xensource.com/archives/html/xen-devel/2011-05/msg01611.html
• Description of things to check/modify for VGA passthru: http://lists.xensource.com/archives/html/xen-devel/2011-05/msg01004.html
• Discussion about how to map the BARs: http://lists.xensource.com/archives/html/xen-devel/2011-05/msg01471.html
• Nvidia Xen VGA passthru patches (2013/03): http://lists.xen.org/archives/html/xen-devel/2013-03/msg01578.html (http://www.grim-is-a-geek.fr/xen-gfx-passthrough-patchs.bz2)
• xen-devel mailinglist archives have a lot of discussions about Xen VGA graphics passthru support and development: http://lists.xensource.com/archives/html/xen-devel .

Other links:

• http://blog.xen.org/index.php/2009/09/10/updated-vga-passthrough-video-demonstration/
• Xen PCI passthrough wiki page: XenPCIpassthrough

Where do I find more FAQs on USB, PCI and VGA passthrough?

See Xen Common Problems#USB, PCI and VGA passthrough