Secondary GPU Passthrough
It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly - see VTdHowTo
It's assumed you have Xen (the Hypervisor) installed and a running Dom0 - see Xen_Beginners_Guide
It's assumed you have a ATI Card you want to passthrough,
simply because NVIDIA will only work
a) with certain patches and
b) with certain Windows Versions
- this does not apply for High-End Quadro Cards (see XenVGAPassthroughTestedAdapters for more information).
It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.
It's recommended to use newest Xen (newest stable or even xen-unstable)
- 1. relevant Dom0 kernel Xen-Options
CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=500 CONFIG_XEN_SAVE_RESTORE=y CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_HVC_XEN=y CONFIG_XEN_WDT=y CONFIG_XEN_FBDEV_FRONTEND=m CONFIG_XEN_BALLOON=y CONFIG_XEN_SELFBALLOONING=y CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=y CONFIG_XEN_GRANT_DEV_ALLOC=y CONFIG_SWIOTLB_XEN=y CONFIG_XEN_TMEM=y CONFIG_XEN_PCIDEV_BACKEND=y CONFIG_XEN_PRIVCMD=y
Note: Its recommended to compile these Xen Options into the kernel, not as a module.
Technically it is possible to use modules, but it complicates things,
especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub.
Compiling it into the kernel and giving it a corresponding cmdline (see below) is simpler.
For those that prefer to avoid rebuilding the kernel, and simply use modules as they are in your default kernel, see this script: bind_lib.bash
- 2. adapt Dom0 kernel commandline to hide the relevant PCI Ids.
you get these PCI Ids with "lspci".
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 ... 08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970] 08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] ...
the corresponding grub config may look like:
multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7) module (hd0,1)/boot/initrd.img-3.3.0
In this example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU, the other ones belong to a also to-be-passed-through USB Controller.
- 3. prepare your Dom0 Config
builder = 'hvm' memory = '8195' #for upstream qemu: #device_model_override = '/usr/lib/xen/bin/qemu-system-x86_64' #device_model_version = 'qemu-xen' vcpus = 6 # pinning is highly recommended for latency reasons (see corresponding xen commandline in grub) #cpus="2-7" # for installation with physical cdrom: disk = [ '/dev/vg_ssd/win7system,,hda' , 'phy:/dev/cdrom,hdc:cdrom,r' ] # for after the os-installation: #disk = [ '/dev/vg_ssd/win7system,,hda' , '/dev/vg0/win7data,,hdb' ] name = 'win' serial = 'pty' # mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/ # without gplpv drivers installed: vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ] # with gplpv drivers installed: #vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ] # uuid generator e.g. here: http://www.guidgenerator.com uuid = "48a09a51-d6e8-4984-b380-4dff0c9dc7f9" # after installation "d" can be ommited for a little faster DomU boot boot="dc" sdl=0 # it's not necessary to have the passed-through GPU as primary within the DomU, so: gfx_passthru=0 keymap = "en" vnc=1 vncconsole=0 #vncpasswd="supersecret" vnclisten="127.0.0.1" vncdisplay=10 pci = [ '08:00.0' , '08:00.1' , '00:1d.0' , '00:1d.1' , '00:1d.2' , '00:1d.7' ] #only working with upstream-qemu #soundhw='hda' on_xend_stop = "shutdown" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" monitor=1 pci_power_mgmt=1 xen_platform_pci=1 pci_msitranslate=1 viridian=1 hpet=1 acpi=1 apic=1 pae=1
Note: In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)
You should perhaps adapt:
- the number of cpus (and their pinning)
- the paths of the discs and cdrom
- the uuid
- the mac address
- the name
- the memory ammount
- the pci ids
- 4. start the DomU
xl create /etc/xen/win
attach a VNC Viewer to the Primary GPU of your new DomU (i.e. normally a simulated cirrus vga chip):
and install your DomU-OS
- 5. Notes/Hints on Installation
For additional Infos and Tips regarding VGA Passthrough see Xen_VGA_Passthrough
It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config, i.e. without any passed through devices.
After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers.
LVM is especially useful for this as it can create snapshots:
NOTE: All these LVM Snapshot-Create/Merge/Delete operations should be done when the affected DomU is DOWN
- create a snapshot
lvcreate -s /dev/vg_ssd/win7system -L 5G -n win7system_snapshot
adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system
now you can fiddle around with drivers/programms, and if it works
- merge the snapshot into the origin lv:
lvconvert --merge /dev/vg_ssd/win7system_snapshot
or - you can
- delete the snapshot and return to the state before: