Secondary GPU Passthrough

It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly - see VTdHowTo

It's assumed you have Xen (the Hypervisor) installed and a running Dom0 - see Xen_Beginners_Guide

It's assumed you have a ATI Card you want to passthrough, simply because NVIDIA will only work
a) with certain patches and
b) with certain Windows Versions
- a) and b) do not apply for High-End Quattro Cards - see XenVGAPassthroughTestedAdapters for more information.

It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.

It's recommended to use newest Xen (newest stable or even xen-unstable)

1. relevant Dom0 kernel Xen-Options

CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y

Note: Its recommended to compile these Xen Options into the kernel, not as a module.
Technically it is possible to use modules, but it complicates things,
especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub.
Compiling it into the kernel and giving it a corresponding cmdline (see below) is simpler.

2. adapt Dom0 kernel commandline to hide the relevant PCI Ids.

you get these PCI Ids with "lspci".

00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
...
08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970]
08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
...

the corresponding grub config may look like:

       multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin
       module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7)
       module (hd0,1)/boot/initrd.img-3.3.0

In this example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU, the other ones belong to a also to-be-passed-through USB Controller.

3. prepare your Dom0 Config

file: /etc/xen/win

builder = 'hvm'
memory = '8195'
#for upstream qemu:
#device_model_override = '/usr/lib/xen/bin/qemu-system-x86_64'
#device_model_version = 'qemu-xen' 
vcpus = 6
# pinning is highly recommended for latency reasons (see corresponding xen commandline in grub)
#cpus="2-7"
# for installation with physical cdrom:
disk = [ '/dev/vg_ssd/win7system,,hda' , 'phy:/dev/cdrom,hdc:cdrom,r' ]
# for after the os-installation:
#disk = [ '/dev/vg_ssd/win7system,,hda' , '/dev/vg0/win7data,,hdb' ]
name = 'win'
serial = 'pty'
# mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/
# without gplpv drivers installed:
vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ]
# with gplpv drivers installed:
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ]
# uuid generator e.g. here: http://www.guidgenerator.com
uuid = "48a09a51-d6e8-4984-b380-4dff0c9dc7f9"
# after installation "d" can be ommited for a little faster DomU boot 
boot="dc"
sdl=0
# it's not necessary to have the passed-through GPU as primary within the DomU, so:
gfx_passthru=0
keymap = "en"
vnc=1
vncconsole=0
#vncpasswd="supersecret"
vnclisten="127.0.0.1"
vncdisplay=10
pci = [ '08:00.0' , '08:00.1'  , '00:1d.0' , '00:1d.1' , '00:1d.2' , '00:1d.7' ]
#only working with upstream-qemu
#soundhw='hda'
on_xend_stop = "shutdown"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
monitor=1
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1
acpi=1
apic=1
pae=1

Notes: In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)

You should perhaps adapt:

• the number of cpus (and their pinning)
• the paths of the discs and cdrom
• the uuid
• the mac address
• the name
• the memory ammount
• the pci ids

4. start the DomU

xl create /etc/xen/win

attach a VNC Viewer to the Primary GPU of your new DomU (i.e. normally a simulated cirrus vga chip):

xvncviewer localhost:5910

and install your DomU-OS

5. Notes/Hints on Installation

It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config, i.e. without any passed through devices.

After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers.

LVM is especially useful for this as it can create snapshots:

NOTE: All these LVM Snapshot-Create/Merge/Delete operations should be done when the affected DomU is DOWN

• create a snapshot

 lvcreate -s /dev/vg_ssd/win7system -L 5G  -n win7system_snapshot 

adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system

now you can fiddle around with drivers/programms, and if it works

• merge the snapshot the origin lv:

lvconvert --merge /dev/vg_ssd/win7system_snapshot

or - you can

• delete the snapshot and return to the state before:

lvremove /dev/vg_ssd/win7system_snapshot