Secondary GPU Passthrough

It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly - see VTdHowTo

It's assumed you have Xen (the Hypervisor) installed and a running Dom0 - see Xen_Beginners_Guide

It's assumed you have a ATI Card you want to passthrough, simply because NVIDIA will only work
a) with certain patches and
b) with certain Windows Versions
- this does not apply for High-End Quadro Cards (see XenVGAPassthroughTestedAdapters for more information).

It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.

It's recommended to use newest Xen (newest stable or even xen-unstable)

1. relevant Dom0 kernel Xen-Options

CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y

Note: Its recommended to compile these Xen Options into the kernel, not as a module.
Technically it is possible to use modules, but it complicates things,
especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub.
Compiling it into the kernel and giving it a corresponding cmdline (see below) is simpler.

For those that prefer to avoid rebuilding the kernel, and simply use modules as they are in your default kernel, see this script: bind_lib.bash

2. adapt Dom0 kernel commandline to hide the relevant PCI Ids.

you get these PCI Ids with "lspci".

00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
...
08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970]
08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
...

the corresponding grub config may look like:

       multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin iommu=1
       module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7)
       module (hd0,1)/boot/initrd.img-3.3.0

In this example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU, the other ones belong to a also to-be-passed-through USB Controller.

The iommu parameter must be set. See VTd_HowTo for details.

3. prepare your Dom0 Config

file: /etc/xen/win

builder = 'hvm'
memory = '8195'
#for upstream qemu:
#device_model_override = '/usr/lib/xen/bin/qemu-system-x86_64'
#device_model_version = 'qemu-xen' 
vcpus = 6
# pinning is highly recommended for latency reasons (see corresponding xen commandline in grub)
#cpus="2-7"
# for installation with physical cdrom:
disk = [ '/dev/vg_ssd/win7system,,hda' , 'phy:/dev/cdrom,hdc:cdrom,r' ]
# for after the os-installation:
#disk = [ '/dev/vg_ssd/win7system,,hda' , '/dev/vg0/win7data,,hdb' ]
name = 'win'
serial = 'pty'
# mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/
# without gplpv drivers installed:
vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ]
# with gplpv drivers installed:
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ]
# uuid generator e.g. here: http://www.guidgenerator.com
uuid = "48a09a51-d6e8-4984-b380-4dff0c9dc7f9"
# after installation "d" can be ommited for a little faster DomU boot 
boot="dc"
sdl=0
# it's not necessary to have the passed-through GPU as primary within the DomU, so:
gfx_passthru=0
keymap = "en"
vnc=1
vncconsole=0
#vncpasswd="supersecret"
vnclisten="127.0.0.1"
vncdisplay=10
pci = [ '08:00.0' , '08:00.1'  , '00:1d.0' , '00:1d.1' , '00:1d.2' , '00:1d.7' ]
#only working with upstream-qemu
#soundhw='hda'
on_xend_stop = "shutdown"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
monitor=1
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1
acpi=1
apic=1
pae=1

Note: In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)

You should perhaps adapt:

• the number of cpus (and their pinning)
• the paths of the discs and cdrom
• the uuid
• the mac address
• the name
• the memory ammount
• the pci ids

4. start the DomU

xl create /etc/xen/win

attach a VNC Viewer to the Primary GPU of your new DomU (i.e. normally a simulated cirrus vga chip):

xvncviewer localhost:5910

and install your DomU-OS

5. Notes/Hints on Installation

For additional Infos and Tips regarding VGA Passthrough see Xen_VGA_Passthrough

It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config, i.e. without any passed through devices.

After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers. Secondary passthrough may not work before installing the graphics driver.

LVM is especially useful for this as it can create snapshots:

NOTE: All these LVM Snapshot-Create/Merge/Delete operations should be done when the affected DomU is DOWN

• create a snapshot

 lvcreate -s /dev/vg_ssd/win7system -L 5G  -n win7system_snapshot 

adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system

now you can fiddle around with drivers/programms, and if it works

• merge the snapshot into the origin lv:

lvconvert --merge /dev/vg_ssd/win7system_snapshot

or - you can

• delete the snapshot and return to the state before:

lvremove /dev/vg_ssd/win7system_snapshot