Difference between revisions of "Secondary GPU Passthrough"

From Xen
m
(added "Secondary passthrough may not work before installing the graphics driver.")
 
(20 intermediate revisions by 3 users not shown)
Line 1: Line 1:
It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly
+
It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly - see [[VTdHowTo]]
   
It's assumed you have Xen (the Hypervisor) installed and a running Dom0
+
It's assumed you have Xen (the Hypervisor) installed and a running Dom0 - see [[Xen_Beginners_Guide]]
 
It's recommended to use newest Xen (newest stable or even xen-unstable)
 
   
 
It's assumed you have a ATI Card you want to passthrough,
 
It's assumed you have a ATI Card you want to passthrough,
  +
simply because NVIDIA will only work<br>
simply because NVIDIA will only work a) with certain patches and b) with certain Windows Versions - a) and b) do not apply for High-End Quattro Cards - see [[XenVGAPassthroughTestedAdapters]]
 
  +
a) with certain patches and <br>
  +
b) with certain Windows Versions <br>
  +
- this does not apply for High-End Quadro Cards (see [[XenVGAPassthroughTestedAdapters]] for more information).
   
 
It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.
 
It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.
  +
  +
It's recommended to use newest Xen (newest stable or even xen-unstable)
   
   
Line 48: Line 51:
 
</code>
 
</code>
   
Note: Its recommended to compile these Xen Options into the kernel, not as a module.
+
Note: Its recommended to compile these Xen Options into the kernel, not as a module.<br>
Technically it is possible to use modules, but it complicates things
+
Technically it is possible to use modules, but it complicates things, <br>
(especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub).
+
especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub.<br>
  +
Compiling it into the kernel and giving it a corresponding cmdline (see below) is simpler.
   
  +
For those that prefer to avoid rebuilding the kernel, and simply use modules as they are in your default kernel, see this script: [[bind_lib.bash]]
; 2. adapt Dom0 kernel commandline to hide the relevant PCI Ids. You get these with "lspci".
 
   
  +
; 2. adapt Dom0 kernel commandline to hide the relevant PCI Ids.
<code>
 
multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin
 
module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7)
 
module (hd0,1)/boot/initrd.img-3.3.0
 
</code>
 
   
  +
you get these PCI Ids with "lspci".
In this Example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU,
 
the other ones belong to a also to-be-passed-through USB Controller:
 
   
 
<code>
 
<code>
Line 69: Line 68:
 
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
 
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
 
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
 
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
  +
...
 
08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970]
 
08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970]
 
08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
 
08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
  +
...
   
 
</code>
 
</code>
  +
  +
the corresponding grub config may look like:
  +
  +
<code>
  +
multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin iommu=1
  +
module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7)
  +
module (hd0,1)/boot/initrd.img-3.3.0
  +
</code>
  +
  +
In this example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU,
  +
the other ones belong to a also to-be-passed-through USB Controller.
  +
  +
The iommu parameter must be set. See [[VTd_HowTo]] for details.
  +
   
 
; 3. prepare your Dom0 Config
 
; 3. prepare your Dom0 Config
Line 96: Line 111:
 
# mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/
 
# mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/
 
# without gplpv drivers installed:
 
# without gplpv drivers installed:
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ]
+
vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ]
 
# with gplpv drivers installed:
 
# with gplpv drivers installed:
 
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ]
 
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ]
Line 132: Line 147:
   
   
  +
Note:
Notes:
 
 
In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)
 
In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)
  +
You _should_ adapt:
 
  +
You '''should''' perhaps adapt:
   
 
* the number of cpus (and their pinning)
 
* the number of cpus (and their pinning)
Line 142: Line 158:
 
* the name
 
* the name
 
* the memory ammount
 
* the memory ammount
  +
* the pci ids
 
   
   
Line 151: Line 167:
 
</code>
 
</code>
   
attach a VNC Viewer to the Primary GPU of your new DomU (i.e. a simulated cirrus vga chip normaly):
+
attach a VNC Viewer to the Primary GPU of your new DomU (i.e. normally a simulated cirrus vga chip):
   
 
<code>
 
<code>
Line 158: Line 174:
   
 
and install your DomU-OS
 
and install your DomU-OS
  +
  +
   
 
; 5. Notes/Hints on Installation:
 
; 5. Notes/Hints on Installation:
  +
  +
For additional Infos and Tips regarding VGA Passthrough see [[Xen_VGA_Passthrough]]
  +
 
It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config,
 
It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config,
i.e. without any passed through devices.
+
i.e. without any passed through devices.
  +
After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers.
 
  +
After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers. Secondary passthrough may not work before installing the graphics driver.
   
 
LVM is especially useful for this as it can create snapshots:
 
LVM is especially useful for this as it can create snapshots:
Line 173: Line 195:
 
</code>
 
</code>
   
* adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system
+
adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system
   
now you can fiddle around with drivers/programms, and if it works the snapshot can be merged to the origin lv:
+
now you can fiddle around with drivers/programms, and if it works
  +
  +
* merge the snapshot into the origin lv:
   
 
<code>
 
<code>
Line 181: Line 205:
 
</code>
 
</code>
   
  +
or - you can
* or - you can delete it and return to the state before the snapshot:
 
  +
  +
* delete the snapshot and return to the state before:
   
 
<code>
 
<code>
 
lvremove /dev/vg_ssd/win7system_snapshot
 
lvremove /dev/vg_ssd/win7system_snapshot
 
</code>
 
</code>
  +
  +
[[Category:Xen]]
  +
[[Category:Users]]
  +
[[Category:Overview]]
  +
[[Category:Example]]
  +
[[Category:VGA]]

Latest revision as of 10:03, 28 February 2013

It's assumed you have VT-D capable Hardware (CPU/Chipset/Bios) and enabled it accordingly - see VTdHowTo

It's assumed you have Xen (the Hypervisor) installed and a running Dom0 - see Xen_Beginners_Guide

It's assumed you have a ATI Card you want to passthrough, simply because NVIDIA will only work
a) with certain patches and
b) with certain Windows Versions
- this does not apply for High-End Quadro Cards (see XenVGAPassthroughTestedAdapters for more information).

It's assumed you're familiar with compiling a custom kernel and fiddling around with grub options.

It's recommended to use newest Xen (newest stable or even xen-unstable)


1. relevant Dom0 kernel Xen-Options

CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y

Note: Its recommended to compile these Xen Options into the kernel, not as a module.
Technically it is possible to use modules, but it complicates things,
especially with pciback, because when used as a module you have to first unbind a perhaps after the boot attached driver and then assign it to pci-stub.
Compiling it into the kernel and giving it a corresponding cmdline (see below) is simpler.

For those that prefer to avoid rebuilding the kernel, and simply use modules as they are in your default kernel, see this script: bind_lib.bash

2. adapt Dom0 kernel commandline to hide the relevant PCI Ids.

you get these PCI Ids with "lspci".

00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
...
08:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman XT [Radeon HD 6970]
08:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
...

the corresponding grub config may look like:

       multiboot (hd0,1)/boot/xen.gz watchdog cpufreq=xen:performance dom0_mem=4096M dom0_max_vcpus=2 dom0_vcpus_pin iommu=1
       module (hd0,1)/boot/vmlinuz-3.3.0 ro root=LABEL=pcroot xen-pciback.hide=(08:00.0)(08:00.1)(00:1d.0)(00:1d.1)(00:1d.2)(00:1d.7)
       module (hd0,1)/boot/initrd.img-3.3.0

In this example, the PCI IDs 08:00.0 and 08:00.1 belong to the to-be-passed-through GPU, the other ones belong to a also to-be-passed-through USB Controller.

The iommu parameter must be set. See VTd_HowTo for details.


3. prepare your Dom0 Config

file: /etc/xen/win

builder = 'hvm'
memory = '8195'
#for upstream qemu:
#device_model_override = '/usr/lib/xen/bin/qemu-system-x86_64'
#device_model_version = 'qemu-xen' 
vcpus = 6
# pinning is highly recommended for latency reasons (see corresponding xen commandline in grub)
#cpus="2-7"
# for installation with physical cdrom:
disk = [ '/dev/vg_ssd/win7system,,hda' , 'phy:/dev/cdrom,hdc:cdrom,r' ]
# for after the os-installation:
#disk = [ '/dev/vg_ssd/win7system,,hda' , '/dev/vg0/win7data,,hdb' ]
name = 'win'
serial = 'pty'
# mac adress generator e.g. here: http://www.miniwebtool.com/mac-address-generator/
# without gplpv drivers installed:
vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=e1000' ]
# with gplpv drivers installed:
#vif = ['bridge=br0, mac=00:12:3e:6c:2f:02, model=virtio' ]
# uuid generator e.g. here: http://www.guidgenerator.com
uuid = "48a09a51-d6e8-4984-b380-4dff0c9dc7f9"
# after installation "d" can be ommited for a little faster DomU boot 
boot="dc"
sdl=0
# it's not necessary to have the passed-through GPU as primary within the DomU, so:
gfx_passthru=0
keymap = "en"
vnc=1
vncconsole=0
#vncpasswd="supersecret"
vnclisten="127.0.0.1"
vncdisplay=10
pci = [ '08:00.0' , '08:00.1'  , '00:1d.0' , '00:1d.1' , '00:1d.2' , '00:1d.7' ]
#only working with upstream-qemu
#soundhw='hda'
on_xend_stop = "shutdown"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
monitor=1
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1
acpi=1
apic=1
pae=1


Note: In this example it's assumed you use LVM for your DomU Disks (recommended regarding performance)

You should perhaps adapt:

  • the number of cpus (and their pinning)
  • the paths of the discs and cdrom
  • the uuid
  • the mac address
  • the name
  • the memory ammount
  • the pci ids


4. start the DomU

xl create /etc/xen/win

attach a VNC Viewer to the Primary GPU of your new DomU (i.e. normally a simulated cirrus vga chip):

xvncviewer localhost:5910

and install your DomU-OS


5. Notes/Hints on Installation

For additional Infos and Tips regarding VGA Passthrough see Xen_VGA_Passthrough

It may be a good idea to install the DomU initially WITHOUT the "pci" line in the DomU config, i.e. without any passed through devices.

After you have a defined stable DomU, enable the "pci" in the DomU config and install the necessary drivers. Secondary passthrough may not work before installing the graphics driver.

LVM is especially useful for this as it can create snapshots:

NOTE: All these LVM Snapshot-Create/Merge/Delete operations should be done when the affected DomU is DOWN

  • create a snapshot

 lvcreate -s /dev/vg_ssd/win7system -L 5G  -n win7system_snapshot 

adapt the "disk" line in the DomU accordingly - here: /dev/vg_ssd/win7system_snapshot instead of /dev/vg_ssd/win7system

now you can fiddle around with drivers/programms, and if it works

  • merge the snapshot into the origin lv:

lvconvert --merge /dev/vg_ssd/win7system_snapshot

or - you can

  • delete the snapshot and return to the state before:

lvremove /dev/vg_ssd/win7system_snapshot