XCP Ubuntu PCIPassthrough

Introduction

The below is only applicable to PV hosts. In this example the OS used was Ubuntu 10.04 LTS.

I don't believe you can do USB-only pass through with XCP 1.0. However, you can do PCI pass-through and pass-through the entire USB controller device.

Read this guide: it contains most of the answers, even if it's hard to parse: Xen_PCI_Passthrough

NOTE: Although this entire procedure worked, in the end I have to admit I found that PCI passthrough was too complex and not reliable enough for my purposes, so please give it a try, but I recommend doing it on a test machine.

Confirm that your dom0 supports pciback

I used XCP 1.0 in my case, which does support pciback, so you can skip this if you're on XCP.

# (on dom0)
# Look for 'pciback' here
ls /sys/bus/pci/drivers/

Confirm domU supports pciback

Now check your Linux domU (guest OS) for pci-passthrough support (needs xen-pcifront module). Your Ubuntu system must have xen-pcifront/xen_pcifront installed in the kernel for pci pass-through to work. Basically your kernel version must be v2.6.37+ to have it included by default.

# (on domU)
# check the version
uname -a

"uname -a" should return a kernel version greater than 2.6.37.

Custom domU pciback kernel

If your kernel version is less than 2.6.37, and there are no readily available replacement kernels, you can compile a custom kernel yourself.

Compiling the kernel

No Ubuntu 10.04 packages exist for a kernel of this version. I also tried the kernels located here: http://kernel.ubuntu.com/~kernel-ppa/mainline/, but apparently they're not compiled with Xen support or something because my XCP refused to boot the kernel complaining about missing modules... so I finally gave up and compiled my own kernel:

# (on domU)

# v2.6.37+ linux has pcifront built in:
# if you have a lower version, you need a newer kernel
# here's compile instructions I used on Lucid 10.04:
# reference: http://linuxtweaking.blogspot.com/2011/02/how-to-compile-kernel-from-kernelorg-in.html
#
# If you feel like living dangerously/lazily, you can try to use the 32-bit kernel I compiled..
# I've put up a link here, but naturally this comes with NO warranty at all, and Ubuntu specifically states
# compiling your own kernel voids any potential warranty/support.. so there's that..
# anyways:
# http://dev.locatrix.com/linux-headers-2.6.37_2.6.37-10.00.Custom_i386.deb
# http://dev.locatrix.com/linux-image-2.6.37_2.6.37-10.00.Custom_i386.deb
# so skip most of the below and follow along from where it starts with dpkg -i for the above files 

cd ~ # or somewhere

apt-get install fakeroot kernel-wedge build-essential makedumpfile kernel-package libncurses5 libncurses5-dev
apt-get build-dep --no-install-recommends linux-image-$(uname -r)

mkdir src
cd src

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.37.tar.gz
tar xvf linux-2.6.37.tar.gz
cd linux-2.6.37

# basically I left all the defaults alone.. seemed to already include xen and everything else
#for some god unknown reason I did have to uncheck the option labeled "one laptop per child" (OLC) support... no clue..
make menuconfig

export CONCURRENCY_LEVEL=3
make-kpkg clean
time fakeroot make-kpkg --initrd kernel-image kernel-headers

cd ..
ls -l # note the package file names and install them

dpkg -i linux-headers-2.6.37_2.6.37-10.00.Custom_i386.deb
dpkg -i linux-image-2.6.37_2.6.37-10.00.Custom_i386.deb
update-initramfs -c -k 2.6.37

update-grub

# note down the new kernel/init versions
ls -la /
# e.g. /vmlinuz-2.6.37 /initrd.img-2.6.37

#shutdown
halt

Configure XCP to use the new kernel

Here's how I re-configured my dom0 XCP to use the new kernel (this is the same as you configure it in the first place for which kernel to use):

# BACK on dom0

#find VM's uuid
xe vm-list

#take down a copy of this stuff before you change it
xe vm-param-list uuid=MYUUID

#specifically we'll modify the bootloader
xe vm-param-list uuid=MYUUID | grep PV-bootloader-args

#make sure your VM is now set to boot from the new kernel
#this may or may not apply to you.. but I configured my VM to have the kernel/ramdisk set this way.
#make sure to modify the below to match your exact kernel/ramdisk and remember the above in case you need to change it back.
xe vm-param-set uuid=MYUUID PV-bootloader-args="--kernel=/vmlinuz-2.6.37 --ramdisk=/initrd.img-2.6.37"

xe vm-param-list uuid=MYUUID  | grep PV-args
# note the args.. modify to add "iommu=soft swiotlb=force", for example:
xe vm-param-set uuid=MYUUID  PV-args="root=/dev/MYBOOTDRIVE ro quiet iommu=soft swiotlb=force"
Now search for the PCI device you want to passthrough. I wanted to passthrough a USB device, so that's what I'm doing here:

# (on dom0)
#search for the device/pci information.. couple different ways:

#easy way to identify pci devices:
lspci -tv

#best way to get the exact pci number, the device number should be from here:
ls /sys/bus/pci/devices

#some more general info for USB for confirmation
#for usb 2.0
dmesg | grep ehci_hcd
#e.g. ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 21 (level, low) -> IRQ 21
#for usb 1.1
dmesg | grep uhci_hcd
#e.g. uhci_hcd 0000:00:1d.3: irq 23, io base 0x0000ff20
#note down the pci address 0000:00:1d.7
# how I found the usb 1.1 UPS device:
dmesg | tail
#usb 2-1: USB disconnect, address 2
#usb 2-1: new low speed USB device using uhci_hcd and address 3

# regardless really use the device number from /sys/bus/pci/devices above 

Setup PCI passthrough

Now set the PCI devices to pass through. The below seems to be correct. The 0/ seems to be the virtual bus number.. so from what I understand it should always be 0.

# (on dom0)
#the below is a comma separated list of each PCI device to pass-through.. I passed through all of my USB controllers that I found
#the list could be just one device, like "other-config:pci=0/0000:00:1d.0".. I passed through all of them:
xe vm-param-set other-config:pci=0/0000:00:1d.0,0/0000:00:1d.1,0/0000:00:1d.2,0/0000:00:1d.3,0/0000:00:1d.7 uuid=MYUUID 

#You can later remove this setting if needed
xe vm-param-remove param-name=other-config param-key=pci  uuid=MYUUID

Start up the VM again

# (on dom0)
xe vm-start uuid=MYUUID 

Watch the xcp system logs as the machine boots to catch the pciback passthrough messages

# (on dom0)
tail -f /var/log/kern.log

I received messages like "FLR functionality not supported", but apparently that shouldn't stop it from working. The bottom of this page talks about FLR: XenPCIpassthrough

The below allows live attaching a pci device to the VM. This also worked once I fixed up the linux kernel as above.

<pre>
# (on dom0)
xl list
# note down the "Name" for your VM.. use below...
# again, the PCI device number here is from "ls /sys/bus/pci/devices" (with "lspci -tv" helping you find the right one)
xl pci-attach mymachinename 0000:00:1d.7

# there's other handy xl commands..
# xl pci-detach mymachinename  0000:00:1d.7
# xl pci-list mymachinename