Comprehensive Xen Debian Wheezy PCI Passthrough Tutorial

Comprehensive Debian Wheezy Xen Tutorial

Introduction

---

The following guide is comprised of my personal documentation and excessive filtering for human consumption. It details the series of steps I took to successfully install and compile a custom Linux Kernel, Xen 4.2 unstable, and run three virtual machines to perform unique tasks all on a single physical computer.

To achieve the desired functionality, I used IOMMU for PCI Passthrough with a multimedia operating system, granting complete access to a graphics card for GPU Acceleration.

I am a college student, not a trained professional, and I am sharing this documentation for educational purposes. Blind use of this document for a production environment in a business setting would be ill-advised.

---

To quickly summarize my experiences, I decided to try Xen and began researching in early January 2012.

I purchased equipment in March 2012, and began what I thought would take "at most two weeks".

By late April I finally had a (mostly) functional system.

I have been fine tuning the system and my process since then to produce this guide for others.

The purpose of this guide is to turn a 6 month project into a series of steps that can be reproduced inside the time frame of 1-2 days.

---

Objectives Overview

---

I wanted a single physical computer, that could handle three computers worth of separated activities at all times, including these four specifics:

• Router/Firewall (PFSense)
• DNS & Web Development (Debian Squeeze)
• Application Development (Debian Wheezy)
• Multimedia & Gaming (Windows 7)

I investigated alternative software, including VMWare's ESXi and Citrix's XenServer.

I had come from a VMWare platform having used both VMware Server 2 and VMWare Workstation 8 for the same key objectives previously but with a Windows Host, subject to Windows Updates which let to my investigation.

ESXi was easy to install, but missing numerous drivers for hardware components. I quickly ruled it out as I wanted a home-use system, and there was no convenient or well-documented method to installing or even launching a virtual machine from their on-server interface.

Citrix's XenServer was easily my favorite of the options, it's design made for a great user experience. Everything was easy to find and do. However it came with a minimum price tag of $1000, and if I wanted PCI passthrough for graphics cards $2500. This was beyond my reach as a college student.

I chose Xen since it had support for my hardware, was a free open-source project, and had a great community with lots of activity.

---

Things you Will Need:

• Compatible Hardware
• ATI Graphics Card
• Motherboard with UEFI & VT-d & Onboard Graphics
• Latest Ubuntu Live DVD
• Debian Wheezy Beta1 (Or Newer) Installer

nVidia Cards can be made to work, with extensive patching, in Windows XP and supposedly Windows 8 Preview, for more information visit David Techer's blog.

If you plan to pass your graphics card to a virtual machine, you will need either a second graphics card, onboard graphics, or a second computer to manage your Dom0 system and install virtual machines over VNC.

UEFI compatible boot DVD will save you an undocumented step for setting up a UEFI boot loader.

---

Xen is picky, different hardware may yield different results, both at compile time and runtime. If you want to save yourself some hassle, here is my hardware list and some suggestions to avoid:

• Motherboard:
  • ASRock Z68 Extreme4 Gen3
• CPU:
  • Intel Core i2600
• RAM:
  • 12GB 1333Mhz Corsair XMS (2x2G 2x4G)
• Boot Disk:
  • 240GB OCZ Vertex 3
• GPU:
  • ATI Radeon HD 6870
• LAN:
  • Onboard Broadcom BCM57781
  • PCIe EXPI9301CTBLK

Hardware and Configurations to Avoid:

• NF200 Chipsets are not IOMMU compatible
• nVidia Graphics Cards
• RAID5 yields horrible performance

NF200 is a PCI Switch for motherboards sporting SLI and CrossFire, avoid it if you want be able to use those PCI slots for passthrough.

---

UEFI Configuration

---

These instructions are for an Intel CPU and ASRock UEFI Motherboard, and may vary depending on your manufacturer as well as your choice of CPU. If you are unfamiliar with motherboard configuration, you may want to watch the video for a visual walk through.

Before starting reset your CMOS so you have a clean slate to work from.

List of key settings:

• Turn Legacy USB 3.0 off
• Turn on VT-x and VT-d
• Set drives to ACHI Mode
• Change default video to Onboard

Justification:

Consider yourself warned, if you leave Legacy USB on and leave a backup USB drive connected, your system will fail to boot. To my surprise that is because it is trying to boot from that USB drive (even if that drive is USB 2.0), which threw me off so I recommend disabling that feature.

Most motherboards will have Intel Virtualization enabled by default (VT-x), but that is not the same as IOMMU, be sure to look for VT-d or check the manual for details. For my system VT-x was in the CPU configuration area, and VT-d was in the Northbridge configuration area.

ACHI is a superior choice for performance with modern hard drives, and while RAID is an alternative that uses ACHI as its underlying type, it increases boot time by checking RAID configurations, and most onboard RAID is software RAID that is built for Windows and rarely helpful for Linux.

ASRock boards will use a PCI GPU by default if it is plugged into a PCI slot, regardless of if any video cables are connected, and if you plan to pass that device to a virtual machine, you do not want to use it for video.

---

Wheezy Installation

---

Select Advanced Options from the menu, and then Advanced Install.

To expedite this process we can select defaults for most options (such as for networking).

Once we reach Hard Drive Partitioning options, select Manual at the bottom.

Select the disk(s) we will be using by title not partition space and hit enter, we can now select gpt from the partition tables instead of msdos.

We will then create three partitions:

• 256MB FAT32 Partition mounted to /boot/efi
• 256MB Ext4 Partition mounted to /boot
• Remainder LVM Partition

Select Configure the Logical Volume Manager and create a volume group.

Create three Logical Volumes:

• 8GB or More for Linux
• 20GB or More for Home
• 2GB or More for Swap

Complete LVM Configuration, and then back to our partition settings we can now select formats and mount points:

• 8GB LV Ext4 for root (/)
• 20GB LV Ext4 for /home
• 2GB LV swap

Optionally add noatime flags to all the Ext4 partitioned disks.

Complete the Partition steps by selecting done and create the partitions.

Continue through the options with defaults, until we get to the main Packages.

De-select Debian Desktop Environment and continue

Do not install a boot loader, select the Continue without a bootloader option and be sure to jott down the information it provides us for later.

Complete the installation and restart our computer.

Insert an Ubuntu Live DVD (or live USB created from the DVD iso) and press whatever key opens your motherboard's boot menu.

Select the UEFI boot option from the boot menu.

When presented with the Grub menu screen, press c on the keyboard to access the Emergency Grub Console.

Emergency Grub allows us to manually boot our Linux, since we did not install a boot loader.

Type ls to list the disks, we should have an (hd0) and three gpt partitions. If booting off a Live USB these will appear under (hd1) instead as the USB drive is (hd0).

Assuming the same partitioning as mine, in the below example tab indicates hit the tab key to auto-complete, and we can either use the data provided when we selected Continue without a bootloader or just substitute xen for the name we gave our Volume Group name, and linux for our root (/)Logical Volume:

grub> set root=(hd0,gpt2)
grub> linux /vmlinu_tab_ ro root=/dev/mapper/xen-linux
grub> initrd /initrd_tab_
grub> boot 

Gibberish may be displayed when tabbing to auto-complete, this is normal and can be ignored. After entering boot it will begin starting Linux.

If done successfully a terminal login will be presented, which concludes this segment of the guide.

In the next segment we will start by enter our credentials and we can start installing some key packages to get UEFI working.

---

References

---

EFI Grub on Wheezy

---

Wheezy Configuration

---

Note:

I have done my best to separate my crazier desires from the average persons needs.

To do this I created bold headers with line breaks for special sub-sections, whose subsequent contents you can skip if said bold text does not apply to your situation.

---

Assuming you have just installed Debian Wheezy and are at the login prompt, you want to login as root so you can install packages (alternatively you can use su to change users if you prefer).

---

UEFI STEPS

Assuming you came from my previous guide and have just manually booted the Debian system in EFI mode. Your first objective should be getting an efi bootloader installed:

aptitude install grub-efi-amd64 

Once installed you will want to run the install and update scripts:

grub-install
update-grub 

Successful installation of the boot manager will result in a list of the boot order with an entry "debian" listed. If this didn't work then you may not have booted in EFI mode.

END UEFI STEPS

---

Now let's install a couple more basic packages:

aptitude install ssh sudo 

With ssh installed we can, optionlly, access remotely for the remainder of the guide. With sudo we can add our user to the sudoers group like so (cdelorme is my username):

usermod -aG sudo cdelorme 

If you omit the a you will replace all existing groups your user belongs to, so be careful.

I recommend rebooting now to get ssh working, and also so you can login as your user to use the sudo command (if you used su you may need to logout and log back in for sudo to work).

Next we want to prime our server with some general utility packages:

• parted
• samba
• ntp
• screen
• p7zip-full

Delicious Copy Pasta:

sudo aptitude install parted samba ntp screen p7zip-full 

Note: I used to install ntfsprogs but the package is currently bugged and eats CPU cycles, so I would avoid it if possible.

Configuration of samba and ntp will vary by system, I recommend using other guides online for this since as long as they work they will be fine when we start using Xen. If you want my configurations specifically, ask and I may consider adding them to a future edition of this guide.

---

Development Environment Steps

Since I intend to use Debian Wheezy as a development environment, I also installed these packages:

• openjdk-7-jdk
• ant
• g
• libboost-dev
• gnustep
• python
• python-dev

Delicious Copy Pasta:

sudo aptitude install openjdk-7-jdk ant g   libboost-dev gnustep python python-dev 

End Development Environment Steps

---

GUI Environment Steps

To be honest, I don't care for the CD Installation of Gnome3, it's less work but the cost is 30 minutes of installation time, and a whole slew of packages I never use, so I went minimalist and found exactly what I needed:

• gnome-session
• gnome-terminal
• gnome-screensaver
• xserver-xorg-core
• gdm3
• gksu
• ia32-libs
• ia32-libs-gtk
• libc6-dev-i386
• binfmt-support

Delicious Copy Pasta:

sudo aptitude install gnome-session gnome-terminal gnome-screensaver xserver-xorg-core gdm3 gksu ia32-libs ia32-libs-gtk libc6-dev-i386 binfmt-support 

If you intend to use Xen with SDL later on, I recommend replacing xserver-xorg-core with xorg and xorg-dev packages, as these are what we will install later, and they contain the xserver-xorg-core.

As a bare-bones install of Gnome3, it should take roughly 5 minutes as opposed to 30.

At this stage you can run startx to launch the GUI environment, but I generally reboot to test the login screen.

End GUI Environment Steps

---

Optional GUI Additions

Optional additions include enabling audio, installing some basic multimedia, utilities, and important software like Guake terminal, Google Chrome, and Sublime Text.

Starting with Audio:

sudo aptitude install alsa-base
sudo alsactl init 

Now with GUI utilities:

• python-xdg
• font-manager
• gnome-disk-utility
• gnome-screenshot
• guake

Delicious Copy Pasta:

sudo aptitude install python-xdg font-manager gnome-disk-utility gnome-screenshot guake 

Note: Guake has a bug with Gnome3 that prevents it from starting at boot, here is how you can fix it:

This is actually a very simple edit, in this specific version of guake (located at /usr/bin/guake) line 653 contains a notification.show(), we just have to wrap it in a simple try-catch so it doesn't crash when it is run before the GUI is ready to accept a pynotify event:

try:
notification.show()
except Exception:
pass 

Next we want to set Guake to start at login. You can make an auto-start file via:

mkdir ~/.config/autostart
cd ~/.config/autostart
echo "[Desktop Entry]" >> guake.desktop
echo "Name=Guake Terminal" >> guake.desktop
echo "Name[pt]=Guake Terminal" >> guake.desktop
echo "Name[pt_BR]=Guake Terminal" >> guake.desktop
echo "Comment=Use the command line in a Quake-like terminal" >> guake.desktop
echo "TryExec=guake" >> guake.desktop
echo "Exec=guake" >> guake.desktop
echo "Icon=/usr/share/pixmaps/guake/guake.png" >> guake.desktop
echo "Type=Application" >> guake.desktop
echo "Categories=GNOME;GTK;Utility;TerminalEmulator;" >> guake.desktop 

For google chrome we need to add a new debian source to /etc/apt/sources.list, and download their package key:

su root
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
aptitude update
exit 

Next we can install these packages:

• google-chrome-unstable
• eog
• gnash
• vlc
• gtk-recordmydesktop

Delicious Copy Pasta:

sudo aptitude install google-chrome-unstable eog gnash vlc gtk-recordmydesktop 

Note: I am aware that google source is for Squeeze, I have had no problems running it in Wheezy.

After google-chrome has been installed, it will create its own sources.list file entry, so we want to go back in and remove the line we added from /etc/apt/sources.list, and re-run sudo aptitude update to make sure it works.

Finally, I am in love with Sublime Text 2, so I went to their website for the download, and went through these steps for a complete installation.

Once downloaded, let's extract it and move it to /usr/local, and create a new command to run it from terminal:

tar xf sublime-text.tar.bz2
sudo mv sublime-text /usr/local/
cd /usr/local/sublime-text
sudo ln -s /usr/local/sublime-text/sublime_text /usr/bin/subl 

Next let's create a desktop file so we can run it as an application and set default mimetypes from the GUI. Let's navigate to /usr/share/applications/ and create a new file subl.desktop with these contents:

[Desktop Entry]
Name=Sublime Text 2
Comment=The Best Text Editor in the World!
TryExec=subl
Exec=subl
Icon=/usr/local/sublime-text/Icon/256x256/sublime_text.png
Type=Application 

Next to set the mimetypes we add these lines to /usr/share/applications/defaults.list:

text/plain=subl.desktop
text/css=subl.desktop
text/htm=subl.desktop
text/javascript=subl.desktop
text/x-c=subl.desktop
text/csv=subl.desktop
text/x-java-source=subl.desktop
text/java=subl.desktop 

There are user-account alternatives that may seem more appropriate, but I wanted these to apply system-wide.

End Optional GUI Additions

---

Thus concludes our Debian Wheezy configuration, and we are now ready to move onto compiling a Custom Linux Kernel.

---

References

---

Missing ia32-libs-gtk for Gnome3

Samba Config

---

Compiling a Custom Linux Kernel

---

Compiling a custom Linux Kernel will make PCI passthrough much easier, but it also allows us to improve performance in various areas for Dom0.

For starters let's create a folder for compiling our packages in our home directory:

mkdir -p ~/src/linux
cd ~/src/linux 

Next we will install the required packages:

• wget
• tar
• bzip2
• build-essential
• libncurses-dev
• kernel-package
• fakeroot

Delicious copy pasta:

sudo aptitude install wget tar bzip2 build-essential libncurses-dev kernel-package fakeroot 

To improve compiling speed we can set the concurrency to double the number of physical CPU cores by running these commands:

su root
echo "CONCURRENCY_LEVEL=8" >> /etc/kernel-pkg.conf
exit 

IMPORTANT NOTE: Kernel 3.5 and 3.5.2 are bugged^[1] and passthrough with USB Controllers fails (though VGA Passthrough works for some odd reason). That is why this guide is using 3.4.9, however 3.3.8 has also been tested and works.

Our next step is downloading, extracting, and inserting the current kernel configuration before adjusting the kernel we are about to compile:

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.9.tar.bz2
tar jxf linux-3.4.9.tar.bz2 && rm linux-3.4.9.tar.bz2
cd linux-3.4.9/
cp /boot/config-3.2.0-3-amd64 .config
make menuconfig 

Configuring a kernel is not as complex as it seems, the menu is just a hierarchy of options, and thankfully you can search for settings with / and get details on individual settings with h.

Here is a tree-map list of all the important settings we need to change:

• Processor Type and Features
  • Preemption Model (Voluntary Kernel Preemption (Desktop))
    • Preemptible Kernel (Low-Latency Desktop)
  • Enable cleancache driver to cache clean pages if tmem is present
  • Timer frequency (250 HZ)
    • 1000 HZ
• Bus options (PCI etc.)
  • PCI Stub driver
  • Xen PCI Frontend
• Device Drivers
  • Block devices
    • Xen virtual block device support
    • Xen block-device backend driver
  • Network device support
    • Xen network device frontend driver
    • Xen backend network device
    • Ethernet driver support
      • QLogic devices
        • NetXen Multi port (1/10) Gigabit Ethernet NIC
  • Xen driver support
    • Xen memory balloon driver
    • Dynamically self-balloon kernel memory to target
    • Memory hotplug support for Xen balloon driver
    • Scrub pages before returning them to system
    • Xen /dev/xen/evtchn device
    • Backend driver support
    • Xen filesystem
    • Create compatibility mount point /proc/xen
    • Create xen entries under /sys/hypervisor
    • userspace grant access device driver
    • User-space grant reference allocator driver
    • Xen PCI-device backend driver
• Firmware Drivers
  • EFI Variable Support via sysfs

---

For SSH Users

For users using ssh, you will want to use screen, this way if your machine goes to sleep or your connection is lost while compiling the process will not halt in the middle.

Starting screen couldn't be easier:

screen 

End For SSH Users

---

Now we can clean the system and then compile our kernel:

make-kpkg clean
fakeroot make-kpkg --initrd --revision=3.4.9.custom kernel_image 

Note that in later versions of gcc the revision must begin with a number.

On my machine it takes roughly 20 minutes to compile a kernel, so feel free to take a break.

Once the compilation is complete, we can back up a directory where we should now find a portable .deb installation package. I highly recommend making a backup of this file if you want to reinstall that kernel without all this effort in the future:

cd ..
sudo dpkg -i linux-image-3.4.9_3.4.9.custom_amd64.deb 

At the end of the installation, it should automatically run update-grub, at which point you can reboot your system.

Once the machine has rebooted and you are logged in, type uname -r and if the value it spits out is 3.4.9 (or whatever kernel version you compiled) then you are successful!

Strap in, because the next segment will cover compiling the latest Xen from source.

---

References:

---

1. ^  Bug 46341 - Xen PCI passthrough does not work on kernel 3.5.2

Kernel Source

Amazingly Simple Kernel Compiling Guide

Kernel Flag Database for Lookup

Performance Regarding Paravirt Spinlocks

---

Compiling and Installing Xen

---

I have great news, the latest Xen source, RC3 currently, was the first time I have been able to compile the source without debugging make errors since I started back in March.

Before we proceed I should run through the fact that Xen can be effected by:

• Hardware
• Environment (Dom0 OS)
• Software
• Xen Source
• Dependent Packages

Xen is a spiderweb of dependencies simply due to the number of facets it has to fill, and we have control over barely half of these major factors.

My point is primarily that you can use the same hardware, platform, and revision I use and still encounter brand new errors due to changes in dependent packages downloaded during compilation. Be prepared.

---

The following set of packages is a bare-minimum set to compile and run Xen successfully, and omits well over a gigabyte of the recommended dependencies:

sudo aptitude install bridge-utils build-essential libncurses-dev python-dev uuid uuid-dev libglib2.0-dev libyajl-dev bcc gcc-multilib iasl libpci-dev mercurial flex bison libaio-dev 

Note: libpci-dev replaces pciutils-dev (which routes to said package), just an FYI.

These packages appear to be on the requirements list, but were already on my system at this stage. If you have not been following from the previous segment you may want to run this for good measure;

sudo aptitude install bcc patch libncurses5-dev python python-dev libglib2.0-dev bin86 bzip2 module-init-tools make gcc libc6-dev libcurl3 iproute xz-utils 

---

GUI Xen Packages

For those of us using Gnome or a GUI environment, you may want to include these packages for vnc access and/or sdl:

sudo aptitude install libsdl-dev gvncviewer 

Obviously you have the choice to use only one, you do not need both, but if you intend to use the GUI environment in Debian I highly recommend you give SDL a spin, it's much faster than VNC, though VNC does offer remote access.

End GUI Xen Packages

---

Complete The Dependencies

The list I already provided is a minimalists list, containing only exactly what was required to compile and run Xen for my own purposes, which includes PVHVM systems of Windows, FreeBSD, and Linux, as well as PCI Passthrough, and both VNC and SDL. It has worked for me since I started tinkering back in March, with minor modifications since.

For those who want the whole package and have a spare gigabyte of OS Disk space available, I have put together this sub-section.

I should also note that some of these packages will change flags marked as no from ./configure to a yes, but that did not stop me from compiling and running the system:

• zlib1g-dev
• libbz2-dev
• liblzo2-dev
• e2fslibs-dev
• ocaml
• ocaml-nox
• ocaml-findlib
• lzma
• lzma-dev
• liblzma-dev
• markdown
• libcurl4-openssl-dev
• transfig
• tgif
• libvncserver-dev
• python-twisted
• libjpeg62-dev
• gawk
• git-core
• texinfo
• texlive-latex-base
• texlive-latex-recommended
• texlive-fonts-extra
• texlive-fonts-recommended

If anyone knows exactly what any of these dependencies do or are used for and can shed some light on them I would greatly appreciate it.

End Complete The Dependencies

---

Next we can visit xenbits to get the latest revision. In our case we will use revision 25824^[2][3], and use mercurial to download the source:

mkdir -p ~/src/xen && cd ~/src/xen
rev=25824
hg clone -r $rev http://xenbits.xen.org/hg/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
cd xen-unstable.hg-rev-${rev} 

2. ^  Original guide suggests using Xen 4.2-rc4-pre, revision 25777. Xen 4.2-rc4, revision 25824 has also worked for another user, User:V
3. ^  Xen 4.2-rc4, revision 25784 compiled with an error for a potentially uninstantiated variable, which may suggest a gcc compatibility bug

---

EFI Source Modification

Given that I encountered no make errors, I was shocked to find that they still hadn't addressed the grub efi compatibility bug where it fails to recognize available system memory.

The fix for this is a source modification, though they have supposedly added a build process for xen.efi, which would replace the debian.efi generated by grub. I have yet to find adequate instructions to make that work, and even if I did may break adding xen-pciback.hide to the grub configuration, which we need if we are planning to do the easy method of PCI Passthrough.

So, let's open up xen/arch/x86/setup.c with our favorite text editor, and find the line containing ( e820_raw_nr != 0 ), and make that area look like this:

#if 0
 else if ( e820_raw_nr != 0 )
 {
 memmap_type = "Xen-e820";
 }
 else if ( bootsym(lowmem_kb) )
 {
 memmap_type = "Xen-e801";
 e820_raw[0].addr = 0;
 e820_raw[0].size = bootsym(lowmem_kb) << 10;
 e820_raw[0].type = E820_RAM;
 e820_raw[1].addr = 0x100000;
 e820_raw[1].size = bootsym(highmem_kb) << 10;
 e820_raw[1].type = E820_RAM;
 e820_raw_nr = 2;
 }
 #endif
 else if ( mbi->flags & MBI_MEMMAP ) 

Notice we are commenting out a section of code using "if 0", which always evaluates to false, to eliminate the section of e801 mapping which breaks our ram recognition.

I have no idea what kind of adverse effects this has on other systems, but it has worked for me since Xen 4.1.2.

End EFI Source Modification

---

SSH Users

Those of us using SSH to follow along should now run screen so we don't loose our footing while compiling.

End SSH Users

---

We can now run the configuration script and compile the source:

./configure && make -j4 world && make -j4 deb 

Notice the -j4 flag which sets concurrency for us, with that setting I have compiled Xen ins just under 7 minutes. It will consume roughly 1GB of space once fully compiled.

Instead of make install I used make deb to create a debian package which we can backup for future installations, and to simplify removal. This file will be placed in the dist/ folder.

At this stage if you don't see any make errors and the debian file is in the dist folder, you have successfully compiled Xen and are ready to move to the next phase of installing and configuring Xen!

---

References:

---

Xenbits Repositories

Latest Instructions

4.1 Compilation Instructions

4.0 Compilation Instructions

David Techer's Blog

EFI Memory Error

---

Xen Configuration

---

In the previous segment we had just finished compiling Xen to the dist/ folder, which is where we will go now to install it!

cd dist/
sudo dpkg -i xen-upstream-4.2.0-rc4.deb  

Our work is not done yet, we have a lot of post-install configuration to take care of before we can reboot and verify that Xen is working.

Let's start with a trip to /boot to delete extra symbolic links and related files:

cd /boot
sudo rm xen.gz xen-4.gz xen-4.2.gz xen-syms-4.2.0-rc4  

Next we want to have Xen boot before the Linux Dom0 kernel, Xen will load that after. To do this we need to move the grub configuration file forward and update grub via:

sudo mv /etc/grub.d/20_linux_xen /etc/grub.d/09_linux_xen
sudo update-grub  

Finally, before we reboot the system we need to make sure we initialize the Xen toolstack and related features at boot time via:

sudo update-rc.d xencommons defaults 20 19
sudo update-rc.d xendomains defaults 21 22
sudo update-rc.d xen-watchdog defaults 23 22  

At this stage we reboot our system, and if all goes well the first option in grub will be Xen, which will bring us to our linux login.

Our next step is logging in and testing the xl toolstack with sudo xl dmesg, if the output you receive is a not an error, then you have successfully installed Xen.

Next we need to setup a virtual bridge for our virtual machines going forward. Here is a simple recommended configuration you can use to replace the contents of /etc/network/interfaces (See my PFSense Segment for a more complex multi-bridge configuration):

auto lo xenbr0
iface lo inet loopback
iface eth0 inet manual
iface xenbr0 inet dhcp
bridge_ports eth0  

To make this change take effect you can restart the network service, ifconfig, or take my approach and reboot.

---

Making Things Easier

As many are aware, security comes at the cost of usability.

So this section is a way to make using Xen easier at the slight cost of some security.

Add one of the following segments of code to the end of /etc/sudoers to omit entering a password when running the xl command:

Option 1:

# Add no-password xl access
%sudo ALL=(ALL:ALL) ALL, !/usr/sbin/xl, NOPASSWD: /usr/sbin/xl

Option 2:

# Cmnd alias specification
Cmnd_Alias XL = /usr/sbin/xl
# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL, !XL, NOPASSWD: XL </nowiki>

If using vi you may have to use :wq! to force write quit when saving.

Next let's add an alias so whenever we type xl it assumes sudo xl by adding an alias to /etc/bash.bashrc:

# XL Alias
 alias xl='sudo xl'  

Now when we login next instead of using sudo xl and entering a password, we can just use xl and no password is necessary!

End Making Things Easier

---

This concludes Xen Configuration, we are ready to move onto setting up HVM Virtual Machines!

I recommend checking the table of contents for details on managerial activities, which includes backing up and restoring, LVM and partitioning, and a few other helpful related activities.

---

References

---

Xen Man Pages

Outdated Document Mentions Xen Syms File

---

Installing Windows HVM

---

While you are welcome to follow along, I highly recommend you spend a few hours reviewing the Xen Man Pages.

Our first step is preparing a Logical Volume partition for Xen.

With Xen, this partition will be treated as a whole hard drive, and will create a sub-partition-table and partitions inside of it, hence you cannot simply mount the logical volume without an offset going forward. For more information on this, check out the managerial tasks section of my documentation.

I recommended Samba because there are drive limitations with Xen, if you intend to share a partition between multiple machines mounting with write permissions can ruin your data. Xen will allow multiple mounts, but neither system will inform the other of changes, hence conflicting read and write operations.

Other disk limitations include only up to 4 hdX drives, which are treated as Parallel ATA, and only up to 15 sdX drives which are treated as SCSI can be passed at any time.

You will need a Windows 7 installation DVD, I recommend a disk image copy, which we can use to install Windows 7.

For now we are using xenbr0 for our network bring, but when we set the vif in our configuration it must have a mac address with a prefix of 00:16:3e. For this HVM I will be using the emulated Intel e1000 gigabit ethernet adapter.

Here is my Windows 7 configuration file for installation:

name='windows'
builder='hvm'
vcpus=4
memory=6144
disk=[
'/dev/mapper/xen-win2,,hda,w',
'/media/samba/docs/win7.iso,,hdc,r,devtype=cdrom'
]
vif=[
'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
]
boot='dc'
pae=1
nx=1
videoram=16
stdvga=1
sdl=1
vnc=0
usb=1
usbdevice="tablet"
localtime=1

Once Windows has been installed we can change boot='dc' to boot='c', and remove the iso from the drives.

Changes to the configuration will not take effect until we have shut down the HVM and run the xl create command again.

---

Passthrough

Passthrough is an important and moderately complex topic.

First we should cover the basics of PCI devices in Linux.

Linux recognizes devices using a BDF code, which stands for Bus, Device, Function. You will commonly see it in the format of BB:DD.f, and it is how we point to devices when we want Linux to do something with them.

To check your systems PCI device identifiers you can use lspci. This will print them in BDF format. You can get more details by adding the verbosity flag -v, you can add up to triple verbosity, which is several pages of in formation.

We usually just need basic names to identify our devices, here is an example lspci from my ASRock Z68 Extreme4 Gen3 Motherboard:

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6800 Series]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Barts HDMI Audio [Radeon HD 6800 Series]
03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9120 SATA 6Gb/s Controller (rev 12)
04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
06:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
07:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
09:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
0b:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)

Unfortunately USB devices are not labeled on the board as they are seen in the lspci printout, but I have 2 of each type of controller, and which items are connected to which controller is unknown. I had to spend some time testing which controller was which by passing one at a time to a virtual machine. When passed I could not use them in Dom0, so be sure you have some form of remote control or a PS/2 keyboard and mouse as a backup.

I tested my USB 3.0 ports in linux and had zero gains in speed, so I decided to pass both USB 3.0 controllers and the larger USB 2.0 controller to the pciback module for passthrough.

I ended up with these five items as shown in BDF format:

• 00:1d.0 (USB 2.0)
• 01:00.0 (Graphics Card Video Function)
• 01:00.1 (Graphics Card Audio Function)
• 04:00.0 (USB 3.0)
• 05:00.0 (USB 3.0)

Notice that the graphics card has two functions, video and audio. If I pass one I have to pass both, I cannot pass just Video and keep audio accessible in Dom0. This creates a serious problem when you have extremely dense motherboards that have shared PCI Bridges with multiple functions leading to several individual components. It may be important to keep that in mind when purchasing a motherboard, this is not information you will have until the board is installed and you are running the lspci command.

Passing these devices to pciback essentially tells the Linux OS not to assign drivers and use or activate them. However if you run lspci after hiding these devices, you will still see them.

There are two methods to using pciback, one is as a module but I have had no luck with that approach hence why we compiled a custom kernel. With a custom kernel my pciback is built-in and I can modify my grub bootloader to automatically hide the devices.

Find this section of your grub configuration in the file /boot/grub/grub.cfg:

submenu "Xen 4.2.0-rc4-pre" {
menuentry 'Debian GNU/Linux, with Xen 4.2.0-rc4-pre and Linux 3.4.9' --class debian --class gnu-linux --class gnu --class os --class xen {
insmod part_gpt
insmod ext2
set root='(hd0,gpt2)'
search --no-floppy --fs-uuid --set=root 9b2dde6d-5888-429c-a6dd-35e7fa11f632
echo    'Loading Xen 4.2.0-rc4-pre ...'
multiboot       /xen-4.2.0-rc4-pre.gz placeholder
echo    'Loading Linux 3.4.9 ...'
module  /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro  quiet
echo    'Loading initial ramdisk ...'
module  /initrd.img-3.4.9
}

The line we want to modify starts with module, and here is what we add:

module  /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro  quiet xen-pciback.hide=(00:1d.0)(01:00.0)(01:00.1)(04:00.0)(05:00.0)

We will have to force write-quit to save the file (:wq!), then when we reboot the system our devices will be available for passthrough!

Next we add this to our HVM Configuration File:

pci = [
'00:1d.0',
'04:00.0',
'05:00.0',
'01:00.0',
'01:00.1'
]

It is important to note that you may NOT want to pass through devices until after you have installed and updated the virtual machine. I have encountered many problems with passing devices during installation.

---

Important Note for Graphics Passthrough

nVidia cards do not work out of the box, but ATI does.

However, most cards do not support FLR (Function Level Reset), which is important because it prevents the card from operating properly without the right steps.

Essentially, unlike a physical computer, when the virtual machine is shut down or restarted, it does not change the power supplied to the graphics card. This means the graphics card is never re-initialized, so FLR was invented. However, because most graphics cards do not support FLR you have to mimic this effect by ejecting the card after every reboot, otherwise you will see a severe performance degradation.

In addition, if you do not reset the card, and it is not fresh when you attempt to install or uninstall drivers, the process may fail leaving your system crippled, plus a BSOD is likely to appear.

So, my recommendation when dealing with GPU drivers and passthrough, always reboot the entire machine, or take extra care to reset the card before making any changes.

After the installation, if you reboot the HVM you can use the Eject USB Media icon in the taskbar at the lower right hand corner of the screen to eject the card which will attempt a manual reset. You will loose video for a few seconds as the card re-initializes. This should fix performance on reboot.

---

PV on HVM Drivers are awesome

They can improve network and hard drive performance by a substantial amount. In Windows these are referred to as GPLPV Drivers.

To my understanding they bypass the emulated hardware layers reducing the latency between operations.

My Windows Experience Index with a SSD went from 6.8 for hard drive to 7.8 by installing these drivers.

Supposedly to enable PV on HVM you have to add this line to your HVM Configuration:

xen_platform_pci=1

The final configuration after my devices are passed and PVHVM Drivers are installed looks like this:

name='windows'
 builder='hvm'
 vcpus=4
 memory=6144
 disk=[
 '/dev/mapper/xen-win2,,hda,w'
 ]
 vif=[
 'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
 ]
 pci = [
 '00:1d.0',
 '04:00.0',
 '05:00.0',
 '01:00.0',
 '01:00.1'
 ]
 boot='c'
 pae=1
 nx=1
 videoram=16
 stdvga=1
 #sdl=1
 vnc=1
 vnclisten="0.0.0.0:10"
 vncpasswd='password'
 usb=1
 usbdevice="tablet"
 localtime=1
 xen_platform_pci=1

If you do not have ntp setup on the host machine (Dom0) do so immediately. Setting localtime=1 should use the correct host time, without it Windows clock will drift at boot and during runtime by a substantial amount.

Note that as unimportant as clock time seems, if it drifts far enough your cookies, sessions and SSL certifications will disappear, become invalid, and the operating system itself becomes absolutely unstable. For example all of my taskbar icons disappeared and were repalced with blank spaces, including my eject USB Media icon. It was a scary experience.

---

Since I use this system for recreational purposes, I have posted a performance demonstration video that shows HDD, SSD, and USB data transfer speeds, as well as graphics performance with video games.

Windows Performance Demonstration Video

---

References

---

Xen Man Pages

---

Installing PFSense HVM

---

WORKING ON THIS STILL

I have been re-using an imaged backup of a former configuration of PFSense for the last several installations, so I have not created updated documentation on installing and configuring PFSense yet.

I hope to have this completed within the next few weeks, and a video to match shortly after.

---

Managerial Activities

---

There are a few tricks I can share with regards to managing Xen virtual machines.

However, having an understanding of the fundamentals is also helpful.

Here are the key areas:

• Drive Partitioning
• LVM
• Using mount With LVM Virtual Partitions
• dd

---

Drive Partitioning

---

This is a fundamental, you are welcome to skip ahead if you want the how and not the why.

I find a lot of emails floating around the xen-users email list due to misunderstandings in how Xen recognizes and managed partitions.

It is important to understand how Xen Virtual Machines treat partitions.

It sees them as entire hard drives, and will create a brand new partition table and partitions inside of it.

The result is sub-partitioning, and this causes an offset between the start of the actual partition and the start of the sub-partition containing all of the data on the virtual machine.

---

In more detail

Computers typically segment hard drives into three components.

• Partition Tables
• Partitions
• File Systems

A partition table tells the disk where the beginning and end of a partition exists and usually the file system.

The Partitions are simple blocks of storage.

The File System tells the operating system how to access the partition.

A normal installation of Windows for example, will look like:

• Drive Block 0
  • Partition Table:
    • Starts at Block 256 File System NTFS
• Drive Block 256
  • Partition
    • Data

---

LVM

---

Linux has LVM handle all the tough parts of managing the positions of data blocks, and makes everything else easier for the user.

Hence it is called the Logical Volume Manager, because it manages logical volumes.

What this means is when we create a LVM Partition, we are actually using an LVM "File System", which looks like this:

• Drive Block 0
  • Partition Table:
    • Starts at Block 256 File System LVM
• Drive Block 256
  • Partition
    • Data

We can then create partitions using LVM which automatically handles the location of the data inside the LVM Partition.

For example, we can create two LVM Partitions for a Windows and Linux Virtual Machine, which would look like this:

• Drive Block 0
  • Partition Table:
    • Starts at Block 256 File System LVM
• Drive Block 256
  • Partition
    • Data
      • LV Windows
      • LV Linux

Only an operating system that recognizes LVM can see "LV Windows" or "LV Linux" though, so if you plug that drive into a basic Windows machine, it won't be able to do anything with the data.

---

Finally we have Xen

With Xen we give one of those partitions to our virtual machine, but as stated it sees that as a hard drive.

If we use the previous drive configuration and just add a Windows Virtual Machine to LV Windows our configuration looks like this:

• Drive Block 0
  • Partition Table:
    • Starts at Block 256 File System LVM
• Drive Block 256
  • Partition
    • Data
      • LV Windows
        • LV Block 0
          • Partition Table
            • Partition at Block 256 File System NTFS
        • LV Block 256
          • Partition
            • Data
      • LV Linux

At this point, we cannot simple access the data on our Xen partition because it has been sub-partitioned.

If you try simply using mount it will fail, because it requires the offset to the start of the NTFS sub-partition!

---

Using mount With LVM Virtual Partitions

---

To get the offset of the partition, we will use a tool like fsdisk or parted:

sudo parted /dev/mapper/xen-windows unit B print

Using unit B we indicate bytes which will be used by the mount command, and we get this:

Number  Start       End            Size           Type     File system  Flags
1      1048576B    105906175B     104857600B     primary  ntfs         boot
2      105906176B  171796594687B  171690688512B  primary  ntfs

Important to note that Windows 7 creates a 100MB Boot Partition, so the first item with the boot flag is actually the boot partition, we want the Start value of the second, which is 105906176B.

We can now mount the partition using:

sudo mount -o offset=105906176 /dev/mapper/xen-windows /path/to/mount/dir

That is how we can access the data inside our sub-partitions.

Obviously it is ill-advised to access the data while the operating system is running, mounted disks will NOT inform the running platform of changes, so data can get overwritten and cause all kinds of problems.

---

dd

---

The utility dd is probably one of my favorites among the tools that come with most linux/unix systems.

It can be used to create an exact copy of data, and is not subject to a number of limitations with higher level software.

It is not the fastest utility, but supplemented with tools like gzip you can compress a complete backup into a size similar to that you might see with professional software.

Here is a backup Example that compressed an 80GB Windows LV with 70GB of Data into a 43GB GZipped Image File:

sudo dd if=/dev/mapper/xen-windows of=/path/to/backup/windows.img bs=1M
sudo gzip -9 windows.img

Alternative one-liner (Takes a lot longer since it has to compress "on the fly"):

sudo dd if=/dev/mapper/xen-windows bs=1M | sudo gzip -9 > /path/to/backup/windows.img.gz

I can also use this command to restore my system from that compressed image:

sudo gzip -d /path/to/backup/windows.img.gz
sudo dd if=/path/to/backup/windows.img of=/dev/mapper/xen-windows bs=1M

Once again, one-liner alternative:

sudo gzip -dc /path/to/backup/windows.img.gz | sudo dd of=/dev/mapper/xen-windows bs=1M

There are plenty of other uses I have found for the dd command, but backups and restoration make Linux amazingly easy to restore after terrible failure, and believe me in writing this guide I encountered that plenty of times.

---

You've reached the end of my guide!

Thanks for taking your time to read through my documentation, if you found it at all helpful please feel free to shoot me an email on the Xen Users email list.

Any feedback or corrections are also welcomed.

I am still working on the PFSense segment and will have that up as soon as I can.