FreeBSD Dom0

This guide explains how to setup a FreeBSD Dom0 for testing purposes. In order to set it up a box with an Intel CPU that supports EPT is needed. This guide assumes that the user has already installed a plain version of FreeBSD amd64 into the box (either a RELEASE or a snapshot of HEAD or STABLE branches). In my specific case I'm using a stable/10 snapshot and the install has been done using ZFS-on-Root.

Checking out the code and updating the system

FreeBSD Dom0 support is only available as a set of patches on top of the HEAD branch. The following is needed in order to check it out:

# pkg install git
# git clone git://xenbits.xen.org/people/royger/freebsd.git
# cd freebsd
# git checkout pvh_dom0_v6

Now the normal update world procedure from The FreeBSD Handbook should be followed.

Installing Xen

Before installing Xen we need to install the following dependencies:

# pkg install mercurial git bash python bcc glib pkgconf yajl gmake pixman perl5 markdown bison gettext gawk gcc47

Then the Xen source should be fetched using git:

# git clone git://xenbits.xen.org/xen.git

And finally we compile the Xen hypervisor and install the toolstack:

# cd xen
# ./configure HOSTCC=gcc47 CC=gcc47
# gmake -j8 xen HOSTCC=gcc47 CC=gcc47
# cp xen/xen /boot/
# gmake -j8 install-tools HOSTCC=gcc47 CC=gcc47

In order to have a working Xen host, the xencommons rc.d script has to be started, we are going to add it to /etc/rc.conf. We will also setup a bridge so our guests can have network and enable the Xen console in order to get a login prompt:

# echo 'xencommons_enable="YES"' >> /etc/rc.conf
# echo 'cloned_interfaces="bridge0"' >> /etc/rc.conf
# echo 'ifconfig_bridge0="addm bge0 SYNCDHCP"' >> /etc/rc.conf
# echo 'ifconfig_bge0="up"' >> /etc/rc.conf
# echo 'xc0 	"/usr/libexec/getty Pc"         xterm   on  secure' >> /etc/ttys
# mkdir -p /var/lock
# mkdir -p /var/run/xen

A new loader(8) option has been added in order to pass the boot parameters to the Xen kernel. In the following example Xen is setup to boot to the serial console, and Dom0 is limited to 4 vCPUs and 2G of RAM:

# echo 'xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1"' >> /boot/loader.conf

Now we are ready to boot into the newly installed FreeBSD Xen Dom0. In order to do this some manual interaction with the bootloader is needed. When the bootloader appears drop into the prompt and type the following:

OK unload
OK load /boot/xen
OK set module_path=/boot/kernel
OK load kernel
OK load zfs
OK load ...
OK boot

NOTE: in some cases (specially when using ZFS), it might be needed to increase the amount of allowed wired memory, this can be done by increasing the vm.max_wired sysctl tunable.

Creating your first PV guest

In this example we will show how to create a Debian PV guest, first we need to fetch the Debian kernel, initramfs and config file:

# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/initrd.gz > initrd.gz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/vmlinuz > vmlinuz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/debian.cfg > debian.cfg

For the disk we are going to use a ZFS ZVOL:

# zfs create -V 10g tank/debian

And the edited configuration file:

#============================================================================
# AT INSTALLATION TIME
#============================================================================

kernel = "vmlinuz"
ramdisk = "initrd.gz"

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

#bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Launch the guest and attach to the console:

# xl create -c debian.cfg

After the install has finished, destroy the guest:

# xl destroy debian

And edit the configuration file, so it looks like:

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Creating your first HVM guest

In this example we will see how to create a FreeBSD PVHVM guest running 10.0-RELEASE. The first step is to download the ISO image and create a zvol that will be used as the disk:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso > FreeBSD-10.0-RELEASE-amd64-bootonly.iso
# zfs create -V 10g tank/freebsd

The following configuration file will be used:

# This configures an HVM rather than PV guest
builder = "hvm"

# Guest name
name = "freebsd"

# Initial memory allocation (MB)
memory = 512

# Number of VCPUS
vcpus = 4

# Network devices
vif = [ 'bridge=bridge0' ]

# Disk Devices
disk =	[
	'/dev/zvol/tank/freebsd,raw,hda,rw',
	'/root/FreeBSD-10.0-RELEASE-amd64-bootonly.iso,raw,hdc:cdrom,r'
	]

vnc = 1
vnclisten = "0.0.0.0"
serial="pty"

In order to run HVM guests the if_tap module has to be loaded. Then we can launch the guest and attach to it using vncviewer:

# kldload if_tap
# xl create freebsd.cfg
# vncviewer <addr>

When the install has finished the second disk device (hdc) can be removed from the configuration file.

Shortcomings

• Missing MSI-X PV interrupt support (FreeBSD).
• PCI devices with virtual functions will not work correctly (FreeBSD).
• Missing migrate/save/restore support for guests (Xen).
• Missing PCI Passthrough support (Xen/FreeBSD).