FreeBSD Dom0

This guide explains how to setup a FreeBSD Dom0 for testing purposes. In order to set it up a box with an Intel CPU that supports EPT and has a IOMMU is needed. This guide assumes that the user has already installed a plain version of FreeBSD amd64 into the box (either a RELEASE or a snapshot of HEAD or STABLE branches). In my specific case I'm using a stable/10 snapshot and the install has been done using ZFS-on-Root.

Please note that in order to run a FreeBSD PVH Dom0 an Intel box with EPT and a working IOMMU is required. Booting from UEFI is not yet supported, only legacy BIOS boot is supported at the moment.

FreeBSD version

Basic FreeBSD Dom0 support has been merged into FreeBSD HEAD. In order to follow this tutorial you need to be running HEAD revision r280954 or any later version.

Installing Xen

Before installing Xen we need to install the following dependencies:

# pkg install mercurial git bash python bcc glib pkgconf yajl gmake pixman perl5 markdown bison gettext gawk argp-standalone gcc48

Then the Xen source should be fetched using git:

# git clone git://xenbits.xen.org/xen.git

And finally we compile the Xen hypervisor and install the toolstack:

# cd xen
# ./configure HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
# gmake -j8 xen HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
# cp xen/xen /boot/
# gmake -j8 install-tools HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48

In order to have a working Xen host, the xencommons rc.d script has to be started, we are going to add it to /etc/rc.conf. We will also setup a bridge so our guests can have network and enable the Xen console in order to get a login prompt:

# echo 'xencommons_enable="YES"' >> /etc/rc.conf
# echo 'cloned_interfaces="bridge0"' >> /etc/rc.conf
# echo 'ifconfig_bridge0="addm bge0 SYNCDHCP"' >> /etc/rc.conf
# echo 'ifconfig_bge0="up"' >> /etc/rc.conf
# echo 'vm.max_wired=-1' >> /etc/sysctl.conf
# echo 'xc0 	"/usr/libexec/getty Pc"         xterm   on  secure' >> /etc/ttys
# mkdir -p /var/lock
# mkdir -p /var/run/xen

A new loader(8) option has been added in order to pass the boot parameters to the Xen kernel. In the following example Xen is setup to boot to the serial console, and Dom0 is limited to 4 vCPUs and 2G of RAM:

# echo 'xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1,vga"' >> /boot/loader.conf
# echo 'xen_kernel="/boot/xen"' >> /boot/loader.conf

If you are using Xen 4.6 or greater the following should also be added to /boot/loader.conf:

# echo 'hw.pci.mcfg=0' >> /boot/loader.conf

Now we are ready to reboot into the newly installed FreeBSD Xen Dom0.

Creating your first PV guest

In this example we will show how to create a Debian PV guest, first we need to fetch the Debian kernel, initramfs and config file:

# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/initrd.gz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/vmlinuz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/debian.cfg

For the disk we are going to use a ZFS ZVOL:

# zfs create -V 10g tank/debian

And the edited configuration file:

#============================================================================
# AT INSTALLATION TIME
#============================================================================

kernel = "vmlinuz"
ramdisk = "initrd.gz"

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

#bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Launch the guest and attach to the console:

# xl create -c debian.cfg

After the install has finished, destroy the guest:

# xl destroy debian

And edit the configuration file, so it looks like:

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Creating your first HVM guest

In this example we will see how to create a FreeBSD PVHVM guest running 10.0-RELEASE. The first step is to download the ISO image and create a zvol that will be used as the disk:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso
# zfs create -V 10g tank/freebsd

The following configuration file will be used:

# This configures an HVM rather than PV guest
builder = "hvm"

# Guest name
name = "freebsd"

# Initial memory allocation (MB)
memory = 512

# Number of VCPUS
vcpus = 4

# Network devices
vif = [ 'bridge=bridge0' ]

# Disk Devices
disk =	[
	'/dev/zvol/tank/freebsd,raw,hda,rw',
	'/root/FreeBSD-10.0-RELEASE-amd64-bootonly.iso,raw,hdc:cdrom,r'
	]

vnc = 1
vnclisten = "0.0.0.0"
serial="pty"

In order to run HVM guests the if_tap module has to be loaded. Then we can launch the guest and attach to it using vncviewer:

# kldload if_tap
# xl create freebsd.cfg
# vncviewer <addr>

When the install has finished the second disk device (hdc) can be removed from the configuration file.

External Articles

• OpenStack on FreeBSD/Xen Proof of Concept
• libvirt/libxl on FreeBSD

Shortcomings

• Missing MSI-X PV interrupt support (FreeBSD).
• PCI devices with virtual functions will not work correctly (FreeBSD).
• Missing migrate/save/restore support for guests (Xen).
• Missing PCI Passthrough support (Xen/FreeBSD).