Difference between revisions of "FreeBSD Dom0"

From Xen
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This guide explains how to setup a FreeBSD Dom0 for testing purposes. In order to set it up a box with an Intel CPU that supports EPT is needed. This guide assumes that the user has already installed a plain version of FreeBSD amd64 into the box (either a RELEASE or a snapshot of HEAD or STABLE branches). In my specific case I'm using a stable/10 snapshot and the install has been done using ZFS-on-Root.
+
This guide explains how to setup a FreeBSD Dom0 for testing purposes. In order to set it up a box with an Intel CPU that supports EPT and has a IOMMU is needed. This guide assumes that the user has already installed a plain version of FreeBSD amd64 into the box (either a RELEASE or a snapshot of HEAD or STABLE branches). In my specific case I'm using a stable/10 snapshot and the install has been done using ZFS-on-Root.
   
  +
{{WarningLeft|Please note that in order to run a FreeBSD PVH Dom0 an Intel box with EPT and a working IOMMU is required.
=== Checking out the code and updating the system ===
 
  +
Booting from UEFI is not yet supported, only legacy BIOS boot is supported at the moment.}}
   
  +
=== FreeBSD version ===
FreeBSD Dom0 support is only available as a set of patches on top of the HEAD branch. The following is needed in order to check it out:
 
   
  +
Basic FreeBSD Dom0 support has been merged into FreeBSD HEAD. In order to follow this tutorial you need to be running HEAD revision r280954 or any later version.
<pre>
 
# pkg install git
 
# git clone git://xenbits.xen.org/people/royger/freebsd.git
 
# cd freebsd
 
# git checkout pvh_dom0_v6
 
</pre>
 
 
Now the normal update world procedure from [https://www.freebsd.org/doc/handbook/makeworld.html The FreeBSD Handbook] should be followed.
 
   
 
=== Installing Xen ===
 
=== Installing Xen ===
Line 19: Line 13:
   
 
<pre>
 
<pre>
# pkg install mercurial git bash python bcc glib pkgconf yajl gmake pixman perl5 markdown bison gettext gawk gcc47
+
# pkg install mercurial git bash python bcc glib pkgconf yajl gmake pixman perl5 markdown bison gettext gawk argp-standalone gcc48
 
</pre>
 
</pre>
   
Line 32: Line 26:
 
<pre>
 
<pre>
 
# cd xen
 
# cd xen
# ./configure HOSTCC=gcc47 CC=gcc47
+
# ./configure HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
# gmake -j8 xen HOSTCC=gcc47 CC=gcc47
+
# gmake -j8 xen HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
 
# cp xen/xen /boot/
 
# cp xen/xen /boot/
# gmake -j8 install-tools HOSTCC=gcc47 CC=gcc47
+
# gmake -j8 install-tools HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
 
</pre>
 
</pre>
   
Line 45: Line 39:
 
# echo 'ifconfig_bridge0="addm bge0 SYNCDHCP"' >> /etc/rc.conf
 
# echo 'ifconfig_bridge0="addm bge0 SYNCDHCP"' >> /etc/rc.conf
 
# echo 'ifconfig_bge0="up"' >> /etc/rc.conf
 
# echo 'ifconfig_bge0="up"' >> /etc/rc.conf
  +
# echo 'vm.max_wired=-1' >> /etc/sysctl.conf
 
# echo 'xc0 "/usr/libexec/getty Pc" xterm on secure' >> /etc/ttys
 
# echo 'xc0 "/usr/libexec/getty Pc" xterm on secure' >> /etc/ttys
 
# mkdir -p /var/lock
 
# mkdir -p /var/lock
Line 53: Line 48:
   
 
<pre>
 
<pre>
# echo 'xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1"' >> /boot/loader.conf
+
# echo 'xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1,vga"' >> /boot/loader.conf
  +
# echo 'xen_kernel="/boot/xen"' >> /boot/loader.conf
 
</pre>
 
</pre>
   
  +
If you are using Xen 4.6 or greater the following should also be added to /boot/loader.conf:
Now we are ready to boot into the newly installed FreeBSD Xen Dom0. In order to do this some manual interaction with the bootloader is needed. When the bootloader appears drop into the prompt and type the following:
 
   
 
<pre>
 
<pre>
  +
# echo 'hw.pci.mcfg=0' >> /boot/loader.conf
OK unload
 
OK load /boot/xen
 
OK set module_path=/boot/kernel
 
OK load kernel
 
OK load zfs
 
OK load ...
 
OK boot
 
 
</pre>
 
</pre>
   
  +
Now we are ready to reboot into the newly installed FreeBSD Xen Dom0.
'''NOTE''': in some cases (specially when using ZFS), it might be needed to increase the amount of allowed wired memory, this can be done by increasing the vm.max_wired sysctl tunable.
 
   
 
=== Creating your first PV guest ===
 
=== Creating your first PV guest ===
Line 75: Line 65:
   
 
<pre>
 
<pre>
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/initrd.gz > initrd.gz
+
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/initrd.gz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/vmlinuz > vmlinuz
+
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/vmlinuz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/debian.cfg > debian.cfg
+
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/debian.cfg
 
</pre>
 
</pre>
   
Line 197: Line 187:
   
 
<pre>
 
<pre>
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso > FreeBSD-10.0-RELEASE-amd64-bootonly.iso
+
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso
 
# zfs create -V 10g tank/freebsd
 
# zfs create -V 10g tank/freebsd
 
</pre>
 
</pre>
Line 227: Line 217:
 
vnc = 1
 
vnc = 1
 
vnclisten = "0.0.0.0"
 
vnclisten = "0.0.0.0"
serial='pty'
+
serial="pty"
 
</pre>
 
</pre>
   
Line 239: Line 229:
   
 
When the install has finished the second disk device (hdc) can be removed from the configuration file.
 
When the install has finished the second disk device (hdc) can be removed from the configuration file.
  +
  +
=== External Articles ===
  +
* [http://empt1e.blogspot.co.uk/2015/06/openstack-on-freebsdxen-proof-of-concept.html OpenStack on FreeBSD/Xen Proof of Concept]
  +
* [http://empt1e.blogspot.ru/2015/06/libvirtlibxl-on-freebsd.html libvirt/libxl on FreeBSD]
  +
  +
=== Shortcomings ===
  +
  +
* Missing MSI-X PV interrupt support (FreeBSD).
  +
* PCI devices with virtual functions will not work correctly (FreeBSD).
  +
* Missing migrate/save/restore support for guests (Xen).
  +
* Missing PCI Passthrough support (Xen/FreeBSD).
   
 
[[Category:FreeBSD]]
 
[[Category:FreeBSD]]

Latest revision as of 09:41, 26 October 2016

This guide explains how to setup a FreeBSD Dom0 for testing purposes. In order to set it up a box with an Intel CPU that supports EPT and has a IOMMU is needed. This guide assumes that the user has already installed a plain version of FreeBSD amd64 into the box (either a RELEASE or a snapshot of HEAD or STABLE branches). In my specific case I'm using a stable/10 snapshot and the install has been done using ZFS-on-Root.

Icon Ambox.png Please note that in order to run a FreeBSD PVH Dom0 an Intel box with EPT and a working IOMMU is required.

Booting from UEFI is not yet supported, only legacy BIOS boot is supported at the moment.


FreeBSD version

Basic FreeBSD Dom0 support has been merged into FreeBSD HEAD. In order to follow this tutorial you need to be running HEAD revision r280954 or any later version.

Installing Xen

Before installing Xen we need to install the following dependencies:

# pkg install mercurial git bash python bcc glib pkgconf yajl gmake pixman perl5 markdown bison gettext gawk argp-standalone gcc48

Then the Xen source should be fetched using git:

# git clone git://xenbits.xen.org/xen.git

And finally we compile the Xen hypervisor and install the toolstack:

# cd xen
# ./configure HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
# gmake -j8 xen HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48
# cp xen/xen /boot/
# gmake -j8 install-tools HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48

In order to have a working Xen host, the xencommons rc.d script has to be started, we are going to add it to /etc/rc.conf. We will also setup a bridge so our guests can have network and enable the Xen console in order to get a login prompt:

# echo 'xencommons_enable="YES"' >> /etc/rc.conf
# echo 'cloned_interfaces="bridge0"' >> /etc/rc.conf
# echo 'ifconfig_bridge0="addm bge0 SYNCDHCP"' >> /etc/rc.conf
# echo 'ifconfig_bge0="up"' >> /etc/rc.conf
# echo 'vm.max_wired=-1' >> /etc/sysctl.conf
# echo 'xc0 	"/usr/libexec/getty Pc"         xterm   on  secure' >> /etc/ttys
# mkdir -p /var/lock
# mkdir -p /var/run/xen

A new loader(8) option has been added in order to pass the boot parameters to the Xen kernel. In the following example Xen is setup to boot to the serial console, and Dom0 is limited to 4 vCPUs and 2G of RAM:

# echo 'xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1,vga"' >> /boot/loader.conf
# echo 'xen_kernel="/boot/xen"' >> /boot/loader.conf

If you are using Xen 4.6 or greater the following should also be added to /boot/loader.conf:

# echo 'hw.pci.mcfg=0' >> /boot/loader.conf

Now we are ready to reboot into the newly installed FreeBSD Xen Dom0.

Creating your first PV guest

In this example we will show how to create a Debian PV guest, first we need to fetch the Debian kernel, initramfs and config file:

# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/initrd.gz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/vmlinuz
# fetch http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/netboot/xen/debian.cfg

For the disk we are going to use a ZFS ZVOL:

# zfs create -V 10g tank/debian

And the edited configuration file:

#============================================================================
# AT INSTALLATION TIME
#============================================================================

kernel = "vmlinuz"
ramdisk = "initrd.gz"

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

#bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Launch the guest and attach to the console:

# xl create -c debian.cfg

After the install has finished, destroy the guest:

# xl destroy debian

And edit the configuration file, so it looks like:

#============================================================================
# TO BOOT INSTALLED SYSTEM
#
# Comment all of the above installation options and uncomment the
# below instead
#============================================================================

bootloader="pygrub"

#============================================================================
# STANDARD OPTIONS
#============================================================================
#
# The following options are common to both installation time and normal booting.
#
# Only a subset of the available options are included below.
# See /usr/share/doc/xen-utils-common/examples for full examples.

#----------------------------------------------------------------------------

# Initial memory allocation (in megabytes) for the new domain.
memory = 512

# A name for your domain. All domains must have different names.
name = "debian"

# Number of Virtual CPUS to use, default is 1
vcpus = 2

#----------------------------------------------------------------------------
# Define network interfaces.

vif = ['bridge=bridge0']

#----------------------------------------------------------------------------
# Define disks

disk = ['phy:/dev/zvol/tank/debian,xvda,w']

Creating your first HVM guest

In this example we will see how to create a FreeBSD PVHVM guest running 10.0-RELEASE. The first step is to download the ISO image and create a zvol that will be used as the disk:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso
# zfs create -V 10g tank/freebsd

The following configuration file will be used:

# This configures an HVM rather than PV guest
builder = "hvm"

# Guest name
name = "freebsd"

# Initial memory allocation (MB)
memory = 512

# Number of VCPUS
vcpus = 4

# Network devices
vif = [ 'bridge=bridge0' ]

# Disk Devices
disk =	[
	'/dev/zvol/tank/freebsd,raw,hda,rw',
	'/root/FreeBSD-10.0-RELEASE-amd64-bootonly.iso,raw,hdc:cdrom,r'
	]

vnc = 1
vnclisten = "0.0.0.0"
serial="pty"

In order to run HVM guests the if_tap module has to be loaded. Then we can launch the guest and attach to it using vncviewer:

# kldload if_tap
# xl create freebsd.cfg
# vncviewer <addr>

When the install has finished the second disk device (hdc) can be removed from the configuration file.

External Articles

Shortcomings

  • Missing MSI-X PV interrupt support (FreeBSD).
  • PCI devices with virtual functions will not work correctly (FreeBSD).
  • Missing migrate/save/restore support for guests (Xen).
  • Missing PCI Passthrough support (Xen/FreeBSD).