Compiling Xen From Source

Introduction

The purpose of this document is to guide users through the process of installing Xen Project software from source (either from the tarball releases or from a source code repository).

This document was written targeting the Xen Project 4.2 release, but an attempt will be made to point out differences from previous releases where relevant.

An assumption is made of some familiarity with the general concept of building software and with using your distributions package manager to install relevant build tools etc.

Why Build From Source?

Before embarking on the process of building Xen Project software yourself it is worth considering whether this is even necessary. There are many distributions around these days which have excellent support for Xen available right from the package manager, a partial list is available at Dom0 Kernels for Xen. Where possible it is highly recommended that users consume Xen Project via their chosen distribution wherever possible. Using the distribution packaging will give you a much more integrated solution and allow you to take advantage of all the resources provided by your distribution (e.g. documentation, support etc). You can find articles on how to install Xen Project on various distributions in Category:Host Install.

The remainder of this document assumes that you have considered this and really do want to build from source.

Host (Domain 0) OS Installation

Before installing Xen Project you will first need to install your domain 0 OS, unless you have already done so. Host OS Install Considerations contains some things which you might want to consider while doing this.

Obtaining the Xen Project Source Code

The primary ways to obtain the Xen Project source code for a stable release are via the release tarballs or by cloning from the appropriate Mercurial source repository. For the development version of Xen Project (xen-unstable) Git is the primary source and Mercurial is the secondary source.

Release Tarballs

The latest Xen Project releases are linked to from The Xen.org download page

Git

The source code repositories are hosted using the Git version control system on xenbits.

Each stable release has its own branch stable-X.Y (e.g. stable-4.2 on Xen.git summary page). The code intended for the next stable point release is added to the branch staging-X.Y (e.g. staging-4.2 on Xen.git summary page), and after they pass automated tests they will be pushed to stable-X.Y for the next release. The point releases are tagged in the main tree with RELEASE-X.Y.Z (e.g. RELEASE-4.2.2 on Xen.git summary page). The result of automated tests are available on xen-devel mailing list.

Xen Project Repositories contains information on the various repositories for the stable and development branches.

To clone the source first install Git using your distro's package manager. Then execute the following command:

$ git clone URL

Where URL is the URL of the repository you wish to clone. In our case we should use:

$ git clone git://xenbits.xen.org/xen.git

As the default HEAD in xen.git is master, your local repository will also have a branch called master pointing to the upstream branch. If you wish to use different HEAD (say, staging), you can:

$ cd xen; git checkout -b staging origin/staging

You can also checkout any tags, branches as you wish.

Mercurial

Please note that XenProject.org has moved to using Git. The following section is deprecated. We do maintain Mercurial repositories mirroring Git ones, so we retain following section for reference.

Xen Project's source code repositories are hosted using the Mercurial version control system on xenbits.

Each stable release has it's own branch xen-X.Y-testing.hg (e.g. xen-4.2-testing.hg) where code intended for the next stable point release is added. The Xen Project development branch is known as xen-unstable and has its own repository xen-unstable.hg.

Each stable and development branch is available in two forms either tested (the main branch) or untested (the staging branch). When commits are made to a Xen Project tree they are first added to the staging branch and only propagated to the main branch after automated testing has passed. For example all commits to the Xen Project development branch will initially appear in staging/xen-unstable.hg and then propagate to xen-unstable.hg after automated testing has completed. The automated test results are posted to the xen-devel mailing list.

Xen Project Repositories contains information on the various repositories for the stable and development branches.

To clone the source first install the mercurial tool using your distributions package manager. Then execute the following command:

$ hg clone URL

Where URL is the URL of the repository you wish to clone. e.g. to clone the latest tested xen-unstable tree:

$ hg clone http://xenbits.xen.org/hg/xen-unstable.hg

Or to clone the staging (e.g. not yet tested) xen-unstable tree:

$ hg clone http://xenbits.xen.org/hg/staging/xen-unstable.hg

You may want to get a specific changeset (revision), for example when trying to replicate someone else's build, or when dealing with other patches you have to apply afterwards. You can do this with the -r option. For example, to get changeset 25364:

$ hg clone -r 25364 http://xenbits.xen.org/hg/xen-unstable.hg

Quick-Start

The README at the top level of the Xen Project source code tree contains a quick-start guide to building the software. This provides a quick overview of the process and requirements for building Xen Project software and will generally contain the most up to date information specific to the particular source tree you are looking at. After obtaining the Xen Project source this is the first document you should read.

Building from Source

This section documents all known requirements to build from source.

Updated /sbin/installkernel on Linux

Linux distributions shipping with grub2 will need to ensure that their /sbin/installkernel script, which has to be provided by each Linux distribution, copies the the kernel configuration upon a custom kernel install time. The requirement for the config file comes from upstream grub2 /etc/grub.d/20_linux_xen which will only add xen as an instance to your grub.cfg if and only if it finds in your config file either of:

   CONFIG_XEN_DOM0=y
   CONFIG_XEN_PRIVILEGED_GUEST=y

Without this a user compiling and installing their own kernel with proper support for xen and with the xen hypervisor present will not get their respective grub2 update script to pick up the xen hypervisor. Debian testing has proper support for this, OpenSUSE required this change upstream on mkinitrd, so OpenSUSE folks will want to get the latest /sbin/installkernel hosted on the OpenSUSE mkinitrd repository on github.

   # If on OpenSUSE update your /sbin/installkernel
   git clone https://github.com/openSUSE/mkinitrd.git
   cd mkinitrd
   sudo cp sbin/installkernel /sbin/installkernel 

Fedora might need a similar update. Please edit this wiki once you confirm.

Build Dependencies

Xen Project uses several external libraries and tools. The primary list of these prerequisites is the list present in the README file.

Even this list assumes some sort of basic development environment. A good starting point for this is to use your distributions development install package option.

Build Dependencies - Debian / Ubuntu

Under Debian / Ubuntu (and derived distributions) install the build-essential package:

# apt-get install build-essential

you also need to install these additional debs:

# apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif 
# apt-get install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial
# apt-get install make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev
# apt-get install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev
# apt-get install gettext libpixman-1-dev libaio-dev markdown pandoc

And if you're building on Wheezy onward:

# apt-get install libc6-dev-i386

One useful shortcut can be to use your distributions package manager to install all the prerequisite packages is to install those packages which are noted as being required to build the distribution's own Xen packages. e.g. under Debian or a Debian derived distribution:

# apt-get build-dep xen

However you need to be mindful of new prerequisites added between whatever version of Xen Project software is in your distribution and the version you are building when using this trick.

Build Dependencies - OpenSUSE

If you're now on the latest OpenSUSE you'll note its now a rolling distribution base. The default instructions do not actually encourage you to install the source repositories, and even if you did install them the instructions disable them by default, so be sure to install them and enable them otherwise the command zypper source-install -d won't work. To enable the required repository if you already had it installed:

# zypper mr -e repo-src-oss 

Now get the build dependencies for Xen

# zypper source-install -d xen

Here's a list of things not currentloy picked up by the build dependencies which have been found to be needed:

 # zypper install systemd-devel gettext-tools\
   ocaml ocaml-compiler-libs ocaml-runtime \
   ocaml-ocamldoc ocaml-findlib glibc-devel-32bit make patch

While at it also get the build dependencies for Linux:

# zypper source-install -d kernel-desktop

Build Dependencies - Fedora

These instructions are kept separate from RHEL / Centos as Fedora is updated more frequently, so you are expected to do less work to build the latest and greatest Xen.

Install known build dependencies:

# yum-builddep xen

Install things not picked up by the current build dependencies:

# yum install glibc-devel.x86_64 systemd-devel.x86_64

Get build dependencies for Linux, the kernel, while at it:

# yum-builddep kernel 

Build Dependencies - RHEL / Centos

Under RHEL, CentOS, Fedora based distributions you want to install the Development Tools package groups:

# yum groupinstall "Development Tools"

(On old CentOS and Fedora, a group called Development Libraries should, if available, installed too. That is not present in recent releases of the distributions any longer, though).

Then install known build dependencies (confirmation needed if this works on RHEL / Centos):

# yum-builddep xen

You also need to install these additional rpms:

# yum install transfig wget tar less texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel
# yum install pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo
# yum install libidn-devel yajl yajl-devel ocaml ocaml-findlib ocaml-findlib-devel python-devel uuid-devel libuuid-devel openssl-devel
# yum install python-markdown pandoc systemd-devel glibc-devel.i686

If on CentOS, for some of this packages (e.g., dev86 and pandoc), enabling the EPEL additional repository is necessary.

Additional notes

Having installed this you should then install each of the prerequisites listed in the README using your distribution's package management tool. In general the Xen Project code tries to only depend upon external tools and libraries which are commonly available in distributions therefore obtaining the prerequisites other than from your distribution's package management system is out of scope for this document. If you have trouble locating a particular prerequisite then please contact the xen-users mailing list.

Configure

From Xen Project 4.2 onwards, the software uses the commonly used autoconf tool to provide compile time configurability of the toolstack. This allows some control of what features are built into Xen Project, as well as compile time sanity checking. To configure Xen Project, simply run the provided configure script:

$ ./configure

To see the various options run the configure script with --help e.g.:

$ ./configure --help
[...]
 Optional Features:
   --disable-option-checking  ignore unrecognized --enable/--with options
   --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
   --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
   --enable-githttp        Download GIT repositories via HTTP (default is
                           DISABLED)

This step is only required from Xen Project 4.2 onwards. Prior to Xen Project 4.2 these options could be configured by passing a variable on the make command line during build & install or by writing the variable to a file named .config at the top level of the source tree.

Use http:// Rather Than git:// to Clone Additional Repositories

When building the software from Mercurial, the build system will automatically clone several additional repositories from the network. Some of these repositories use the version control system which uses its own protocol on a specific port. Sometimes this causes issues due to firewalls etc blocking the git port. This can be worked around by instructing the Xen build system to clone such repositories using a less efficient HTTP based protocol:

$ ./configure --enable-githttp

Prior to Xen Project 4.2 this could be achieved by writing GIT_HTTP=y to your .config:

$ cat .config
GIT_HTTP = y

Library Installation Directory

Xen Project 4.2 onwards defaults to installing libraries into /usr/lib by default and from 4.3 onwards defaults to installing to /usr/local/lib by default.

Users on systems which use /usr/local/lib64 for 64-bit libraries should use the --libdir option. e.g:

$ ./configure --libdir=/usr/local/lib64

Failure to do this usually results in errors about libraries not found or using older versions of the libraries which will likely not work.

_NB_: If you are using `--prefix=/usr` you should use `--libdir=/usr/lib64` too.

Systemd

If the target system uses systemd, do not forget to enable it:

$ ./configure --enable-systemd

Python Prefix and Module Layout

On some distros (e.g. Debian and Ubuntu) Xen Project may install the python parts of the code into the wrong place (See Debian bug #693721). Therefore it is necessary to set PYTHON_PREFIX_ARG=--install-layout=deb:

$ cat .config
PYTHON_PREFIX_ARG=--install-layout=deb

Some versions of Ubuntu have a bug which requires instead that PYTHON_PREFIX_ARG is to set the empty string:

$ cat .config
PYTHON_PREFIX_ARG=

As of 4.2 this option is not yet supported by the configure script and therefore should still be set via .config or on the make command line.

The most common symptom of this issue is pygrub failing to work, with output similar to:

Traceback (most recent call last):
   File "/usr/lib/xen/bin/pygrub", line 20, in <module>
     import xen.lowlevel.xc
ImportError: No module named xen.lowlevel.xc

Build & Install

Xen Project downloads various components at build time. This means that building Xen Project currently requires an active connection to the Internet

To build all components (hypervisor, tools, docs, stubdomains, etc) you can use the dist target.

$ make dist

If you wish to just (re)build a single component you can use the appropriate dist-COMPONENT target:

$ make dist-xen
$ make dist-tools
$ make dist-docs
$ ... etc ...

If you want to rebuild a tree as if from a fresh check then you can use the world target. This is effectively the same as clean and the dist

$ make world

All of the above targets will build and install the appropriate component into the dist subdirectory but not actually install onto the system.

To install onto the local machine simply call the install target (as root):

# make install

As with dist you can also install individual components using the appropriate install-COMPONENT target:

# make install-xen
# make install-tools
# make install-docs
# ... etc ...

If you want to install onto a remote machine then you can simply copy the dist directory over and TBD

A better option is to make a "package-ball" -- a package which is the equivalent of a tarball with no dependency checking or set-up. There are target for either deb-based systems:

# make debball

or rpm-based systems:

# make rpmball

Packages are placed in the dist/ directory.

Installing the resulting package is functionally equivalent to the make install target above, but because the files are tracked by the package manager, it is easier to remove or update.

After installing (either via make install or a packageballl) you rebuild your dynamic linker cache by running:

# /sbin/ldconfig

To get more information on the available targets use the help target:

$ make help

Linux grub2 update

When on Linux and using grub2 after installing Xen you will also need to make sure grub2 will pick up your new shiny Xen hypervisor. Distributions differ on how to do this. This section documents how to do this for known distributions.

update grub2 config on OpenSUSE

# update-bootloader --refresh

update grub2 config on Debian

# update-grub

update grub2 config on Fedora

TBD

update grub2 config on RHEL / Centos

TBD

Troubleshooting

• If SeaBIOS fails to compile when building Xen 4.2 on a system with a non-English locale, try setting LC_ALL to en_US.UTF-8 before calling make:

# export LC_ALL=en_US.UTF-8
# make world

Kernels

The Linux kernel now supports different hypervisors with one kernel. There is also no requirement for a domain 0 (or guest) kernel to match your hypervisor so you are free to pick the kernel which best suits your needs (e.g. many distributions supply a kernel which is compatible with Xen Project, which is a quick and easy path). Dom0 Kernels for Xen contains some guidance on this issue.

Compiling latest Linux kernel support

If compiling Linux from source for either dom0 or a guest you can now enable Xen support for both through one simple kernel configuration helper, this was added as of v4.2:

   make xenconfig

This lets you build a kernel which can support xen dom0 or xen guests on i386, x86-64 and arm64.

Post-Installation

When installing Xen Project code from source (.tar.gz) on Debian/Ubuntu, this is how you can enable automatic start of Xen Project services on system startup:

update-rc.d xencommons defaults 19 18
update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20
update-rc.d xen-watchdog defaults 22 23

Or, if on a systemd enabled distro:

systemctl enable xencommons
systemctl enable xendomains
systemctl enable xen-watchdog

After making those changes reboot the system.

Host Configuration

Once the Xen Project code is installed there is still some host level setup required. Category:Host Configuration covers this.