Xen ARM with Virtualization Extensions/CrossCompiling

At least gcc version 4.7.3 is known to miscompile certain parts of Xen, most often leading to a segmentation fault in the xl toolstack while starting a guest. We recommend using 4.8.x or later

This page deals with cross-compiling the Xen userspace tools. Cross compiling just the hypervisor is much simpler and is covered on the main Xen on ARM page

Introduction

When working with low power or constrained environments (and in particular when using software emulated platforms such as the Fast Models) it may not be desirable (or even possible) to build software, such as the Xen userspace tools, on the target device itself. In such cases it is necessary to build on some other more powerful/capable/suitable device. If a more powerful system of the same architecture is available then this can be achieved by simply building on that device and transferring the result to the target device. However if a more powerful system of the same architecture is not available then the software must be "cross-compiled", that is built on a system of a different host architecture using tools which produce binaries that will run on the target architecture. In the most common case this will involve using an x86 host machine to build binaries for an ARM system.

There are three main ways to set up a cross compile environment:

Yocto and meta-virtualization

See this page to use Yocto to cross-compile a minimal dom0 initramfs, including all the Xen userspace tools.

Construct a traditional cross compilation environment

This technique involves installing a compiler on the host system which runs natively on the host but produces binaries for the target system and making available all the of the libraries required for the target architecture in order to build the software in question. This is what is normally meant when people talk about cross compiling. Making all of the necessary libraries available is sometimes easier said than done however and many project's build systems do not cope well with cross-compilation (the Xen tools are cross compile friendly though).

Using a foreign chroot

This technique involves creating a chroot on the host containing a distro for the target system and using the "native" tools and libraries within this chroot to build for the target using the native build processes. This relies on the Qemu system emulator to emulate a userspace environment for the target system (which is much more efficient than doing full system emulation of the target since it uses the host system's native kernel). This method is not normally as fast as traditional cross compilation but it is significantly faster than building using full system emulation and is often faster than building natively on a low powered device. This basics of this are described in a blog post Foreign Chroots with schroot and qemu while a more specific example covering arm64 using openSUSE is described in Virtualization on ARM with Xen.

Target Environment

Yocto automatically sets up the cross-compilation environment for you. If you don't use Yocto, it is important that the build (i.e. cross or foreign chroot) environment matches the runtime (i.e. dom0) environment where the tools will run. This means that the library and compiler versions etc should match. These instructions use the Ubuntu Saucy Salamander release and so that is what you would need in your dom0 as well. See Xen_ARM_with_Virtualization_Extensions/RootFilesystem for instructions on creating a root filesystem.

Xen Version

Xen on ARM is cross buildable on an x86 host from Xen 4.4 onwards.

Typographical Conventions

#

Commands to run as root on the host

$

Commands to run as your regular user on the host

(chroot)#

Commands to run as root within the chroot

(chroot)$

Commands to run as the regular user in the chroot

$USER

Your regular username on the host (which is propagated to the chroot)

Using sbuild and schroot

These instructions use the sbuild and schroot tools, which are part of Debian and Ubuntu, in order to provide a convenient mechanism for creating chroots.

These instructions were tested with sbuild version 0.63.2-1.1 and schroot version 1.6.4-4 on a Debian Wheezy system.

sbuild can be installed as follows, which will also pull in schroot as a dependency:

# apt-get install sbuild
# sbuild-adduser $USER

This installs the sbuild tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.

If sbuild and schroot are not available

If your host distribution does not supply sbuild then it is still possible to use a chroot in a more manual fashion. This is mostly out of scope for this document but some hints:

• sbuild will automatically bind mount things inside the chroot, so if running without you will need to ensure that your source trees etc are available within the chroot, either by bind mounting manually or by copying the source tree into the chroot.

• sbuild will automatically propagate any necessary host level configuration to the chroot, which you may need to do by hand. e.g. you would likely need to copy /etc/resolv.conf into the chroot in order to access the network while within the chroot.

• sbuild will automatically make your $USER available inside the chroot. Either run as root within the chroot (not really recommended) or ensure that your $USER exists within the chroot with the same uid and gid.

Cross-compile chroot environment using multiarch

This section describes how to use traditional cross compilation to build the Xen tools for Xen on ARM using the Multiarch infrastructure available in Debian and Ubuntu to create a cross build chroot environment using (and targeting) Ubuntu Saucy Salamander.

In principal it might be possible to use multiarch on the host to setup a cross environment without using a chroot, however multiarch does not currently support both native and cross compilation at the same time so it is easier to put the cross environment into a chroot.

Creating a base chroot

In order to setup a crossbuild chroot we first need a base chroot. Note that the 32 bit arm architecture in Debian and Ubuntu is called armhf. If you are intending to build for 64-bit substitute arm64 for armhf everywhere in the following.

Create the initial base chroot. This will be a native host (i.e. x86) chroot into which we will install an ARM cross compile environment:

# sbuild-createchroot --components=main,universe saucy /srv/chroots/saucy-armhf-cross http://archive.ubuntu.com/ubuntu/

This creates /etc/schroot/chroot.d/saucy-amd64-sbuild-* (with a random suffix) and a chroot named saucy-amd64-sbuild using the sbuild profile. The profile should be changed to default. Also the naming is confusing if you create multiple cross build chroots. So rename the file and edit the configuration file as shown:

# mv /etc/schroot/chroot.d/saucy-amd64-sbuild-* /etc/schroot/chroot.d/saucy-armhf-cross
# vi /etc/schroot/chroot.d/saucy-armhf-cross
[saucy-amd64-sbuild]                         | [saucy-armhf-cross]
type=directory                               | type=directory
description=Debian saucy/amd64 autobuilder   | description=Debian saucy/armhf crossbuilder
directory=/srv/chroots/saucy-armhf-cross     | directory=/srv/chroots/saucy-armhf-cross
groups=root,sbuild                           | groups=root,sbuild
root-groups=root,sbuild                      | root-groups=root,sbuild
profile=sbuild                               | profile=default

You now have a base chroot named saucy-armhf-cross or saucy-arm64-cross.

32-bit crossbuild

Configuring an armhf crossbuild chroot

After creating a base saucy-armhf-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c saucy-armhf-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the armhf repositories from the ports archive:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
   
deb [arch=armhf] http://ports.ubuntu.com/ saucy main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add armhf as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture armhf
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-armhf

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:armhf libncurses-dev:armhf uuid-dev:armhf libglib2.0-dev:armhf libssl-dev:armhf libssl-dev:armhf libaio-dev:armhf libyajl-dev:armhf python gettext gcc git libpython2.7-dev:armhf libfdt-dev:armhf

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm32 tools

Enter your 32-bit crossbuild chroot as your regular user:

$ schroot -c saucy-armhf-cross

Change to the directory where you have cloned Xen and cross compile with:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.armhf ./configure --build=x86_64-unknown-linux-gnu --host=arm-linux-gnueabihf
(chroot)$ make dist-tools CROSS_COMPILE=arm-linux-gnueabihf- XEN_TARGET_ARCH=arm32

That's all there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm32 rootfs.

64-bit crossbuild

Introduction

These instructions have been massively simplified with the release of Saucy Salamander vs the previous Raring Ringtail release. They are essentially identical to the armhf variant.

Configuring an arm64 crossbuild chroot

After creating a base saucy-arm64-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c saucy-arm64-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the arm64 repositories from the ports archive:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
   
deb [arch=arm64] http://ports.ubuntu.com/ saucy main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add arm64 as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture arm64
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-arm64

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64 libfdt-dev:arm64

We also need the autotools-dev package to workaround out of date autoconf machinery in the Xen 4.4.0 release:

(chroot)# apt-get install autotools-dev

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm64 tools

Enter your 64-bit crossbuild chroot as your regular user:

$ schroot -c saucy-arm64-cross

Change to the directory where you have cloned Xen and cross compile.

If building Xen 4.4.0 then it is first necessary to update the autoconf machinery with versions which know about arm64:

(chroot)$ cp /usr/share/misc/config.{sub,guess} .

Now we can cross compile:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64

You should now have a dist/install directory containing the installed bits which can be copied into your arm64 rootfs.

References

The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.