Xen ARM with Virtualization Extensions/CrossCompiling

Introduction

When working with low power or constrained environments (and in particular when using software emulated platforms such as the Fast Models) it may not be desirable (or even possible) to build software, such as the Xen userspace tools, on the target device itself. In such cases it is necessary to build on some other more powerful/capable/suitable device. If a more powerful system of the same architecture is available then this can be achieved by simply building on that device and transferring the result to the target device. However if a more powerful system of the same architecture is not available then the software must be "cross-compiled", that is built on a system of a different host architecture using tools which produce binaries that will run on the target architecture. In the most common case this will involve using an x86 host machine to build binaries for an ARM system.

There are two main ways to set up a cross compile environment:

Construct a traditional cross compilation environment

This technique involves installing a compiler on the host system which runs natively on the host but produces binaries for the target system and making available all the of the libraries required for the target architecture in order to build the software in question. This is what is normally meant when people talk about cross compiling. Making all of the necessary libraries available is sometimes easier said than done however and many project's build systems do not cope well with cross-compilation (the Xen tools are cross compile friendly though).

Using a foreign chroot

This technique involves creating a chroot on the host containing a distro for the target system and using the "native" tools and libraries within this chroot to build for the target using the native build processes. This relies on the Qemu system emulator to emulate a userspace environment for the target system (which is much more efficient than doing full system emulation of the target since it uses the host system's native kernel). This method is not normally as fast as traditional cross compilation but it is significantly faster than building using full system emulation and is often faster than building natively on a low powered device. This is described in Foreign Chroots with schroot and qemu.

Target Environment

Regardless of which technique is used it is important that the build (i.e. cross or foreign chroot) environment matches the runtime (i.e. dom0) environment where the tools will run. This means that the library and compiler versions etc should match. These instructions use Ubuntu Raring and so that is what you would need in your dom0 as well. See Xen_ARM_with_Virtualization_Extensions/RootFilesystem for instructions on creating a root filesystem.

Xen Version

Xen on ARM is cross buildable on an x86 host from Xen 4.4 onwards.

Typographical Conventions

#

Commands to run as root on the host

$

Commands to run as your regular user on the host

(chroot)#

Commands to run as root within the chroot

(chroot)$

Commands to run as the regular user in the chroot

$USER

Your regular username on the host (which is propagated to the chroot)

Using sbuild and schroot

These instructions use the sbuild and schroot tools, which are part of Debian and Ubuntu, in order to provide a convenient mechanism for creating chroots.

These instructions were tested with sbuild version 0.63.2-1.1 and schroot version 1.6.4-4 on a Debian Wheezy system.

sbuild can be installed as follows, which will also pull in schroot as a dependency:

# apt-get install sbuild
# sbuild-adduser $USER

This installs the sbuild tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.

If sbuild and schroot are not available

If your host distribution does not supply sbuild then it is still possible to use a chroot in a more manual fashion. This is mostly out of scope for this document but some hints:

• sbuild will automatically bind mount things inside the chroot, so if running without you will need to ensure that your source trees etc are available within the chroot, either by bind mounting manually or by copying the source tree into the chroot.

• sbuild will automatically propagate any necessary host level configuration to the chroot, which you may need to do by hand. e.g. you would likely need to copy /etc/resolv.conf into the chroot in order to access the network while within the chroot.

• sbuild will automatically make your $USER available inside the chroot. Either run as root within the chroot (not really recommended) or ensure that your $USER exists within the chroot with the same uid and gid.

Cross-compile chroot environment using multiarch

This section describes how to use traditional cross compilation to build the Xen tools for Xen on ARM using the Multiarch infrastructure available in Debian and Ubuntu to create a cross build chroot environment using (and targeting) Ubuntu Raring.

In principal it might be possible to use multiarch on the host to setup a cross environment without using a chroot, however multiarch does not currently support both native and cross compilation at the same time so it is easier to put the cross environment into a chroot.

Creating a base chroot

In order to setup a crossbuild chroot we first need a base chroot. Note that the 32 bit arm architecture in Debian and Ubuntu is called armhf. If you are intending to build for 64-bit substitute arm64 for armhf everywhere in the following.

Create the initial base chroot. This will be a native host (i.e. x86) chroot into which we will install an ARM cross compile environment:

# sbuild-createchroot --components=main,universe raring /srv/chroots/raring-armhf-cross http://archive.ubuntu.com/ubuntu/

This creates /etc/schroot/chroot.d/raring-amd64-sbuild-* and a chroot named raring-amd64-sbuild using the sbuild profile. The profile should be changed to default. Also the naming is confusing if you create multiple cross build chroots. So rename the file and edit the configuration file as shown:

# mv /etc/schroot/chroot.d/raring-amd64-sbuild-* /etc/schroot/chroot.d/raring-armhf-cross
# vi /etc/schroot/chroot.d/raring-armhf-cross
[raring-amd64-sbuild]                        | [raring-armhf-cross]
type=directory                               | type=directory
description=Debian raring/amd64 autobuilder  | description=Debian raring/armhf crossbuilder
directory=/srv/chroots/raring-armhf-cross    | directory=/srv/chroots/raring-armhf-cross
groups=root,sbuild                           | groups=root,sbuild
root-groups=root,sbuild                      | root-groups=root,sbuild
profile=sbuild                               | profile=default

You now have a base chroot named raring-armhf-cross or raring-arm64-cross.

32-bit crossbuild

Configuring an armhf crossbuild chroot

After creating a base raring-armhf-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c raring-armhf-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the armhf repostories from the ports:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
   
deb [arch=armhf] http://ports.ubuntu.com/ raring main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add armhf as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture armhf
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-armhf

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:armhf libncurses-dev:armhf uuid-dev:armhf libglib2.0-dev:armhf libssl-dev:armhf libssl-dev:armhf libaio-dev:armhf libyajl-dev:armhf python gettext gcc git libpython2.7-dev:armhf libfdt-dev:armhf

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm32 tools

Enter your 32-bit crossbuild chroot as your regular user:

$ schroot -c raring-armhf-cross

Change to the directory where you have cloned Xen and cross compile with:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.armhf ./configure --build=x86_64-unknown-linux-gnu --host=arm-linux-gnueabihf
(chroot)$ make dist-tools CROSS_COMPILE=arm-linux-gnueabihf- XEN_TARGET_ARCH=arm32

That's all there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm32 rootfs.

 /usr/include/libfdt.h:54:24: fatal error: libfdt_env.h: No such file or directory

If you meet this error while compiling, it is probably a bug from libfdt-dev 1.3.0-2build1 in raring packages. This problem was fixed in 1.3.0-3, so upgrading the package would fix this.[1]

64-bit crossbuild

Introduction

The arm64 port of Debian & Ubuntu is currently a work in progress and is not formally part of either distributions. The following uses Wookey's ARMv8 bootstrap repositories to provide the necessary crossbuild tooling. Note that this is a work in progress and therefore something of a moving target so things are not as simple as with the 32-bit armhf version above.

The main rough edge is that where you have both the amd64 (native) and arm64 (cross) versions of a library installed their versions must match exactly. However it is not unusual for the main Ubuntu archive (containing the amd64 packages) to get ahead of Wookey's archive (containing the arm64 packages). This manifests as apt-get refusing to install a development package because of a dependency on a particular library. If this happens then you need to find the version of the amd64 package matching the current version in Wookey's repo and install it by hand. Note that you may need to do this recursively over the dependencies of the package you are trying to install.

For example trying to install libglib2.0-dev:arm64:

(chroot)# apt-get install libglib2.0-dev:arm64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libglib2.0-dev:arm64 : Depends: libglib2.0-0:arm64 (= 2.35.8-0ubuntu1profile1) but it is not going to be installed
                        Depends: libpcre3-dev:arm64 (>= 1:8.31) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

It is complaining about the libpcre3 development package and therefore we look at the libpcre3 library package. Looking at the versions available for amd64 and arm64:

(chroot)# apt-cache policy libpcre3
libpcre3:
  Installed: 1:8.31-2
  Candidate: 1:8.31-2
  Version table:
 *** 1:8.31-2 0
        500 http://archive.ubuntu.com/ubuntu/ raring/main amd64 Packages
        100 /var/lib/dpkg/status
(chroot)# apt-cache policy libpcre3:arm64
libpcre3:arm64:
  Installed: (none)
  Candidate: 1:8.31-1ubuntu1
  Version table:
     1:8.31-1ubuntu1 0
       1001 http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap/main arm64 Packages

Here we see that Ubuntu currently contains version 1:8.31-2 of the package for amd64 but the arm64 repo only contains 1:8.31-1ubuntu1. Therefore we need to downgrade the amd64 version by visiting https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1 and downloading libpcre3_8.31-1ubuntu1_amd64.deb which we install by hand.

The instructions below incorporate the necessary workarounds of this type as required on 2013-03-14. The set of these workarounds which are required will vary over time.

Configuring an arm64 crossbuild chroot

After creating a base raring-arm64-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c raring-arm64-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding Wookey's arm64 bootstrap repostory:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
   
deb http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe
deb-src http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe

We must also arrange to trust Wookey's repository:

(chroot)# wget http://people.debian.org/~wookey/bootstrap/bootstrap-archive.key
(chroot)# apt-key add bootstrap-archive.key

(validating the chain of trust on this key is left as an exercise for the reader)

Configure the bootstrap repository to have higher priority than the standard one (this avoids some subset of the problematic cases described above and generally makes things easier). Create /etc/apt/preferences.d/bootstrap containing:

Package: *
Pin: release n=raring-bootstrap
Pin-Priority: 1001

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add arm64 as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture arm64
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-arm64

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64

This will possibly fail due to the package skew issue described above. As described these must be resolved manually. On 2013-03-14 this involved the following. YMMV

(chroot)# wget http://launchpadlibrarian.net/126062829/libpcre3_8.31-1ubuntu1_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1
(chroot)# dpkg -i libpcre3_8.31-1ubuntu1_amd64.deb

(chroot)# wget http://launchpadlibrarian.net/129614595/libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpython2.7-minimal/2.7.3-15ubuntu2
(chroot)# wget http://launchpadlibrarian.net/129614593/libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpython2.7-stdlib/2.7.3-15ubuntu2
(chroot)# apt-get install libexpat1 libsqlite3-0 mime-support
(chroot)# dpkg -i libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb

(chroot)# wget http://launchpadlibrarian.net/132602064/libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libssl1.0.0/1.0.1c-4ubuntu5
(chroot)# dpkg -i libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb

Now we have done that we can try installing Xen's build dependencies again:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git

Lastly there are a couple of required libraries which are not available in the arm64 repository yet. I have made available versions of libaio (patched with the fix to Debian bug #702409) and yajl (patched with the fix to Debian bug #702357):

(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio-dev_0.3.109-3ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_amd64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl-dev_2.0.4-2ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_amd64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_arm64.deb

(chroot)# dpkg -i libaio-dev_0.3.109-3ijc1arm64_arm64.deb libaio1_0.3.109-3ijc1arm64_amd64.deb libaio1_0.3.109-3ijc1arm64_arm64.deb libyajl-dev_2.0.4-2ijc1arm64_arm64.deb libyajl2_2.0.4-2ijc1arm64_amd64.deb libyajl2_2.0.4-2ijc1arm64_arm64.deb

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm64 tools

Enter your 64-bit crossbuild chroot as your regular user:

$ schroot -c raring-arm64-cross

Change to the directory where you have cloned Xen and cross compile. Note that we need to update some autoconf machinery with versions which know about arm64.

(chroot)$ cp /usr/share/misc/config.{sub,guess} .
(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64

That's all (!) there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm64 rootfs.

References

The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.