Xen ARM with Virtualization Extensions/CrossCompiling

Introduction

This page describes how to cross compile the tools for Xen ARM with Virtualization Extensions using the Multiarch cross-compilation infrastructure available in Debian and Ubuntu to create cross build chroot environments.

Since Debian is currently frozen for the Wheezy release the multiarch support is better in Ubuntu's Raring tree at the moment at so that is what we use here.

These instructions use the sbuild tool, which is part of Debian and Ubuntu, in order to provide a convenient mechanisms for creating chroots These instructions were tested with sbuild version 0.63.2-1.1 on a Debian Wheezy system.

Note that the 32 bit arm architecture in Debian and Ubuntu is called armhf.

Typographical Conventions

#

Commands to run as root on the host

$

Commands to run as your regular user on the host

(chroot)#

Commands to run as root within the chroot

(chroot)$

Commands to run as the regular user in the chroot

$USER

Your regular username on the host (which is propagated to the chroot)

Obtaining a cross-buildable Xen tree

The majority of the patches required for cross building Xen have been incorporated in the main xen.git tree as of commit 6aaee4960d8a (committed around 30 April 2013).

Target Environment

The environment which you run on your dom0 must be the same. The instructions use Ubuntu Raring and therefore that is what you would need in your dom0 as well.

Preparing the host environment

First prepare the host environment:

# apt-get install sbuild
# sbuild-adduser $USER

This installs the sbuild tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.

Creating a base chroot

In order to setup a crossbuild chroot we first need a base chroot. If you are intending to build for 64-bit substitute arm64 for armhf everywhere in the following.

Create the initial base chroot:

# sbuild-createchroot --components=main,universe raring /srv/chroots/raring-armhf-cross http://archive.ubuntu.com/ubuntu/

This creates /etc/schroot/chroot.d/raring-amd64-sbuild-* and a chroot named raring-amd64-sbuild which is confusing if you create multiple cross build chroots. So rename and edit the configuration file as shown:

# mv /etc/schroot/chroot.d/raring-amd64-sbuild-* /etc/schroot/chroot.d/raring-armhf-cross
# vi /etc/schroot/chroot.d/raring-armhf-cross
[raring-amd64-sbuild]                        | [raring-armhf-cross]
type=directory                               | type=directory
description=Debian raring/amd64 autobuilder  | description=Debian raring/armhf crossbuilder
directory=/srv/chroots/raring-armhf-cross    | directory=/srv/chroots/raring-armhf-cross
groups=root,sbuild                           | groups=root,sbuild
root-groups=root,sbuild                      | root-groups=root,sbuild
profile=sbuild                               | profile=default

You now have a base chroot named raring-armhf-cross or raring-arm64-cross.

32-bit crossbuild

Configuring an armhf crossbuild chroot

After creating a base raring-armhf-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c raring-armhf-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the armhf repostories from the ports:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
   
deb [arch=armhf] http://ports.ubuntu.com/ raring main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add armhf as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture armhf
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-armhf

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:armhf libncurses-dev:armhf uuid-dev:armhf libglib2.0-dev:armhf libssl-dev:armhf libssl-dev:armhf libaio-dev:armhf libyajl-dev:armhf python gettext gcc git libpython2.7-dev:armhf

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm32 tools

Enter your 32-bit crossbuild chroot as your regular user:

$ schroot -c raring-armhf-cross

Change to the directory where you have cloned Xen and cross compile with:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.armhf ./configure --build=x86_64-unknown-linux-gnu --host=arm-linux-gnueabihf
(chroot)$ make dist-tools CROSS_COMPILE=arm-linux-gnueabihf- XEN_TARGET_ARCH=arm32

That's all there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm32 rootfs.

64-bit crossbuild

Introduction

The arm64 port of Debian & Ubuntu is currently a work in progress and is not formally part of either distributions. The following uses Wookey's ARMv8 bootstrap repositories to provide the necessary crossbuild tooling. Note that this is a work in progress and therefore something of a moving target so things are not as simple as with the 32-bit armhf version above.

The main rough edge is that where you have both the amd64 (native) and arm64 (cross) versions of a library installed their versions must match exactly. However it is not unusual for the main Ubuntu archive (containing the amd64 packages) to get ahead of Wookey's archive (containing the arm64 packages). This manifests as apt-get refusing to install a development package because of a dependency on a particular library. If this happens then you need to find the version of the amd64 package matching the current version in Wookey's repo and install it by hand. Note that you may need to do this recursively over the dependencies of the package you are trying to install.

For example trying to install libglib2.0-dev:arm64:

(chroot)# apt-get install libglib2.0-dev:arm64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libglib2.0-dev:arm64 : Depends: libglib2.0-0:arm64 (= 2.35.8-0ubuntu1profile1) but it is not going to be installed
                        Depends: libpcre3-dev:arm64 (>= 1:8.31) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

It is complaining about the libpcre3 development package and therefore we look at the libpcre3 library package. Looking at the versions available for amd64 and arm64:

(chroot)# apt-cache policy libpcre3
libpcre3:
  Installed: 1:8.31-2
  Candidate: 1:8.31-2
  Version table:
 *** 1:8.31-2 0
        500 http://archive.ubuntu.com/ubuntu/ raring/main amd64 Packages
        100 /var/lib/dpkg/status
(chroot)# apt-cache policy libpcre3:arm64
libpcre3:arm64:
  Installed: (none)
  Candidate: 1:8.31-1ubuntu1
  Version table:
     1:8.31-1ubuntu1 0
       1001 http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap/main arm64 Packages

Here we see that Ubuntu currently contains version 1:8.31-2 of the package for amd64 but the arm64 repo only contains 1:8.31-1ubuntu1. Therefore we need to downgrade the amd64 version by visiting https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1 and downloading libpcre3_8.31-1ubuntu1_amd64.deb which we install by hand.

The instructions below incorporate the necessary workarounds of this type as required on 2013-03-14. The set of these workarounds which are required will vary over time.

Configuring an arm64 crossbuild chroot

After creating a base raring-arm64-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c raring-arm64-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding Wookey's arm64 bootstrap repostory:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ raring main universe
   
deb http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe
deb-src http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe

We must also arrange to trust Wookey's repository:

(chroot)# wget http://people.debian.org/~wookey/bootstrap/bootstrap-archive.key
(chroot)# apt-key add bootstrap-archive.key

(validating the chain of trust on this key is left as an exercise for the reader)

Configure the bootstrap repository to have higher priority than the standard one (this avoids some subset of the problematic cases described above and generally makes things easier). Create /etc/apt/preferences.d/bootstrap containing:

Package: *
Pin: release n=raring-bootstrap
Pin-Priority: 1001

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add arm64 as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture arm64
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-arm64

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64

This will possibly fail due to the package skew issue described above. As described these must be resolved manually. On 2013-03-14 this involved the following. YMMV

(chroot)# wget http://launchpadlibrarian.net/126062829/libpcre3_8.31-1ubuntu1_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1
(chroot)# dpkg -i libpcre3_8.31-1ubuntu1_amd64.deb

(chroot)# wget http://launchpadlibrarian.net/129614595/libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpython2.7-minimal/2.7.3-15ubuntu2
(chroot)# wget http://launchpadlibrarian.net/129614593/libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libpython2.7-stdlib/2.7.3-15ubuntu2
(chroot)# apt-get install libexpat1 libsqlite3-0 mime-support
(chroot)# dpkg -i libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb

(chroot)# wget http://launchpadlibrarian.net/132602064/libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb # via https://launchpad.net/ubuntu/raring/amd64/libssl1.0.0/1.0.1c-4ubuntu5
(chroot)# dpkg -i libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb

Now we have done that we can try installing Xen's build dependencies again:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git

Lastly there are a couple of required libraries which are not available in the arm64 repository yet. I have made available versions of libaio (patched with the fix to Debian bug #702409) and yajl (patched with the fix to Debian bug #702357):

(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio-dev_0.3.109-3ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_amd64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl-dev_2.0.4-2ijc1arm64_arm64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_amd64.deb
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_arm64.deb

(chroot)# dpkg -i libaio-dev_0.3.109-3ijc1arm64_arm64.deb libaio1_0.3.109-3ijc1arm64_amd64.deb libaio1_0.3.109-3ijc1arm64_arm64.deb libyajl-dev_2.0.4-2ijc1arm64_arm64.deb libyajl2_2.0.4-2ijc1arm64_amd64.deb libyajl2_2.0.4-2ijc1arm64_arm64.deb

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm64 tools

Enter your 64-bit crossbuild chroot as your regular user:

$ schroot -c raring-arm64-cross

Change to the directory where you have cloned Xen and cross compile. Note that we need to update some autoconf machinery with versions which know about arm64.

(chroot)$ cp /usr/share/misc/config.{sub,guess} .
(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64

That's all (!) there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm64 rootfs.

References

The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.