Difference between revisions of "Xen ARM with Virtualization Extensions/CrossCompiling"

From Xen
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
  +
{{WarningLeft|This page is outdated. Please refer to [[Xen_ARM_with_Virtualization_Extensions#Building_Xen_on_ARM]]}}
  +
 
= Introduction =
 
= Introduction =
   
  +
When working with low power or constrained environments (and in particular when using software emulated platforms such as the [[Xen_ARM_with_Virtualization_Extensions/FastModels|Fast Models]]) it may not be desirable (or even possible) to build software, such as the Xen userspace tools, on the target device itself. In such cases it is necessary to build on some other more powerful/capable/suitable device. If a more powerful system of the same architecture is available then this can be achieved by simply building on that device and transferring the result to the target device. However if a more powerful system of the same architecture is not available then the software must be "cross-compiled", that is built on a system of a different host architecture using tools which produce binaries that will run on the target architecture. In the most common case this will involve using an x86 host machine to build binaries for an ARM system.
This page describes how to cross compile the tools for [[Xen ARM with Virtualization Extensions]] using the [http://wiki.debian.org/Multiarch Multiarch] cross-compilation infrastructure available in Debian and Ubuntu to create cross build chroot environments.
 
  +
  +
There are three main ways to set up a cross compile environment:
  +
  +
; Yocto and meta-virtualization
  +
: See [[Xen_on_ARM_and_Yocto|this page]] to use Yocto to cross-compile a minimal dom0 initramfs, including all the Xen userspace tools.
  +
; Construct a traditional cross compilation environment
  +
: This technique involves installing a compiler on the host system which runs natively on the host but produces binaries for the target system and making available all the of the libraries required for the target architecture in order to build the software in question. This is what is normally meant when people talk about cross compiling. Making all of the necessary libraries available is sometimes easier said than done however and many project's build systems do not cope well with cross-compilation (the Xen tools are cross compile friendly though).
  +
; Using a foreign chroot
  +
: This technique involves creating a chroot on the host containing a distro for the target system and using the "native" tools and libraries within this chroot to build for the target using the native build processes. This relies on the [http://wiki.qemu.org/Main_Page Qemu system emulator] to emulate a userspace environment for the target system (which is much more efficient than doing full system emulation of the target since it uses the host system's native kernel). This method is not normally as fast as traditional cross compilation but it is significantly faster than building using full system emulation and is often faster than building natively on a low powered device. This basics of this are described in a blog post [http://www.hellion.org.uk/blog/posts/foreign-chroots-with-schroot-and-qemu/ Foreign Chroots with schroot and qemu] while a more specific example covering arm64 using openSUSE is described in [http://community.arm.com/groups/processors/blog/2014/03/28/virtualization-on-arm-with-xen Virtualization on ARM with Xen].
  +
  +
== Target Environment ==
   
  +
Yocto automatically sets up the cross-compilation environment for you. If you don't use Yocto, it is important that the build (i.e. cross or foreign chroot) environment matches the runtime (i.e. dom0) environment where the tools will run. This means that the library and compiler versions etc should match. These instructions use the [http://www.ubuntu.com Ubuntu] [https://wiki.ubuntu.com/SaucySalamander Saucy Salamander] release and so that is what you would need in your dom0 as well. See [[Xen_ARM_with_Virtualization_Extensions/RootFilesystem]] for instructions on creating a root filesystem.
Since Debian is currently frozen for the Wheezy release the multiarch support is better in Ubuntu's Raring tree at the moment at so that is what we use here.
 
   
  +
== Xen Version ==
These instructions use the [http://packages.debian.org/source/wheezy/sbuild sbuild] tool, which is part of Debian and Ubuntu, in order to provide a convenient mechanisms for creating chroots These instructions were tested with sbuild version 0.63.2-1.1 on a Debian Wheezy system.
 
   
  +
Xen on ARM is cross buildable on an x86 host from Xen 4.4 onwards.
Note that the 32 bit arm architecture in Debian and Ubuntu is called ''armhf''.
 
   
= Typographical Conventions =
+
== Typographical Conventions ==
   
 
; #
 
; #
Line 22: Line 35:
 
: Your regular username on the host (which is propagated to the chroot)
 
: Your regular username on the host (which is propagated to the chroot)
   
  +
= Using ''sbuild'' and ''schroot'' =
= Obtaining a cross-buildable Xen tree =
 
   
  +
These instructions use the [http://packages.debian.org/source/wheezy/sbuild sbuild] and [http://packages.debian.org/source/wheezy/schroot schroot] tools, which are part of Debian and Ubuntu, in order to provide a convenient mechanism for creating chroots.
The majority of the patches required for cross building Xen have been incorporated in the main xen.git tree as of commit 6aaee4960d8a (committed around 30 April 2013).
 
   
  +
These instructions were tested with sbuild version 0.63.2-1.1 and schroot version 1.6.4-4 on a Debian Wheezy system.
= Target Environment =
 
   
  +
sbuild can be installed as follows, which will also pull in schroot as a dependency:
It is important that the cross compile environment matches the actual dom0 environment where you intend to run the tools. This means that the library and compiler versions etc should match. These instructions use Ubuntu Raring and therefore that is what you would need in your dom0 as well. See [[Xen_ARM_with_Virtualization_Extensions/RootFilesystem]] for instructions on creating a root filesystem.
 
 
= Preparing the host environment =
 
 
First prepare the host environment:
 
   
 
# apt-get install sbuild
 
# apt-get install sbuild
Line 39: Line 48:
 
This installs the ''sbuild'' tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.
 
This installs the ''sbuild'' tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.
   
  +
== If sbuild and schroot are not available ==
= Creating a base chroot =
 
   
  +
If your host distribution does not supply ''sbuild'' then it is still possible to use a chroot in a more manual fashion. This is mostly out of scope for this document but some hints:
In order to setup a crossbuild chroot we first need a base chroot. If you are intending to build for 64-bit substitute '''arm64''' for '''armhf''' everywhere in the following.
 
   
  +
* ''sbuild'' will automatically bind mount things inside the chroot, so if running without you will need to ensure that your source trees etc are available within the chroot, either by bind mounting manually or by copying the source tree into the chroot.
Create the initial base chroot:
 
   
  +
* ''sbuild'' will automatically propagate any necessary host level configuration to the chroot, which you may need to do by hand. e.g. you would likely need to copy <tt>/etc/resolv.conf</tt> into the chroot in order to access the network while within the chroot.
# sbuild-createchroot --components=main,universe raring /srv/chroots/raring-armhf-cross http://archive.ubuntu.com/ubuntu/
 
   
  +
* ''sbuild'' will automatically make your $USER available inside the chroot. Either run as root within the chroot (not really recommended) or ensure that your $USER exists within the chroot with the same uid and gid.
This creates /etc/schroot/chroot.d/raring-amd64-sbuild-* and a chroot named raring-amd64-sbuild which is confusing if you create multiple cross build chroots. So rename and edit the configuration file as shown:
 
   
  +
= Cross-compile chroot environment using multiarch =
# mv /etc/schroot/chroot.d/raring-amd64-sbuild-* /etc/schroot/chroot.d/raring-armhf-cross
 
  +
# vi /etc/schroot/chroot.d/raring-armhf-cross
 
  +
This section describes how to use traditional cross compilation to build the Xen tools for [[Xen ARM with Virtualization Extensions|Xen on ARM]] using the [http://wiki.debian.org/Multiarch Multiarch] infrastructure available in Debian and Ubuntu to create a cross build chroot environment using (and targeting) Ubuntu Saucy Salamander.
''[raring-amd64-sbuild]'' | '''[raring-armhf-cross]'''
 
  +
  +
In principal it might be possible to use multiarch on the host to setup a cross environment without using a chroot, however multiarch does not currently support both native and cross compilation at the same time so it is easier to put the cross environment into a chroot.
  +
  +
== Creating a base chroot ==
  +
  +
In order to setup a crossbuild chroot we first need a base chroot. Note that the 32 bit arm architecture in Debian and Ubuntu is called ''armhf''. If you are intending to build for 64-bit substitute '''arm64''' for '''armhf''' everywhere in the following.
  +
  +
Create the initial base chroot. This will be a native host (i.e. x86) chroot into which we will install an ARM cross compile environment:
  +
  +
# sbuild-createchroot --components=main,universe saucy /srv/chroots/saucy-armhf-cross http://archive.ubuntu.com/ubuntu/
  +
  +
This creates /etc/schroot/chroot.d/saucy-amd64-sbuild-* (with a random suffix) and a chroot named saucy-amd64-sbuild using the ''sbuild'' profile. The profile should be changed to ''default''. Also the naming is confusing if you create multiple cross build chroots. So rename the file and edit the configuration file as shown:
  +
  +
# mv /etc/schroot/chroot.d/saucy-amd64-sbuild-* /etc/schroot/chroot.d/saucy-armhf-cross
  +
# vi /etc/schroot/chroot.d/saucy-armhf-cross
  +
''[saucy-amd64-sbuild]'' | '''[saucy-armhf-cross]'''
 
type=directory | type=directory
 
type=directory | type=directory
''description=Debian raring/amd64 autobuilder'' | '''description=Debian raring/armhf crossbuilder'''
+
''description=Debian saucy/amd64 autobuilder'' | '''description=Debian saucy/armhf crossbuilder'''
directory=/srv/chroots/raring-armhf-cross | directory=/srv/chroots/raring-armhf-cross
+
directory=/srv/chroots/saucy-armhf-cross | directory=/srv/chroots/saucy-armhf-cross
 
groups=root,sbuild | groups=root,sbuild
 
groups=root,sbuild | groups=root,sbuild
 
root-groups=root,sbuild | root-groups=root,sbuild
 
root-groups=root,sbuild | root-groups=root,sbuild
 
''profile=sbuild'' | '''profile=default'''
 
''profile=sbuild'' | '''profile=default'''
   
You now have a base chroot named ''raring-armhf-cross'' or ''raring-arm64-cross''.
+
You now have a base chroot named ''saucy-armhf-cross'' or ''saucy-arm64-cross''.
   
= 32-bit crossbuild =
+
== 32-bit crossbuild ==
   
== Configuring an armhf crossbuild chroot ==
+
=== Configuring an armhf crossbuild chroot ===
   
After creating a base ''raring-armhf-cross'' chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:
+
After creating a base ''saucy-armhf-cross'' chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:
   
# schroot -c raring-armhf-cross
+
# schroot -c saucy-armhf-cross
   
 
Install some basic utilities:
 
Install some basic utilities:
Line 73: Line 98:
 
(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler
 
(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler
   
Configure the package sources, qualifying the main repositories as ''amd64'' only and adding the ''armhf'' repostories from the ports:
+
Configure the package sources, qualifying the main repositories as ''amd64'' only and adding the ''armhf'' repositories from the ports archive:
   
 
(chroot)# vi /etc/apt/sources.list
 
(chroot)# vi /etc/apt/sources.list
deb '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ raring main universe
+
deb '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ raring main universe
+
deb-src '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ saucy main universe
 
 
'''deb [arch=armhf] http://ports.ubuntu.com/ raring main universe'''
+
'''deb [arch=armhf] http://ports.ubuntu.com/ saucy main universe'''
   
 
Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create ''/etc/apt/apt.conf.d/30norecommends'' containing:
 
Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create ''/etc/apt/apt.conf.d/30norecommends'' containing:
Line 98: Line 123:
 
(chroot)# exit
 
(chroot)# exit
   
== Build arm32 tools ==
+
=== Build arm32 tools ===
   
 
Enter your 32-bit crossbuild chroot as your regular user:
 
Enter your 32-bit crossbuild chroot as your regular user:
   
$ schroot -c raring-armhf-cross
+
$ schroot -c saucy-armhf-cross
   
 
Change to the directory where you have cloned Xen and cross compile with:
 
Change to the directory where you have cloned Xen and cross compile with:
Line 111: Line 136:
 
That's all there is to it. You should now have a ''dist/install'' directory containing the installed bits which can be copied into your arm32 rootfs.
 
That's all there is to it. You should now have a ''dist/install'' directory containing the installed bits which can be copied into your arm32 rootfs.
   
  +
== 64-bit crossbuild ==
/usr/include/libfdt.h:54:24: fatal error: libfdt_env.h: No such file or directory
 
If you meet this error while compiling, it is probably a bug from libfdt-dev 1.3.0-2build1 in raring packages. This problem was fixed in 1.3.0-3, so upgrading the package would fix this.[https://bugs.launchpad.net/ubuntu/+source/device-tree-compiler/+bug/1203916]
 
   
  +
=== Introduction ===
= 64-bit crossbuild =
 
   
  +
These instructions have been massively simplified with the release of Saucy Salamander vs the previous Raring Ringtail release. They are essentially identical to the armhf variant.
== Introduction ==
 
   
  +
=== Configuring an arm64 crossbuild chroot ===
The arm64 port of Debian & Ubuntu is currently a work in progress and is not formally part of either distributions. The following uses Wookey's [http://people.debian.org/~wookey/bootstrap.html ARMv8 bootstrap repositories] to provide the necessary crossbuild tooling. Note that this is a work in progress and therefore something of a moving target so things are not as simple as with the 32-bit armhf version above.
 
   
  +
After creating a base ''saucy-arm64-cross'' chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:
The main rough edge is that where you have both the ''amd64'' (native) and ''arm64'' (cross) versions of a library installed their versions must match ''exactly''. However it is not unusual for the main Ubuntu archive (containing the ''amd64'' packages) to get ahead of Wookey's archive (containing the ''arm64'' packages). This manifests as ''apt-get'' refusing to install a development package because of a dependency on a particular library. If this happens then you need to find the version of the ''amd64'' package matching the current version in Wookey's repo and install it by hand. Note that you may need to do this recursively over the dependencies of the package you are trying to install.
 
   
  +
# schroot -c saucy-arm64-cross
For example trying to install ''libglib2.0-dev:arm64'':
 
(chroot)# apt-get install libglib2.0-dev:arm64
 
Reading package lists... Done
 
Building dependency tree
 
Reading state information... Done
 
Some packages could not be installed. This may mean that you have
 
requested an impossible situation or if you are using the unstable
 
distribution that some required packages have not yet been created
 
or been moved out of Incoming.
 
The following information may help to resolve the situation:
 
 
The following packages have unmet dependencies:
 
libglib2.0-dev:arm64 : Depends: libglib2.0-0:arm64 (= 2.35.8-0ubuntu1profile1) but it is not going to be installed
 
Depends: libpcre3-dev:arm64 (>= 1:8.31) but it is not going to be installed
 
E: Unable to correct problems, you have held broken packages.
 
 
It is complaining about the ''libpcre3'' development package and therefore we look at the ''libpcre3'' library package. Looking at the versions available for ''amd64'' and ''arm64'':
 
(chroot)# apt-cache policy libpcre3
 
libpcre3:
 
Installed: 1:8.31-2
 
Candidate: 1:8.31-2
 
Version table:
 
*** '''1:8.31-2''' 0
 
500 http://archive.ubuntu.com/ubuntu/ raring/main amd64 Packages
 
100 /var/lib/dpkg/status
 
(chroot)# apt-cache policy libpcre3:arm64
 
libpcre3:arm64:
 
Installed: (none)
 
Candidate: 1:8.31-1ubuntu1
 
Version table:
 
'''1:8.31-1ubuntu1''' 0
 
1001 http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap/main arm64 Packages
 
 
Here we see that Ubuntu currently contains version ''1:8.31-2'' of the package for ''amd64'' but the ''arm64'' repo only contains ''1:8.31-1ubuntu1''. Therefore we need to downgrade the ''amd64'' version by visiting https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1 and downloading libpcre3_8.31-1ubuntu1_amd64.deb which we install by hand.
 
 
The instructions below incorporate the necessary workarounds of this type as required on 2013-03-14. The set of these workarounds which are required will vary over time.
 
 
== Configuring an arm64 crossbuild chroot ==
 
 
After creating a base ''raring-arm64-cross'' chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:
 
 
# schroot -c raring-arm64-cross
 
   
 
Install some basic utilities:
 
Install some basic utilities:
Line 169: Line 152:
 
(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler
 
(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler
   
Configure the package sources, qualifying the main repositories as ''amd64'' only and adding Wookey's ''arm64'' bootstrap repostory:
+
Configure the package sources, qualifying the main repositories as ''amd64'' only and adding the ''arm64'' repositories from the ports archive:
   
 
(chroot)# vi /etc/apt/sources.list
 
(chroot)# vi /etc/apt/sources.list
deb '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ raring main universe
+
deb '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ raring main universe
+
deb-src '''[arch=amd64]''' http://archive.ubuntu.com/ubuntu/ saucy main universe
 
 
'''deb http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe'''
+
'''deb [arch=arm64] http://ports.ubuntu.com/ saucy main universe'''
'''deb-src http://people.debian.org/~wookey/bootstrap/ubunturepo/ raring-bootstrap main universe'''
 
 
We must also arrange to trust Wookey's repository:
 
(chroot)# wget http://people.debian.org/~wookey/bootstrap/bootstrap-archive.key
 
(chroot)# apt-key add bootstrap-archive.key
 
(validating the chain of trust on this key is left as an exercise for the reader)
 
 
Configure the bootstrap repository to have higher priority than the standard one (this avoids some subset of the problematic cases described above and generally makes things easier). Create ''/etc/apt/preferences.d/bootstrap'' containing:
 
Package: *
 
Pin: release n=raring-bootstrap
 
Pin-Priority: 1001
 
   
 
Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create ''/etc/apt/apt.conf.d/30norecommends'' containing:
 
Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create ''/etc/apt/apt.conf.d/30norecommends'' containing:
Line 200: Line 172:
   
 
Next install the build dependencies required to build Xen:
 
Next install the build dependencies required to build Xen:
(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64
+
(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64 libfdt-dev:arm64
   
  +
We also need the ''autotools-dev'' package to workaround out of date autoconf machinery in the Xen 4.4.0 release:
This will possibly fail due to the package skew issue described above. As described these must be resolved manually. On 2013-03-14 this involved the following. YMMV
 
  +
(chroot)# apt-get install autotools-dev
 
(chroot)# wget http://launchpadlibrarian.net/126062829/libpcre3_8.31-1ubuntu1_amd64.deb # via ''https://launchpad.net/ubuntu/raring/amd64/libpcre3/1:8.31-1ubuntu1''
 
(chroot)# dpkg -i libpcre3_8.31-1ubuntu1_amd64.deb
 
 
(chroot)# wget http://launchpadlibrarian.net/129614595/libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb # via ''https://launchpad.net/ubuntu/raring/amd64/libpython2.7-minimal/2.7.3-15ubuntu2''
 
(chroot)# wget http://launchpadlibrarian.net/129614593/libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb # via ''https://launchpad.net/ubuntu/raring/amd64/libpython2.7-stdlib/2.7.3-15ubuntu2''
 
(chroot)# apt-get install libexpat1 libsqlite3-0 mime-support
 
(chroot)# dpkg -i libpython2.7-stdlib_2.7.3-15ubuntu2_amd64.deb libpython2.7-minimal_2.7.3-15ubuntu2_amd64.deb
 
 
(chroot)# wget http://launchpadlibrarian.net/132602064/libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb # via ''https://launchpad.net/ubuntu/raring/amd64/libssl1.0.0/1.0.1c-4ubuntu5''
 
(chroot)# dpkg -i libssl1.0.0_1.0.1c-4ubuntu5_amd64.deb
 
 
Now we have done that we can try installing Xen's build dependencies again:
 
(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git
 
 
Lastly there are a couple of required libraries which are not available in the arm64 repository yet. I have made available versions of ''libaio'' (patched with the fix to [http://bugs.debian.org/702409 Debian bug #702409]) and ''yajl'' (patched with the fix to [http://bugs.debian.org/702357 Debian bug #702357]):
 
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio-dev_0.3.109-3ijc1arm64_arm64.deb
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_amd64.deb
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libaio1_0.3.109-3ijc1arm64_arm64.deb
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl-dev_2.0.4-2ijc1arm64_arm64.deb
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_amd64.deb
 
(chroot)# wget http://xenbits.xen.org/people/ianc/raring/arm64/libyajl2_2.0.4-2ijc1arm64_arm64.deb
 
 
(chroot)# dpkg -i libaio-dev_0.3.109-3ijc1arm64_arm64.deb libaio1_0.3.109-3ijc1arm64_amd64.deb libaio1_0.3.109-3ijc1arm64_arm64.deb libyajl-dev_2.0.4-2ijc1arm64_arm64.deb libyajl2_2.0.4-2ijc1arm64_amd64.deb libyajl2_2.0.4-2ijc1arm64_arm64.deb
 
   
 
We have now finished configuring the chroot, so exit:
 
We have now finished configuring the chroot, so exit:
 
(chroot)# exit
 
(chroot)# exit
   
== Build arm64 tools ==
+
=== Build arm64 tools ===
   
 
Enter your 64-bit crossbuild chroot as your regular user:
 
Enter your 64-bit crossbuild chroot as your regular user:
   
$ schroot -c raring-arm64-cross
+
$ schroot -c saucy-arm64-cross
   
Change to the directory where you have cloned Xen and cross compile. Note that we need to update some autoconf machinery with versions which know about arm64.
+
Change to the directory where you have cloned Xen and cross compile.
  +
  +
If building Xen 4.4.0 then it is first necessary to update the autoconf machinery with versions which know about arm64:
   
 
(chroot)$ cp /usr/share/misc/config.{sub,guess} .
 
(chroot)$ cp /usr/share/misc/config.{sub,guess} .
  +
  +
Now we can cross compile:
  +
 
(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
 
(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
 
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64
 
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64
   
That's all (!) there is to it. You should now have a ''dist/install'' directory containing the installed bits which can be copied into your arm64 rootfs.
+
You should now have a ''dist/install'' directory containing the installed bits which can be copied into your arm64 rootfs.
   
= References =
+
== References ==
   
 
The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.
 
The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.
   
[[Category:XenARM]] [[Category:Developers]]
+
[[Category:XenARM]] [[Category:Developers]] [[Category:Yocto]]

Latest revision as of 17:04, 20 May 2020

Icon Ambox.png This page is outdated. Please refer to Xen_ARM_with_Virtualization_Extensions#Building_Xen_on_ARM


Introduction

When working with low power or constrained environments (and in particular when using software emulated platforms such as the Fast Models) it may not be desirable (or even possible) to build software, such as the Xen userspace tools, on the target device itself. In such cases it is necessary to build on some other more powerful/capable/suitable device. If a more powerful system of the same architecture is available then this can be achieved by simply building on that device and transferring the result to the target device. However if a more powerful system of the same architecture is not available then the software must be "cross-compiled", that is built on a system of a different host architecture using tools which produce binaries that will run on the target architecture. In the most common case this will involve using an x86 host machine to build binaries for an ARM system.

There are three main ways to set up a cross compile environment:

Yocto and meta-virtualization
See this page to use Yocto to cross-compile a minimal dom0 initramfs, including all the Xen userspace tools.
Construct a traditional cross compilation environment
This technique involves installing a compiler on the host system which runs natively on the host but produces binaries for the target system and making available all the of the libraries required for the target architecture in order to build the software in question. This is what is normally meant when people talk about cross compiling. Making all of the necessary libraries available is sometimes easier said than done however and many project's build systems do not cope well with cross-compilation (the Xen tools are cross compile friendly though).
Using a foreign chroot
This technique involves creating a chroot on the host containing a distro for the target system and using the "native" tools and libraries within this chroot to build for the target using the native build processes. This relies on the Qemu system emulator to emulate a userspace environment for the target system (which is much more efficient than doing full system emulation of the target since it uses the host system's native kernel). This method is not normally as fast as traditional cross compilation but it is significantly faster than building using full system emulation and is often faster than building natively on a low powered device. This basics of this are described in a blog post Foreign Chroots with schroot and qemu while a more specific example covering arm64 using openSUSE is described in Virtualization on ARM with Xen.

Target Environment

Yocto automatically sets up the cross-compilation environment for you. If you don't use Yocto, it is important that the build (i.e. cross or foreign chroot) environment matches the runtime (i.e. dom0) environment where the tools will run. This means that the library and compiler versions etc should match. These instructions use the Ubuntu Saucy Salamander release and so that is what you would need in your dom0 as well. See Xen_ARM_with_Virtualization_Extensions/RootFilesystem for instructions on creating a root filesystem.

Xen Version

Xen on ARM is cross buildable on an x86 host from Xen 4.4 onwards.

Typographical Conventions

#
Commands to run as root on the host
$
Commands to run as your regular user on the host
(chroot)#
Commands to run as root within the chroot
(chroot)$
Commands to run as the regular user in the chroot
$USER
Your regular username on the host (which is propagated to the chroot)

Using sbuild and schroot

These instructions use the sbuild and schroot tools, which are part of Debian and Ubuntu, in order to provide a convenient mechanism for creating chroots.

These instructions were tested with sbuild version 0.63.2-1.1 and schroot version 1.6.4-4 on a Debian Wheezy system.

sbuild can be installed as follows, which will also pull in schroot as a dependency:

# apt-get install sbuild
# sbuild-adduser $USER

This installs the sbuild tool and configures your existing user to be able to use it. You may need to logout and log back in for this to take affect.

If sbuild and schroot are not available

If your host distribution does not supply sbuild then it is still possible to use a chroot in a more manual fashion. This is mostly out of scope for this document but some hints:

  • sbuild will automatically bind mount things inside the chroot, so if running without you will need to ensure that your source trees etc are available within the chroot, either by bind mounting manually or by copying the source tree into the chroot.
  • sbuild will automatically propagate any necessary host level configuration to the chroot, which you may need to do by hand. e.g. you would likely need to copy /etc/resolv.conf into the chroot in order to access the network while within the chroot.
  • sbuild will automatically make your $USER available inside the chroot. Either run as root within the chroot (not really recommended) or ensure that your $USER exists within the chroot with the same uid and gid.

Cross-compile chroot environment using multiarch

This section describes how to use traditional cross compilation to build the Xen tools for Xen on ARM using the Multiarch infrastructure available in Debian and Ubuntu to create a cross build chroot environment using (and targeting) Ubuntu Saucy Salamander.

In principal it might be possible to use multiarch on the host to setup a cross environment without using a chroot, however multiarch does not currently support both native and cross compilation at the same time so it is easier to put the cross environment into a chroot.

Creating a base chroot

In order to setup a crossbuild chroot we first need a base chroot. Note that the 32 bit arm architecture in Debian and Ubuntu is called armhf. If you are intending to build for 64-bit substitute arm64 for armhf everywhere in the following.

Create the initial base chroot. This will be a native host (i.e. x86) chroot into which we will install an ARM cross compile environment:

# sbuild-createchroot --components=main,universe saucy /srv/chroots/saucy-armhf-cross http://archive.ubuntu.com/ubuntu/

This creates /etc/schroot/chroot.d/saucy-amd64-sbuild-* (with a random suffix) and a chroot named saucy-amd64-sbuild using the sbuild profile. The profile should be changed to default. Also the naming is confusing if you create multiple cross build chroots. So rename the file and edit the configuration file as shown:

# mv /etc/schroot/chroot.d/saucy-amd64-sbuild-* /etc/schroot/chroot.d/saucy-armhf-cross
# vi /etc/schroot/chroot.d/saucy-armhf-cross
[saucy-amd64-sbuild]                         | [saucy-armhf-cross]
type=directory                               | type=directory
description=Debian saucy/amd64 autobuilder   | description=Debian saucy/armhf crossbuilder
directory=/srv/chroots/saucy-armhf-cross     | directory=/srv/chroots/saucy-armhf-cross
groups=root,sbuild                           | groups=root,sbuild
root-groups=root,sbuild                      | root-groups=root,sbuild
profile=sbuild                               | profile=default

You now have a base chroot named saucy-armhf-cross or saucy-arm64-cross.

32-bit crossbuild

Configuring an armhf crossbuild chroot

After creating a base saucy-armhf-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c saucy-armhf-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the armhf repositories from the ports archive:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
   
deb [arch=armhf] http://ports.ubuntu.com/ saucy main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add armhf as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture armhf
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-armhf

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:armhf libncurses-dev:armhf uuid-dev:armhf libglib2.0-dev:armhf libssl-dev:armhf libssl-dev:armhf libaio-dev:armhf libyajl-dev:armhf python gettext gcc git libpython2.7-dev:armhf libfdt-dev:armhf

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm32 tools

Enter your 32-bit crossbuild chroot as your regular user:

$ schroot -c saucy-armhf-cross

Change to the directory where you have cloned Xen and cross compile with:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.armhf ./configure --build=x86_64-unknown-linux-gnu --host=arm-linux-gnueabihf
(chroot)$ make dist-tools CROSS_COMPILE=arm-linux-gnueabihf- XEN_TARGET_ARCH=arm32

That's all there is to it. You should now have a dist/install directory containing the installed bits which can be copied into your arm32 rootfs.

64-bit crossbuild

Introduction

These instructions have been massively simplified with the release of Saucy Salamander vs the previous Raring Ringtail release. They are essentially identical to the armhf variant.

Configuring an arm64 crossbuild chroot

After creating a base saucy-arm64-cross chroot as described above we then configure it to add the multiarch cross capabilities. Enter the chroot as root with:

# schroot -c saucy-arm64-cross

Install some basic utilities:

(chroot)# apt-get install vim-tiny wget sudo less pkgbinarymangler

Configure the package sources, qualifying the main repositories as amd64 only and adding the arm64 repositories from the ports archive:

(chroot)# vi /etc/apt/sources.list
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
deb-src [arch=amd64] http://archive.ubuntu.com/ubuntu/ saucy main universe
   
deb [arch=arm64] http://ports.ubuntu.com/ saucy main universe

Since this is a cross-build chroot Recommended and Suggested packages are largely unnecessary, create /etc/apt/apt.conf.d/30norecommends containing:

APT::Install-Recommends "0";
APT::Install-Suggests "0";

Now add arm64 as an additional architecture and install the basic crossbuild infrastructure:

(chroot)# dpkg --add-architecture arm64
(chroot)# apt-get update
(chroot)# apt-get install crossbuild-essential-arm64

Next install the build dependencies required to build Xen:

(chroot)# apt-get install libc6-dev:arm64 libncurses-dev:arm64 uuid-dev:arm64 libglib2.0-dev:arm64 libssl-dev:arm64 libssl-dev:arm64 libaio-dev:arm64 libyajl-dev:arm64 python gettext gcc git libpython2.7-dev:arm64 libfdt-dev:arm64

We also need the autotools-dev package to workaround out of date autoconf machinery in the Xen 4.4.0 release:

(chroot)# apt-get install autotools-dev

We have now finished configuring the chroot, so exit:

(chroot)# exit

Build arm64 tools

Enter your 64-bit crossbuild chroot as your regular user:

$ schroot -c saucy-arm64-cross

Change to the directory where you have cloned Xen and cross compile.

If building Xen 4.4.0 then it is first necessary to update the autoconf machinery with versions which know about arm64:

(chroot)$ cp /usr/share/misc/config.{sub,guess} .

Now we can cross compile:

(chroot)$ CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu
(chroot)$ make dist-tools CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64

You should now have a dist/install directory containing the installed bits which can be copied into your arm64 rootfs.

References

The procedure here is based somewhat loosely on https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/arm64bootstrap plus Wookey's kind advice at Linaro Connect.