Difference between revisions of "Xen ARM with Virtualization Extensions"

From Xen
(Requirements)
(145 intermediate revisions by 23 users not shown)
Line 1: Line 1:
 
{{Hatnote|For the ARM port supporting paravirtualized guests on processors without the virtualization extensions see [[Xen ARM (PV)]].}}
 
{{Hatnote|For the ARM port supporting paravirtualized guests on processors without the virtualization extensions see [[Xen ARM (PV)]].}}
   
The ARM v7-A architecture includes the optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions will be available in some ARM Cortex A15 and Cortex A7 processors.
+
The ARM v7-A and ARM v8-A architectures include optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions are currently available in some ARM v7 processors such as the Cortex A15 and Cortex A7.
   
  +
== Introduction ==
This port is a ''work-in-progress'' and cannot yet start any guests (other than dom0).
 
  +
If you are interested in the Xen on ARM architecture and how it compares to Xen on x86, read the [[Xen_ARM_with_Virtualization_Extensions_whitepaper| Xen on ARM whitepaper]].
  +
  +
== Status ==
  +
  +
Both the 32-bit (arm32) and the 64-bit (arm64) ports of Xen boot dom0 and unprivileged guests can be created and destroyed using ''xl''. See below for information on the hardware and models. You may also want to read [[Xen ARM with Virtualization Extensions whitepaper|our whitepaper]] that explains the basic architecture and terminology of Xen on ARM.
  +
  +
=== Guest ABI ===
  +
  +
The guest (including dom0) visible [http://xenbits.xen.org/docs/unstable/hypercall/arm/index.html ABI exposed by the hypervisor] has been declared stable as of the 4.4 release and will now be maintained in a backwards compatible manner.
  +
  +
== Contributing ==
  +
  +
Please email [mailto:xen-devel@lists.xen.org xen-devel] with comments, questions and patches. Please see the [http://lists.xen.org/xen-devel list info page] for subscription information and the [http://lists.xen.org/archives/html/xen-devel/ archives]. For patches please see [[Submitting Xen Patches]].
  +
  +
== Testing ==
  +
There are a lot of different ARM boards and servers with many different IP blocks available. The project's CI loop can only test a small subset of these, as we cannot afford to buy and test against all possible boards. As such, we will be relying on manual testing during [[Xen Project Test Days]] to verify which ARM hardware works and which doesn't. The good news, is that from experience the vast majority of board specific issues will occur during boot and thus a a simple [[Xen ARM Manual Smoke Test]] is usually sufficient to identify such issues.
  +
  +
The [[Xen ARM Manual Smoke Test/Results]] contains a list of people who have specific boards and also lists when these boards have been last tested. Feel free to add any test results to the table. By doing so you will help others and yourself.
   
 
== Hardware ==
 
== Hardware ==
   
  +
{|class="prettytable" style="text-align: left;" valign="top"
Currently there are no processors available which implement the virtualisation extensions. Therefore we are currently targetting the Real-time System Model's provided by ARM. See [[Xen ARMv7 with Virtualization Extensions/FastModels]] for more information on obtaining and building the necessary model.
 
  +
!Core/SoC/Board
  +
!Xen Guide
  +
!Notes
  +
|-
  +
!colspan="3"|ARM Cortex [http://www.arm.com/products/processors/cortex-a/cortex-a7.php A7]/[http://www.arm.com/products/processors/cortex-a/cortex-a15.php A15]
  +
|-
  +
|ARM Cortex A7/A15 [http://www.arm.com/products/tools/models/fast-models/index.php Real-time System Model] (FVP)
  +
|[[Xen ARM with Virtualization Extensions/FastModels|Fastmodels]]
  +
|Commercial emulator
  +
|-
  +
|[http://www.arm.com/products/tools/development-boards/versatile-express/ Versatile Express]
  +
|[[Xen ARM with Virtualization Extensions/Vexpress|Versatile Express]]
  +
|With TC2 daughterboard
  +
|-
  +
|Calxeda EXC-2000
  +
|[[Xen ARM with Virtualization Extensions/Midway|Midway]]
  +
|
  +
|-
  +
|colspan="3"|'''[http://www.allwinnertech.com/ Allwinner] sunxi'''
  +
|-
  +
| [http://www.allwinnertech.com/en/clq/processora/A20.html sun7i/A20]
  +
|[[Xen ARM with Virtualization Extensions/Allwinner|Allwinner]]
  +
|[http://linux-sunxi.org/A20 linux-sunxi community]. e.g. [http://linux-sunxi.org/Cubietech_Cubietruck Cubietruck]
  +
|-
  +
| sun6i/A31
  +
|
  +
|[http://linux-sunxi.org/A31 linux-sunxi community]
  +
|-
  +
|colspan="3"|'''Exynos5xxx'''
  +
|-
  +
| [http://www.samsung.com/global/business/semiconductor/product/application/detail?productId=7668 Exynos5250]
  +
|[[Xen ARM with Virtualization Extensions/Arndale|Arndale]]
  +
|[http://www.arndaleboard.org/wiki/index.php/Main_Page www.arndaleboard.org]
  +
|-
  +
| [http://www.samsung.com/global/business/semiconductor/product/application/detail?productId=7976&iaId=2341 Exynos5410]
  +
|[[Xen ARM with Virtualization Extensions/OdroidXU|OdroidXU]]
  +
|[http://www.hardkernel.com/main/products/prdt_info.php?g_code=G137510300620 www.hardkernel.com]
  +
|-
  +
|colspan="3"|'''OMAP5'''
  +
|-
  +
| [http://www.ti.com/product/omap5432 OMAP5432]
  +
|[[Xen ARM with Virtualization Extensions/OMAP5432_uEVM|uEVM]]
  +
|[http://www.ti.com/tool/omap5432-evm www.ti.com]
  +
|-
  +
|colspan="3"|'''Renesas R-Car H2/H3'''
  +
|-
  +
|[http://am.renesas.com/applications/automotive/cis/cis_highend/rcar_h2/index.jsp Renesas R-Car H2]
  +
|[[Xen ARM with Virtualization Extensions/Lager|Lager]]
  +
|-
  +
|[https://elinux.org/R-Car/Boards/Stout Renesas R-Car H2]
  +
|[[Xen ARM with Virtualization Extensions/Stout|Stout]]
  +
|-
  +
|[https://www.renesas.com/en-us/solutions/automotive/products/rcar-h3.html Renesas R-Car H3]
  +
|[[Xen ARM with Virtualization Extensions/Salvator-X|Salvator-X]]
  +
|-
  +
!colspan="3"|ARM Cortex [http://arm.com/products/processors/cortex-a/cortex-a53-processor.php A53]/[http://arm.com/products/processors/cortex-a/cortex-a57-processor.php A57]
  +
|-
  +
|[https://www.qemu.org/ QEMU AArch64 Emulator]
  +
|[[Xen ARM with Virtualization Extensions/qemu-system-aarch64|QEMU]]
  +
|Fast Open Source emulator
  +
|-
  +
|[http://www.arm.com/products/tools/models/fast-models/foundation-model.php ARMv8 Foundation Model]
  +
|[[Xen ARM with Virtualization Extensions/FastModels|Fastmodels]]
  +
|Free as in beer emulator
  +
|-
  +
|ARM AEMv8 [http://www.arm.com/products/tools/models/fast-models/index.php Real-time System Model]
  +
|[[Xen ARM with Virtualization Extensions/FastModels|Fastmodels]]
  +
|Commercial emulator
  +
|-
  +
|ARM Cortex A53/A57 [http://www.arm.com/products/tools/models/fast-models/index.php Real-time System Model]
  +
|[[Xen ARM with Virtualization Extensions/FastModels|Fastmodels]]
  +
|Commercial emulator
  +
|-
  +
![https://www.apm.com/products/data-center/x-gene-family/ Applied Micro X-Gene]
  +
||[https://www.apm.com/products/data-center/x-gene-family/x-gene/ X-Gene]
  +
|[[Xen ARM with Virtualization Extensions/APMXGeneMustang|Mustang]] (XC-1), HP Moonshot (McDivitt)
  +
|
  +
|-
  +
![https://www.96boards.org/products/ce/hikey HiKey board from 96boards.org]
  +
|[[HiKey|HiKey]]
  +
|Open Hardware Platform
  +
|-
  +
![http://www.96boards.org/product/hikey960/ HiKey960 board from 96boards.org]
  +
|[[HiKey960|HiKey960]]
  +
|Open Hardware Platform
  +
|-
  +
![http://www.xilinx.com/products/technology/ultrascale-mpsoc.html Xilinx Zynq UltraScale+ MPSoC (ZynqMP)]
  +
||[http://www.wiki.xilinx.com/XEN+Hypervisor Xilinx Wiki]
  +
|Open Source Emulator [http://www.wiki.xilinx.com/QEMU+-+Zynq+UltraScalePlus QEMU for ZynqMP]
  +
|-
  +
![http://wiki.espressobin.net/tiki-index.php ESPRESSObin based on Marvell ARMADA 3700]
  +
|[[Xen ARM with Virtualization Extensions/ESPRESSObin|ESPRESSObin]]
  +
|Hardware Platform
  +
|-
  +
![http://linux-sunxi.org/A64 Allwinner A64]/[http://linux-sunxi.org/H5 H5] based boards
  +
|[[Xen ARM with Virtualization Extensions/Allwinner|Allwinner]]
  +
|boards like: [http://linux-sunxi.org/Pine64 Pine64], [http://linux-sunxi.org/Xunlong_Orange_Pi_PC_2 OrangePi PC 2]
  +
|-
  +
!colspan="3"|ARM Cortex [https://developer.arm.com/products/processors/cortex-a/cortex-a53 A53]/[https://developer.arm.com/products/processors/cortex-a/cortex-a72 A72]
  +
|-
  +
![http://www.9tripod.com/showpro.php?id=93 Ibox3399 board based on Rockchip RK3399]
  +
|[[Xen ARM with Virtualization Extensions/Ibox3399|Ibox3399]]
  +
|boards like: [https://www.pine64.org/rockpro64 RockPro64]
  +
|-
  +
  +
|}
  +
  +
  +
[[Xen ARM with Virtualization Extensions/Chromebook|Early work]] was also done to support the [http://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/samsung-arm-chromebook Google Chromebook], however this is no longer continued since it is not a good development platform.
   
 
== Requirements ==
 
== Requirements ==
   
;ARM Fast Model
+
;ARM Hardware or Software Model
:Either the v7-A ARM Envelope Model (AEM) or the Cortex A15/A7 models (single CPU only).
+
:See above for details of the hardware and models which are supported.
  +
;Firmware
  +
:Xen requires certain functionality from the system firmware. See below for details.
 
;Device Tree
 
;Device Tree
:A [[device tree]] in the flat device tree format (.dtb). The device tree source for the AEM is here: [http://xenbits.xen.org/gitweb/?p=people/dvrabel/device-trees.git;a=summary git://xenbits.xen.org/people/dvrabel/device-trees.git].
+
:A [[device tree]] in the flat device tree format (.dtb). The host platform must be described in a DTB binary passed to Xen at boot time. This will be used by Xen and Dom0. Normally the regular device tree used when booting natively on the platform should be used.
 
;Xen
 
;Xen
  +
:All current work is now merged into the current development branch [http://xenbits.xen.org/gitweb/?p=xen.git;a=summary git://xenbits.xen.org/xen.git]. It is recommended to use the latest Xen master branch.
:xen-unstable as of 13 Feb 2012.
 
 
;Linux kernel for dom0
 
;Linux kernel for dom0
  +
:The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible.
:vexpress-dt branch of [http://xenbits.xen.org/gitweb/?p=people/dvrabel/linux.git;a=summary git://xenbits.xen.org/people/dvrabel/linux.git]
 
 
;dom0 userspace
 
;dom0 userspace
  +
:The developers are using the armhf port of Debian Wheezy.
:(more details here)
 
  +
;domU kernel
  +
:The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.
   
  +
=== Hypervisor ABI Compatibility ===
== Device Trees ==
 
   
  +
The ABI for Xen on ARM was declared stable from Xen 4.4 onwards.
Xen needs the device trees to be in the flat device tree format (the ''device tree blob'' or ''DTB''). This is built by the ''device tree compiler'' (''dtc'') from the device tree source files (<tt>.dts</tt> and <tt>.dtsi</tt>.
 
   
  +
This ABI is implemented by Linux mainline v3.9-rc1 onwards.
Version 1.3 or later of ''dtc'' is required.
 
   
  +
Although not a hypervisor ABI change Linux versions prior to v3.13-rc5 (specifically [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=380108d891acf8db5cf0d477176c7ed2b62b7928 380108d891ac "xen/block: Correctly define structures in public headers on ARM32 and ARM64"]) incorrectly defined the PV block protocol on ARM. This means that versions of Linux prior to this fix are only able to interoperate (as either front or backend) with other versions of Linux prior to this commit. After this fix Linux correctly uses the same protocol as other ports (e.g. BSD).
Build the <tt>.dtb</tt> files with <tt>make</tt>.
 
   
  +
=== Firmware Requirements ===
A pre-built [http://xenbits.xen.org/people/dvrabel/xen-arm/vexpress-v2p-aem-v7a.dtb vexpress-v2p-aem-v7a.dtb].
 
   
  +
The primary requirement is that the hypervisor must be launched in Non-Secure Hypervisor mode only. If the stock firmware on a platform does not obey this (most commonly by launching in Secure Supervisor mode) then a firmware update may be required. This support is present in u-boot 2014.01.
== Xen ==
 
   
  +
Booting secondary processors on an SMP system requires firmware support for the Power State Coordination Interfaces (PSCI). Initial u-boot support for this interface is available in https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/log/?h=wip/psci .
When running on the model there is no bootloader to provide the DTB. Instead, it is linked with the xen image.
 
   
  +
When running Xen under a FastModel (which typically have no firmware at all) it is sometimes necessary to use a ''boot-wrapper''. See the [[Xen ARM with Virtualization Extensions/FastModels|Fast Model]] page for more information.
The DTB to use is specified by the CONFIG_DTB_FILE config variable which should be set to the absolute path of the DTB. This variable may be set in <tt>.config</tt> or on the make command line.
 
  +
  +
Xen previously included a workaround for firmware which lacked this functionality, however it was unmaintainable and interfered with proper support for other platforms and therefore as of September 2013 it has been removed.
  +
  +
== Booting Natively ==
  +
  +
Before starting to load Xen it is highly recommended to get the kernel you intend to use as dom0 booting natively (i.e. without Xen underneath). This will let you iron out any driver issues and figure out the necessary kernel command line etc before adding Xen into the mix.
  +
  +
== Booting Xen ==
  +
  +
=== ImageBuilder ===
  +
  +
Many of the details necessary to boot Xen from U-Boot can be generated automatically by ImageBuilder's '''uboot-script-gen''', see [[ImageBuilder]].
  +
  +
=== Boot Protocol ===
  +
  +
Xen's boot requirements are spelled out in [http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/booting.txt;h=9802e5e20fd8c7da94eaa7b639b7530b951760eb;hb=HEAD docs/misc/arm/booting.txt] in the Xen tree, which references the Linux [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/Booting arm] and [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm64/booting.txt arm64] booting documentation.
  +
  +
=== Device Trees ===
  +
  +
Xen needs the device trees to be in the flat device tree format (the ''device tree blob'' or ''DTB'').
  +
  +
It is no longer necessary to build a specific DTB for use with Xen. The Device Tree files shipped with [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/ Linux] or from the [http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git Split Device Tree Repository] can be used.
  +
  +
=== Boot Modules ===
  +
  +
At boot time Xen must be provided with a dom0 kernel blob and an optional dom0 initramfs blob. The bootloader must load these into memory and describe their location in the Device Tree Blob using the bindings specified in [http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=8da1e0b8fcf9c98888ed63cd45bd11f1a880288b;hb=HEAD docs/misc/arm/device-tree/booting.txt].
  +
  +
These nodes can either be added by hand (by editing and recompiling the .dts file) or by using u-boot's "fdt" command to add them dynamically at boot time:
  +
  +
fdt addr ${fdt_addr}
  +
fdt resize
  +
  +
fdt set /chosen \#address-cells <1>
  +
fdt set /chosen \#size-cells <1>
  +
  +
fdt mknod /chosen module@0
  +
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
  +
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
  +
fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"
  +
  +
(this assumes the device tree blob is loaded at ${fdt_addr} and the kernel at ${kernel_addr_r}).
  +
  +
See e.g. [[Xen_ARM_with_Virtualization_Extensions/Allwinner]] for a more concrete example of this.
  +
  +
=== Command Lines ===
  +
  +
[http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=8da1e0b8fcf9c98888ed63cd45bd11f1a880288b;hb=HEAD docs/misc/arm/device-tree/booting.txt] describes where Xen looks for both its own command line and the command line to pass to domain 0.
  +
  +
=== Getting Xen output ===
  +
  +
To get output log on the UART, Xen needs to know which UART to use. This should be passed in the hypervisor command line using the "dtuart" parameter. e.g.:
  +
  +
console=dtuart dtuart=myserial
  +
  +
where ''myserial'' is either an alias to the UART in the device tree (aliases are found in the '''aliases''' device tree node) or a full DTB path to the device. As Xen already uses it the UART will be disabled from the point of view of domain 0.
  +
  +
For instance, this is a dummy device tree (won't work) to use the uart0 in Xen:
  +
/ {
  +
choosen {
  +
bootargs = "console=dtuart dtuart=myserial";
  +
}
  +
aliases {
  +
myserial = &myserial_0;
  +
}
  +
myserial_0: uart0 {
  +
... configuration of your UART ...
  +
}
  +
}
  +
  +
Here dtuart is configured using the ''myserial'' alias. Alternatively ''/uart0'' (the full path to the device) could have been used.
  +
  +
'''Note''': If you don't see output from Xen, you can enable [http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/early-printk.txt;hb=HEAD early printk]. This option will turn on platform specific UART and output information before the console is initialized.
   
 
== Dom0 kernel ==
 
== Dom0 kernel ==
   
  +
In general the same kernel configuration as used to boot natively, plus turning on the Xen specific options should work. A good starting point is often the "multi_v7_defconfig" + Xen options.
Enable at least ARCH_VEXPRESS, ARCH_VEXPRESS_DT, ARM_APPENDED_DTB and ARM_ATAG_DTB_COMPAT. Disable SPARSE_IRQ (it doesn't play nicely with device tree support).
 
  +
  +
If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen. It is strongly recommended not to append a DTB to your dom0 kernel (or to disable APPENDED_DTB).
  +
  +
== DomU kernel and DTS ==
  +
  +
Unprivileged guests can be created using ''xl''. A simple VM config file would look like this:
  +
  +
kernel = "/root/image"
  +
memory = 128
  +
name = "guest"
  +
vcpus = 1
  +
disk = [ 'phy:/dev/loop0,xvda,w' ]
  +
extra = "earlyprintk=xenboot console=hvc0 root=/dev/xvda debug rw init=/bin/sh"
  +
  +
where "/root/image" is a Linux zImage.
  +
  +
=== Common DomU Pitfalls ===
  +
  +
; Enabling <tt>CONFIG_DEBUG_LL</tt> in the guest kernel configuration.
  +
: Although this option can work for dom0 if configured appropriately for the host it does not work for domU (which cannot see the host UART). The symptoms of this are that the guest console will be silent because the kernel has taken a fault accessing the early UART. This can be confirmed by using the <tt>xenctx</tt> tool (found in <tt>$PREFIX/lib/xen/bin/</tt>). The tool takes a numeric domid (not a name, use <tt>xl list</tt> or <tt>xl domid $name</tt>) and dumps the VCPU state. A PC of 0x0000000c will usually indicate that an early trap has occurred.
  +
  +
== Building Xen on ARM ==
  +
  +
{{WarningLeft|At least gcc version 4.7.3 is known to miscompile certain parts of Xen, most often leading to a segmentation fault in the xl toolstack while starting a guest. We recommend using 4.8.x or later}}
  +
  +
There are two major components which need to be built for a Xen system. The Xen hypervisor binary itself and the Xen toolstack.
  +
  +
=== Cross Compiling Xen ===
  +
  +
Cross compiling the Xen hypervisor is simple.
  +
  +
Linaro supply cross compilers for both arm32 (arm-linux-gnueabihf-) and arm64 (aarch64-linux-gnu-) via [https://launchpad.net/linaro-toolchain-binaries linaro-toolchain-binaries]. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from [http://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/ kernel.org].
  +
  +
Once you have a suitable cross compiler you can compile Xen with:
  +
  +
$ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf-
  +
  +
or:
  +
  +
$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
  +
  +
This assumes that the command prefix for you cross compiler is <tt>arm-unknown-linux-gnueabihf-</tt> or <tt>aarch64-linux-gnu-</tt> and that the appropriate <tt>arm-unknown-linux-gnueabihf-gcc</tt> or <tt>aarch64-linux-gnu-gcc</tt> and friends are in your $PATH.
  +
  +
  +
=== Building the Toolstack ===
  +
  +
For a complete cross-compilation of the hypervisor and the toolstack, it is recommended to use Yocto, see [[Xen_on_ARM_and_Yocto]].
  +
  +
  +
Alternatively, it is possible to use qemu user to run an ARM64 chroot on a x86 host, i.e. an ARM64 Debian or Ubuntu container on a regular x86 laptop.
  +
  +
$ apt-get install qemu-user-static
  +
  +
It installs <tt>/usr/bin/qemu-aarch64-static</tt>.
  +
  +
Next setup an ARM64 chroot environment on your x86 machine. Follow your distro recommandations. For instance, the following distros offer pre-packaged tarballs ready to be unpackged:
  +
  +
* Ubuntu: http://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04-base-arm64.tar.gz
  +
* Alpine Linux: http://dl-cdn.alpinelinux.org/alpine/v3.11/releases/aarch64/alpine-minirootfs-3.11.6-aarch64.tar.gz
  +
  +
  +
Assuming that the ARM64 chroot is under <tt>/chroot/distro_arm64</tt>, then you can:
  +
  +
$ cp /usr/bin/qemu-aarch64-static /chroot/distro_arm64/usr/bin/qemu-aarch64-static
  +
$ chroot /chroot/distro_arm64
  +
  +
Now you have a full ARM64 environment running on a regular x86 machine. You can automate all the last steps with the following Docker command (the example is running Debian):
  +
  +
$ docker run -it -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/debian /bin/bash
  +
  +
Inside your ARM64 environment you can follow the regular native compilation steps:
  +
  +
$ cd xen.git
  +
# install build dependencies with apt-get/apk/yum etc.
  +
$ ./configure
  +
$ make -j4
  +
  +
=== Native Building ===
  +
  +
In order to build the tools a native build environment is required. For 32-bit the developers mainly use the ''armhf'' port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.
  +
  +
== Use of qemu-system-i386 on ARM ==
  +
  +
This surprises many people.
  +
  +
Xen on ARM uses qemu only to provide certain PV backends (primarily qdisk and pvfb). However the Xen PV backend code on the qemu side is a bit entangled with the x86 stuff, due to the historical use with x86 HVM guests.
  +
  +
Work on untangling things is ongoing but in the meantime qemu-system-i386 is what you want on ARM too. There is no CPU emulation in this mode so there is no danger of it trying to execute anything etc.
  +
  +
  +
== Debugging ==
  +
  +
A small set of [[Xen_ARM_DEBUG_hypercalls|DEBUG hypercalls]] are available to help debugging early boot domU issues.
  +
  +
  +
== Open issues, known problems and workarounds ==
  +
  +
=== <tt>error: "PSR_MODE_EL3h" redefined</tt> ===
  +
  +
When build the tools for arm64 you may see:
  +
  +
In file included from
  +
/root/xenbits/xen/tools/xenstore/../../tools/include/xen/xen.h:35:0,
  +
from /root/xenbits/xen/tools/xenstore/../../tools/libxc/include/xenctrl.h:37,
  +
from xenstored_core.h:23,
  +
from xenstored_core.c:49:
  +
/root/xenbits/xen/tools/xenstore/../../tools/include/xen/arch-arm.h:345:0: error: "PSR_MODE_EL3h" redefined [-Werror]
  +
In file included from /usr/include/sys/user.h:25:0,
  +
from /usr/include/sys/procfs.h:34,
  +
from /usr/include/sys/ucontext.h:26,
  +
from /usr/include/signal.h:360,
  +
from xenstored_core.c:40:
  +
/usr/include/asm/ptrace.h:36:0: note: this is the location of the previous definition
   
  +
and/or other similar <code>PSR_MODE_*</code> errors.
A working configuration is available [http://xenbits.xen.org/people/dvrabel/xen-arm/vexpress-dt.config here].
 
   
  +
This is a glibc bug, see [https://bugs.launchpad.net/linaro-aarch64/+bug/1169164 Linaro launchpad bug #1169164].
=== Device Tree Blob (DTB) ===
 
   
  +
This has been fixed in newer glibc, so first check if your distro has a fixed version available.
Build the device tree blobs (flattened device tree binary files) with <tt>make dtbs</tt>. A suitable DTB file for the envelope model is vexpress-v2p-aem-v7a.dtb.
 
   
  +
If not then you can either apply the patch from the bug directly to the headers under <code>/usr/include</code> (not really recommended, since future upgrades may overwrite them) or edit <code>xen/include/public/arch-arm.h</code> and insert the following before <code>#define PSR_MODE_BIT 0x10</code>:
=== Flash Image ===
 
   
  +
#undef PSR_MODE_BIT
The dom0 kernel is loaded by Xen from the beginning of the flash. The kernel zImage and the DTB should be appended. e.g., <tt>cat zImage vexpress-v2p-aem-v7a.dtb &gt; flash.bin</tt>.
 
  +
#undef PSR_MODE_EL3h
  +
#undef PSR_MODE_EL3t
  +
#undef PSR_MODE_EL2h
  +
#undef PSR_MODE_EL2t
  +
#undef PSR_MODE_EL1h
  +
#undef PSR_MODE_EL1t
  +
#undef PSR_MODE_EL0t
   
=== Native boot ===
+
=== [[Xen_ARM_TODO|TODO]] ===
   
  +
See [[Xen_ARM_TODO|this page]] for a full list of bugs and projects.
To boot the kernel natively use an updated boot-wrapper from [http://xenbits.xen.org/gitweb/?p=people/dvrabel/boot-wrapper.git;a=summary git://xenbits.xen.org/people/dvrabel/boot-wrapper.git].
 
   
== Developer FAQs ==
+
== Also See ==
  +
* [[Automotive_Whitepapers]]
Use the [[Xen ARMv7 with Virtualization Extensions/DevFAQ|Xen ARMv7 Dev FAQ]] page to cover commonly asked questions.
 
   
 
[[Category:XenARM]]
 
[[Category:XenARM]]
  +
[[Category:Developers]]
  +
[[Category:OpenEmbedded]]
  +
[[Category:Xen 4.3]]
  +
[[Category:Xen 4.4]]
  +
[[Category:Xen 4.5]]
  +
[[Category:Xen 4.6]]
  +
[[Category:Xen 4.7]]

Revision as of 17:42, 13 October 2020

The ARM v7-A and ARM v8-A architectures include optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions are currently available in some ARM v7 processors such as the Cortex A15 and Cortex A7.

Introduction

If you are interested in the Xen on ARM architecture and how it compares to Xen on x86, read the Xen on ARM whitepaper.

Status

Both the 32-bit (arm32) and the 64-bit (arm64) ports of Xen boot dom0 and unprivileged guests can be created and destroyed using xl. See below for information on the hardware and models. You may also want to read our whitepaper that explains the basic architecture and terminology of Xen on ARM.

Guest ABI

The guest (including dom0) visible ABI exposed by the hypervisor has been declared stable as of the 4.4 release and will now be maintained in a backwards compatible manner.

Contributing

Please email xen-devel with comments, questions and patches. Please see the list info page for subscription information and the archives. For patches please see Submitting Xen Patches.

Testing

There are a lot of different ARM boards and servers with many different IP blocks available. The project's CI loop can only test a small subset of these, as we cannot afford to buy and test against all possible boards. As such, we will be relying on manual testing during Xen Project Test Days to verify which ARM hardware works and which doesn't. The good news, is that from experience the vast majority of board specific issues will occur during boot and thus a a simple Xen ARM Manual Smoke Test is usually sufficient to identify such issues.

The Xen ARM Manual Smoke Test/Results contains a list of people who have specific boards and also lists when these boards have been last tested. Feel free to add any test results to the table. By doing so you will help others and yourself.

Hardware

Core/SoC/Board Xen Guide Notes
ARM Cortex A7/A15
ARM Cortex A7/A15 Real-time System Model (FVP) Fastmodels Commercial emulator
Versatile Express Versatile Express With TC2 daughterboard
Calxeda EXC-2000 Midway
Allwinner sunxi
 sun7i/A20 Allwinner linux-sunxi community. e.g. Cubietruck
 sun6i/A31 linux-sunxi community
Exynos5xxx
 Exynos5250 Arndale www.arndaleboard.org
 Exynos5410 OdroidXU www.hardkernel.com
OMAP5
 OMAP5432 uEVM www.ti.com
Renesas R-Car H2/H3
Renesas R-Car H2 Lager
Renesas R-Car H2 Stout
Renesas R-Car H3 Salvator-X
ARM Cortex A53/A57
QEMU AArch64 Emulator QEMU Fast Open Source emulator
ARMv8 Foundation Model Fastmodels Free as in beer emulator
ARM AEMv8 Real-time System Model Fastmodels Commercial emulator
ARM Cortex A53/A57 Real-time System Model Fastmodels Commercial emulator
Applied Micro X-Gene X-Gene Mustang (XC-1), HP Moonshot (McDivitt)
HiKey board from 96boards.org HiKey Open Hardware Platform
HiKey960 board from 96boards.org HiKey960 Open Hardware Platform
Xilinx Zynq UltraScale+ MPSoC (ZynqMP) Xilinx Wiki Open Source Emulator QEMU for ZynqMP
ESPRESSObin based on Marvell ARMADA 3700 ESPRESSObin Hardware Platform
Allwinner A64/H5 based boards Allwinner boards like: Pine64, OrangePi PC 2
ARM Cortex A53/A72
Ibox3399 board based on Rockchip RK3399 Ibox3399 boards like: RockPro64


Early work was also done to support the Google Chromebook, however this is no longer continued since it is not a good development platform.

Requirements

ARM Hardware or Software Model
See above for details of the hardware and models which are supported.
Firmware
Xen requires certain functionality from the system firmware. See below for details.
Device Tree
A device tree in the flat device tree format (.dtb). The host platform must be described in a DTB binary passed to Xen at boot time. This will be used by Xen and Dom0. Normally the regular device tree used when booting natively on the platform should be used.
Xen
All current work is now merged into the current development branch git://xenbits.xen.org/xen.git. It is recommended to use the latest Xen master branch.
Linux kernel for dom0
The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible.
dom0 userspace
The developers are using the armhf port of Debian Wheezy.
domU kernel
The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.

Hypervisor ABI Compatibility

The ABI for Xen on ARM was declared stable from Xen 4.4 onwards.

This ABI is implemented by Linux mainline v3.9-rc1 onwards.

Although not a hypervisor ABI change Linux versions prior to v3.13-rc5 (specifically 380108d891ac "xen/block: Correctly define structures in public headers on ARM32 and ARM64") incorrectly defined the PV block protocol on ARM. This means that versions of Linux prior to this fix are only able to interoperate (as either front or backend) with other versions of Linux prior to this commit. After this fix Linux correctly uses the same protocol as other ports (e.g. BSD).

Firmware Requirements

The primary requirement is that the hypervisor must be launched in Non-Secure Hypervisor mode only. If the stock firmware on a platform does not obey this (most commonly by launching in Secure Supervisor mode) then a firmware update may be required. This support is present in u-boot 2014.01.

Booting secondary processors on an SMP system requires firmware support for the Power State Coordination Interfaces (PSCI). Initial u-boot support for this interface is available in https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/log/?h=wip/psci .

When running Xen under a FastModel (which typically have no firmware at all) it is sometimes necessary to use a boot-wrapper. See the Fast Model page for more information.

Xen previously included a workaround for firmware which lacked this functionality, however it was unmaintainable and interfered with proper support for other platforms and therefore as of September 2013 it has been removed.

Booting Natively

Before starting to load Xen it is highly recommended to get the kernel you intend to use as dom0 booting natively (i.e. without Xen underneath). This will let you iron out any driver issues and figure out the necessary kernel command line etc before adding Xen into the mix.

Booting Xen

ImageBuilder

Many of the details necessary to boot Xen from U-Boot can be generated automatically by ImageBuilder's uboot-script-gen, see ImageBuilder.

Boot Protocol

Xen's boot requirements are spelled out in docs/misc/arm/booting.txt in the Xen tree, which references the Linux arm and arm64 booting documentation.

Device Trees

Xen needs the device trees to be in the flat device tree format (the device tree blob or DTB).

It is no longer necessary to build a specific DTB for use with Xen. The Device Tree files shipped with Linux or from the Split Device Tree Repository can be used.

Boot Modules

At boot time Xen must be provided with a dom0 kernel blob and an optional dom0 initramfs blob. The bootloader must load these into memory and describe their location in the Device Tree Blob using the bindings specified in docs/misc/arm/device-tree/booting.txt.

These nodes can either be added by hand (by editing and recompiling the .dts file) or by using u-boot's "fdt" command to add them dynamically at boot time:

   fdt addr ${fdt_addr}
   fdt resize
   
   fdt set /chosen \#address-cells <1>
   fdt set /chosen \#size-cells <1>
   
   fdt mknod /chosen module@0
   fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
   fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
   fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"

(this assumes the device tree blob is loaded at ${fdt_addr} and the kernel at ${kernel_addr_r}).

See e.g. Xen_ARM_with_Virtualization_Extensions/Allwinner for a more concrete example of this.

Command Lines

docs/misc/arm/device-tree/booting.txt describes where Xen looks for both its own command line and the command line to pass to domain 0.

Getting Xen output

To get output log on the UART, Xen needs to know which UART to use. This should be passed in the hypervisor command line using the "dtuart" parameter. e.g.:

console=dtuart dtuart=myserial

where myserial is either an alias to the UART in the device tree (aliases are found in the aliases device tree node) or a full DTB path to the device. As Xen already uses it the UART will be disabled from the point of view of domain 0.

For instance, this is a dummy device tree (won't work) to use the uart0 in Xen:

/ {
    choosen {
       bootargs = "console=dtuart dtuart=myserial";
    }
    aliases {
        myserial = &myserial_0;
    }
    myserial_0: uart0 {
       ... configuration of your UART ...
    }
  }

Here dtuart is configured using the myserial alias. Alternatively /uart0 (the full path to the device) could have been used.

Note: If you don't see output from Xen, you can enable early printk. This option will turn on platform specific UART and output information before the console is initialized.

Dom0 kernel

In general the same kernel configuration as used to boot natively, plus turning on the Xen specific options should work. A good starting point is often the "multi_v7_defconfig" + Xen options.

If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen. It is strongly recommended not to append a DTB to your dom0 kernel (or to disable APPENDED_DTB).

DomU kernel and DTS

Unprivileged guests can be created using xl. A simple VM config file would look like this:

kernel = "/root/image"
memory = 128
name = "guest"
vcpus = 1
disk = [ 'phy:/dev/loop0,xvda,w' ]
extra = "earlyprintk=xenboot console=hvc0 root=/dev/xvda debug rw init=/bin/sh"

where "/root/image" is a Linux zImage.

Common DomU Pitfalls

Enabling CONFIG_DEBUG_LL in the guest kernel configuration.
Although this option can work for dom0 if configured appropriately for the host it does not work for domU (which cannot see the host UART). The symptoms of this are that the guest console will be silent because the kernel has taken a fault accessing the early UART. This can be confirmed by using the xenctx tool (found in $PREFIX/lib/xen/bin/). The tool takes a numeric domid (not a name, use xl list or xl domid $name) and dumps the VCPU state. A PC of 0x0000000c will usually indicate that an early trap has occurred.

Building Xen on ARM

Icon Ambox.png At least gcc version 4.7.3 is known to miscompile certain parts of Xen, most often leading to a segmentation fault in the xl toolstack while starting a guest. We recommend using 4.8.x or later


There are two major components which need to be built for a Xen system. The Xen hypervisor binary itself and the Xen toolstack.

Cross Compiling Xen

Cross compiling the Xen hypervisor is simple.

Linaro supply cross compilers for both arm32 (arm-linux-gnueabihf-) and arm64 (aarch64-linux-gnu-) via linaro-toolchain-binaries. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from kernel.org.

Once you have a suitable cross compiler you can compile Xen with:

   $ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf-

or:

   $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

This assumes that the command prefix for you cross compiler is arm-unknown-linux-gnueabihf- or aarch64-linux-gnu- and that the appropriate arm-unknown-linux-gnueabihf-gcc or aarch64-linux-gnu-gcc and friends are in your $PATH.


Building the Toolstack

For a complete cross-compilation of the hypervisor and the toolstack, it is recommended to use Yocto, see Xen_on_ARM_and_Yocto.


Alternatively, it is possible to use qemu user to run an ARM64 chroot on a x86 host, i.e. an ARM64 Debian or Ubuntu container on a regular x86 laptop.

   $ apt-get install qemu-user-static

It installs /usr/bin/qemu-aarch64-static.

Next setup an ARM64 chroot environment on your x86 machine. Follow your distro recommandations. For instance, the following distros offer pre-packaged tarballs ready to be unpackged:


Assuming that the ARM64 chroot is under /chroot/distro_arm64, then you can:

   $ cp /usr/bin/qemu-aarch64-static /chroot/distro_arm64/usr/bin/qemu-aarch64-static
   $ chroot /chroot/distro_arm64

Now you have a full ARM64 environment running on a regular x86 machine. You can automate all the last steps with the following Docker command (the example is running Debian):

   $ docker run -it -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/debian /bin/bash

Inside your ARM64 environment you can follow the regular native compilation steps:

   $ cd xen.git
     # install build dependencies with apt-get/apk/yum etc.
   $ ./configure
   $ make -j4

Native Building

In order to build the tools a native build environment is required. For 32-bit the developers mainly use the armhf port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.

Use of qemu-system-i386 on ARM

This surprises many people.

Xen on ARM uses qemu only to provide certain PV backends (primarily qdisk and pvfb). However the Xen PV backend code on the qemu side is a bit entangled with the x86 stuff, due to the historical use with x86 HVM guests.

Work on untangling things is ongoing but in the meantime qemu-system-i386 is what you want on ARM too. There is no CPU emulation in this mode so there is no danger of it trying to execute anything etc.


Debugging

A small set of DEBUG hypercalls are available to help debugging early boot domU issues.


Open issues, known problems and workarounds

error: "PSR_MODE_EL3h" redefined

When build the tools for arm64 you may see:

   In file included from
   /root/xenbits/xen/tools/xenstore/../../tools/include/xen/xen.h:35:0,
                    from /root/xenbits/xen/tools/xenstore/../../tools/libxc/include/xenctrl.h:37,
                    from xenstored_core.h:23,
                    from xenstored_core.c:49:
   /root/xenbits/xen/tools/xenstore/../../tools/include/xen/arch-arm.h:345:0: error: "PSR_MODE_EL3h" redefined [-Werror]
   In file included from /usr/include/sys/user.h:25:0,
                    from /usr/include/sys/procfs.h:34,
                    from /usr/include/sys/ucontext.h:26,
                    from /usr/include/signal.h:360,
                    from xenstored_core.c:40:
   /usr/include/asm/ptrace.h:36:0: note: this is the location of the previous definition

and/or other similar PSR_MODE_* errors.

This is a glibc bug, see Linaro launchpad bug #1169164.

This has been fixed in newer glibc, so first check if your distro has a fixed version available.

If not then you can either apply the patch from the bug directly to the headers under /usr/include (not really recommended, since future upgrades may overwrite them) or edit xen/include/public/arch-arm.h and insert the following before #define PSR_MODE_BIT 0x10:

#undef PSR_MODE_BIT
#undef PSR_MODE_EL3h
#undef PSR_MODE_EL3t
#undef PSR_MODE_EL2h
#undef PSR_MODE_EL2t
#undef PSR_MODE_EL1h
#undef PSR_MODE_EL1t
#undef PSR_MODE_EL0t

TODO

See this page for a full list of bugs and projects.

Also See