Difference between revisions of "Xen ARM with Virtualization Extensions"

From Xen
Line 21: Line 21:
 
The arm32 port of Xen currently runs on:
 
The arm32 port of Xen currently runs on:
 
* The Cortex A15 Real-time System Model's provided by ARM. See [[Xen ARM with Virtualization Extensions/FastModels]] for more information on obtaining and building the necessary model.
 
* The Cortex A15 Real-time System Model's provided by ARM. See [[Xen ARM with Virtualization Extensions/FastModels]] for more information on obtaining and building the necessary model.
* A Cortex-A15 Processor running on the Versatile Express. See [[Xen ARM with Virtualization Extensions/Vexpress]]
+
* A Cortex-A15 Processor running on the Versatile Express. See [[Xen ARM with Virtualization Extensions/Vexpress]].
 
* The Arndale board. See [[Xen ARMv7 with Virtualization Extensions/Arndale|Getting Xen on the Arndale]].
 
* The Arndale board. See [[Xen ARMv7 with Virtualization Extensions/Arndale|Getting Xen on the Arndale]].
 
* Calxeda Midway. See ''TBD''.
 
* Calxeda Midway. See ''TBD''.
Line 41: Line 41:
 
:Xen requires certain functionality from the system firmware. See below for details.
 
:Xen requires certain functionality from the system firmware. See below for details.
 
;Device Tree
 
;Device Tree
:A [[device tree]] in the flat device tree format (.dtb). The device tree source for unprivileged Xen ARM guests is available upstream in the Linux tree >= v3.7: [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/arm/boot/dts/xenvm-4.2.dts;h=ec3f9528e180c75e42b633104fb537ced0762691;hb=HEAD xenvm-4.2.dts]. The interface is also documented in the Linux tree: [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/devicetree/bindings/arm/xen.txt;h=0f7b9c2109f855d69b8d950675a1e8e17b836735;hb=HEAD xen.txt]. Regarding Xen and Dom0, the regular DTS for the underlying platform can be used, however it requires the additional ''hypervisor'' node described in the Linux Documentation and the command line arguments under the ''chosen'' node, see for example the Xen DTS for the Versatile Express Cortex A15 machine we use for development [http://xenbits.xen.org/gitweb/?p=people/sstabellini/device-trees.git;a=blob;f=vexpress-v2p-ca15-tc1.dts;h=468642b63cbbf65f6e5ded9a1ff3bab4746069aa;hb=HEAD vexpress-v2p-ca15-tc1.dts].
+
:A [[device tree]] in the flat device tree format (.dtb). The device tree source for unprivileged Xen ARM guests is available upstream in the Linux tree >= v3.7: [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/arm/boot/dts/xenvm-4.2.dts;h=ec3f9528e180c75e42b633104fb537ced0762691;hb=HEAD xenvm-4.2.dts]. The interface is also documented in the Linux tree: [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/devicetree/bindings/arm/xen.txt;h=0f7b9c2109f855d69b8d950675a1e8e17b836735;hb=HEAD xen.txt]. Regarding Xen and Dom0, the regular DTS for the underlying platform can be used.
 
;Xen
 
;Xen
 
:All current work is now merged into the current development branch [http://xenbits.xen.org/gitweb/?p=xen.git;a=summary git://xenbits.xen.org/xen.git]. It is recommended to use the latest Xen master branch.
 
:All current work is now merged into the current development branch [http://xenbits.xen.org/gitweb/?p=xen.git;a=summary git://xenbits.xen.org/xen.git]. It is recommended to use the latest Xen master branch.
 
;Linux kernel for dom0
 
;Linux kernel for dom0
:The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU.
+
:The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible.
 
;dom0 userspace
 
;dom0 userspace
:The developers are using the armhf port of Debian Wheezy (which is currently the ''testing'' release).
+
:The developers are using the armhf port of Debian Wheezy.
 
;domU kernel
 
;domU kernel
 
:The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.
 
:The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.
Line 56: Line 56:
   
 
The most recent ABI change was [http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=6c7d2bad2f3ebba6fe79d9811c2cde237fa4bf16 ''xen: event channel arrays are xen_ulong_t and not unsigned long'']. This change was reflected in Linux
 
The most recent ABI change was [http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=6c7d2bad2f3ebba6fe79d9811c2cde237fa4bf16 ''xen: event channel arrays are xen_ulong_t and not unsigned long'']. This change was reflected in Linux
commit [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=c81611c4e96f595a80d8be9367c385d2c116428b c81611c4e96f595a80d8be9367c385d2c116428b] which is already present in mainline and will be part of v3.9-rc1. The ''xen-arm-3.8.y'' branch of [http://xenbits.xen.org/gitweb/?p=people/ianc/linux.git;a=summary git://xenbits.xen.org/people/ianc/linux.git] contains the current stable release + this change.
+
commit [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=c81611c4e96f595a80d8be9367c385d2c116428b c81611c4e96f595a80d8be9367c385d2c116428b] which is already present in mainline and will be part of v3.9-rc1.
   
 
=== Firmware Requirements ===
 
=== Firmware Requirements ===
Line 117: Line 117:
 
The Linux command line should be set with the <tt>xen,dom0-bootargs</tt> parameter of the <tt>chosen</tt> node of the device tree supplied to Xen. If this parameter is missing then Linux will not get a command line and the wrong console will be used (there's a bug where the kernel doesn't use the default, built-in command line if the <tt>/chosen</tt> node is missing.
 
The Linux command line should be set with the <tt>xen,dom0-bootargs</tt> parameter of the <tt>chosen</tt> node of the device tree supplied to Xen. If this parameter is missing then Linux will not get a command line and the wrong console will be used (there's a bug where the kernel doesn't use the default, built-in command line if the <tt>/chosen</tt> node is missing.
   
  +
The location of the dom0 kernel must be described in the Device Tree, according to the [http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=08ed7751859dbe2d2c32d86d7df371057d7b45a4;hb=HEAD binding]. If the platform uses a new enough u-boot with the ''fdt'' command then this can be done as part of the boot process, using:
=== Flash Image ===
 
   
  +
fdt addr ${fdt_addr}
The dom0 kernel is loaded by Xen from the beginning of the flash device.
 
  +
fdt resize
  +
  +
fdt set /chosen \#address-cells <1>
  +
fdt set /chosen \#size-cells <1>
  +
  +
fdt mknod /chosen module@0
  +
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
  +
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
  +
fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"
   
  +
(this assumes the device tree blob is loaded at ${fdt_addr} and the kernel at ${kernel_addr_r}).
=== Native boot ===
 
   
  +
If the firmware does not support this then it may be necessary to add a hardcoded chosen node to the Device Tree:
To boot the kernel natively use an updated boot-wrapper from [http://xenbits.xen.org/gitweb/?p=people/dvrabel/boot-wrapper.git;a=summary git://xenbits.xen.org/people/dvrabel/boot-wrapper.git].
 
   
  +
chosen {
Build the device tree blobs provided with the kernel with <tt>make dtbs</tt>. A suitable DTB file for the envelope model is vexpress-v2p-aem-v7a.dtb.
 
  +
#address-cells = <1>;
  +
#size-cells = <1>;
  +
  +
module@0 {
  +
compatible = "xen,linux-zimage", "xen,multiboot-module";
  +
reg = <0x80080000 0x800000>;
  +
bootargs = "<DOMAIN 0 COMMAND LINE>";
  +
};
  +
};
   
  +
Specifying the actual kernel address and size of the ''reg'' property.
For a native boot the kernel should have ARM_APPENDED_DTB and ARM_ATAG_DTB_COMPAT enabled. The kernel zImage and the DTB should be appended. e.g., <tt>cat zImage vexpress-v2p-aem-v7a.dtb &gt; image.bin</tt>.
 
   
 
== DomU kernel and DTS ==
 
== DomU kernel and DTS ==
Line 148: Line 166:
 
=== Cross Compiling ===
 
=== Cross Compiling ===
   
  +
Cross compiling the hypervisor is relatively simple.
Cross compiling the hypervisor is relatively simple. For 32-bit you can find a cross compiler in some Linux distros or you can download the arm-unknown-linux-gnueabi compiler from [http://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.6.3/ kernel.org]. For 64-bit Linaro supply a suitable [http://www.linaro.org/engineering/armv8#tab3 cross-compiler].
 
  +
  +
Linaro supply cross compilers for both arm32 (arm-linux-gnueabihf-) and arm64 (aarch64-linux-gnu-) via [https://launchpad.net/linaro-toolchain-binaries linaro-toolchain-binaries]. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from [http://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/ kernel.org].
   
 
Once you have a suitable cross compiler you can compile Xen with:
 
Once you have a suitable cross compiler you can compile Xen with:
   
$ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabi-
+
$ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf-
   
 
or:
 
or:
Line 158: Line 178:
 
$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
 
$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
   
This assumes that the command prefix for you cross compiler is <tt>arm-unknown-linux-gnueabi-</tt> or <tt>arm-unknown-linux-gnueabi-</tt> and that the appropriate <tt>arm-unknown-linux-gnueabi-gcc</tt> or <tt>arm-unknown-linux-gnueabi-gcc</tt> and friends are in your $PATH.
+
This assumes that the command prefix for you cross compiler is <tt>arm-unknown-linux-gnueabihf-</tt> or <tt>aarch64-linux-gnu-</tt> and that the appropriate <tt>arm-unknown-linux-gnueabihf-gcc</tt> or <tt>aarch64-linux-gnu-gcc</tt> and friends are in your $PATH.
   
 
Cross compilation of the tools is described in [[Xen ARM with Virtualization Extensions/CrossCompiling]].
 
Cross compilation of the tools is described in [[Xen ARM with Virtualization Extensions/CrossCompiling]].
Line 164: Line 184:
 
=== Native Building ===
 
=== Native Building ===
   
In order to build the tools a native build environment is required. For 32-bit the developers mainly use the ''armhf'' port of Debian, which is present in Wheezy (the current testing distribution) running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.
+
In order to build the tools a native build environment is required. For 32-bit the developers mainly use the ''armhf'' port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.
   
 
It may also be possible to build using a cross-architecture chroot running on an X86 system as described in [http://www.hellion.org.uk/blog/posts/foreign-chroots-with-schroot-and-qemu/ this blog post], although this hasn't been tried yet.
 
It may also be possible to build using a cross-architecture chroot running on an X86 system as described in [http://www.hellion.org.uk/blog/posts/foreign-chroots-with-schroot-and-qemu/ this blog post], although this hasn't been tried yet.
 
Native build of the 64-bit tools has not been attempted yet.
 
   
 
== Platform specific configuration ==
 
== Platform specific configuration ==

Revision as of 11:03, 30 September 2013

The ARM v7-A and ARM v8-A architectures include optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions are currently available in some ARM v7 processors such as the Cortex A15 and Cortex A7.

Status

Both the 32-bit (arm32) and the 64-bit (arm64) ports of Xen boot dom0 and unprivileged guests can be created and destroyed using xl. See below for information on the hardware and models.

Guest ABI

The guest (including dom0) visible ABI exposed by the hypervisor is not yet set in stone and there may be changes as work progresses. See requirements (below) for specific details of matching kernel and hypervisor versions.

Contributing

Please email xen-devel with comments, questions and patches. Please see the list info page for subscription information and the archives. For patches please see Submitting Xen Patches.

Hardware

The arm32 port of Xen currently runs on:

Work is also underway to support the Google Chromebook.

The arm64 port of Xen currently runs on:

Work is also underway to support the ARM v8 Foundation Model.

Requirements

ARM Hardware or Software Model
See above for details of the hardware and models which are supported.
Firmware
Xen requires certain functionality from the system firmware. See below for details.
Device Tree
A device tree in the flat device tree format (.dtb). The device tree source for unprivileged Xen ARM guests is available upstream in the Linux tree >= v3.7: xenvm-4.2.dts. The interface is also documented in the Linux tree: xen.txt. Regarding Xen and Dom0, the regular DTS for the underlying platform can be used.
Xen
All current work is now merged into the current development branch git://xenbits.xen.org/xen.git. It is recommended to use the latest Xen master branch.
Linux kernel for dom0
The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible.
dom0 userspace
The developers are using the armhf port of Debian Wheezy.
domU kernel
The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.

Hypervisor ABI Compatibility

The ABI for Xen on ARM is currently not set in stone.

The most recent ABI change was xen: event channel arrays are xen_ulong_t and not unsigned long. This change was reflected in Linux commit c81611c4e96f595a80d8be9367c385d2c116428b which is already present in mainline and will be part of v3.9-rc1.

Firmware Requirements

Xen's boot requirements are spelled out in docs/misc/arm/booting.txt in the Xen tree, which references the Linux arm and arm64 booting documentation.

The primary requirement is that the hypervisor must be launched in Non-Secure Hypervisor mode only. If the stock firmware on a platform does not obey this (most commonly by launching in Secure Supervisor mode) then a firmware update may be required. The is a tree available with u-boot patches by Andre Przywara to enable the correct behavior in his u-boot tree. At the time of writing hypmode_v5 was the latest branch, in addition Andre maintains a hypmode-ports branch with specific support for a variety of platforms.

When running Xen under a FastModel (which typically have no firmware at all) it is sometimes necessary to use a boot-wrapper. See the Fast Model page for more information.

Xen previously included a workaround for firmware which lacked this functionality, however it was unmaintainable and interfered with proper support for other platforms and therefore as of September 2013 it has been removed.

Getting Xen output

To get output log on the UART, Xen needs to know which UART to use.

console=dtuart dtuart=myserial

where myserial is either an alias to the UART in the device tree (aliases are found in the aliases device tree node) or a full DTB path to the device. As Xen already uses it the UART will be disabled from the point of view of domain 0.

For instance, this is a dummy device tree (won't work) to use the uart0 in Xen:

/ {
    choosen {
       bootargs = "console=dtuart dtuart=myserial";
    }
    aliases {
        myserial = &myserial_0;
    }
    myserial_0: uart0 {
       ... configuration of your UART ...
    }
  }

Here dtuart is configured using the myserial alias. Alternatively /uart0 (the full path to the device) could have been used.

Note: If you don't see output from Xen, you can enable early printk. This option will turn on platform specific UART and output information before the console is initialized.

Device Trees

Xen needs the device trees to be in the flat device tree format (the device tree blob or DTB).

It is no longer necessary to build a specific DTB for use with Xen. The Device Tree files shipped with Linux or from the Split Device Tree Repository can be used.

Xen

Usually the DTB is provided to Xen via the bootloader (e.g. as the third argument to u-boot's bootz or bootm command) or, when running on the models, via the bootwrapper.

However it can sometimes be convenient to link the DTB directly into the Xen image, overriding any DTB supplied by the bootloader.

The DTB to use is specified by the CONFIG_DTB_FILE config variable which should be set to the absolute path of the DTB. This variable may be set in .config or on the make command line.

The Xen command line may be set with the bootargs parameter of the chosen node of the device tree. Parameters for the domain 0 kernel are passed using the xen,dom0-bootargs parameter.

Dom0 kernel

Enable at least ARCH_VEXPRESS, ARCH_VEXPRESS_DT. Disable SPARSE_IRQ (it doesn't play nicely with device tree support). If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen.

A working configuration is available here.

The Linux command line should be set with the xen,dom0-bootargs parameter of the chosen node of the device tree supplied to Xen. If this parameter is missing then Linux will not get a command line and the wrong console will be used (there's a bug where the kernel doesn't use the default, built-in command line if the /chosen node is missing.

The location of the dom0 kernel must be described in the Device Tree, according to the binding. If the platform uses a new enough u-boot with the fdt command then this can be done as part of the boot process, using:

   fdt addr ${fdt_addr}
   fdt resize
   
   fdt set /chosen \#address-cells <1>
   fdt set /chosen \#size-cells <1>
   
   fdt mknod /chosen module@0
   fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
   fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
   fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"

(this assumes the device tree blob is loaded at ${fdt_addr} and the kernel at ${kernel_addr_r}).

If the firmware does not support this then it may be necessary to add a hardcoded chosen node to the Device Tree:

   chosen {
   		#address-cells = <1>;
   		#size-cells = <1>;
   
   		module@0 {
   			 compatible = "xen,linux-zimage", "xen,multiboot-module";
   			 reg = <0x80080000 0x800000>;
   			 bootargs = "<DOMAIN 0 COMMAND LINE>";
   		};
   };

Specifying the actual kernel address and size of the reg property.

DomU kernel and DTS

Unprivileged guests can be created using xl. A simple VM config file would look like this:

kernel = "/root/image"
memory = 128
name = "win"
vcpus = 1
disk = [ 'phy:/dev/loop0,xvda,w' ]

where image is actually a Linux zImage with an appended xenvm-4.2.dtb:

   cat arch/arm/boot/zImage arch/arm/boot/dts/xenvm-4.2.dtb > /mnt/root/image


Building Xen on ARM

Cross Compiling

Cross compiling the hypervisor is relatively simple.

Linaro supply cross compilers for both arm32 (arm-linux-gnueabihf-) and arm64 (aarch64-linux-gnu-) via linaro-toolchain-binaries. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from kernel.org.

Once you have a suitable cross compiler you can compile Xen with:

   $ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf-

or:

   $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

This assumes that the command prefix for you cross compiler is arm-unknown-linux-gnueabihf- or aarch64-linux-gnu- and that the appropriate arm-unknown-linux-gnueabihf-gcc or aarch64-linux-gnu-gcc and friends are in your $PATH.

Cross compilation of the tools is described in Xen ARM with Virtualization Extensions/CrossCompiling.

Native Building

In order to build the tools a native build environment is required. For 32-bit the developers mainly use the armhf port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.

It may also be possible to build using a cross-architecture chroot running on an X86 system as described in this blog post, although this hasn't been tried yet.

Platform specific configuration

Open issues, known problems and workarounds

See this page for a full list of bugs and projects.

Developer FAQs

Use the Xen ARM Dev FAQ page to cover commonly asked questions.