Xen ARM with Virtualization Extensions

From Xen
Revision as of 13:50, 26 March 2014 by Ijc (talk | contribs)
Jump to navigationJump to search
For the ARM port supporting paravirtualized guests on processors without the virtualization extensions see Xen ARM (PV).

The ARM v7-A and ARM v8-A architectures include optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions are currently available in some ARM v7 processors such as the Cortex A15 and Cortex A7.

Status

Both the 32-bit (arm32) and the 64-bit (arm64) ports of Xen boot dom0 and unprivileged guests can be created and destroyed using xl. See below for information on the hardware and models.

Guest ABI

The guest (including dom0) visible ABI exposed by the hypervisor is not yet set in stone and there may be changes as work progresses. See requirements (below) for specific details of matching kernel and hypervisor versions.

Contributing

Please email xen-devel with comments, questions and patches. Please see the list info page for subscription information and the archives. For patches please see Submitting Xen Patches.

The xen-arm mailing list is focused on Xen ARM (PV) port.

Hardware

The arm32 port of Xen currently runs on:

Work is also underway to support the Google Chromebook.

The arm64 port of Xen currently runs on:

Requirements

ARM Hardware or Software Model
See above for details of the hardware and models which are supported.
Firmware
Xen requires certain functionality from the system firmware. See below for details.
Device Tree
A device tree in the flat device tree format (.dtb). The host platform must be described in a DTB binary passed to Xen at boot time. This will be used by Xen and Dom0. Normally the regular device tree used when booting natively on the platform should be used.
Xen
All current work is now merged into the current development branch git://xenbits.xen.org/xen.git. It is recommended to use the latest Xen master branch.
Linux kernel for dom0
The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible.
dom0 userspace
The developers are using the armhf port of Debian Wheezy.
domU kernel
The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.

Hypervisor ABI Compatibility

The ABI for Xen on ARM is currently not set in stone.

The most recent ABI change was xen: event channel arrays are xen_ulong_t and not unsigned long. This change was reflected in Linux commit c81611c4e96f595a80d8be9367c385d2c116428b which is already present in mainline and will be part of v3.9-rc1.

Although not a hypervisor ABI change Linux versions prior to v3.13-rc5 (specifically 380108d891ac "xen/block: Correctly define structures in public headers on ARM32 and ARM64") incorrectly defined the PV block protocol on ARM. This means that versions of Linux prior to this fix are only able to interoperate (as either front or backend) with other versions of Linux prior to this commit. After this fix Linux correctly uses the same protocol as other ports (e.g. BSD).

Firmware Requirements

The primary requirement is that the hypervisor must be launched in Non-Secure Hypervisor mode only. If the stock firmware on a platform does not obey this (most commonly by launching in Secure Supervisor mode) then a firmware update may be required. This support is present in u-boot 2014.01.

Booting secondary processors on an SMP system requires firmware support for the Power State Coordination Interfaces (PSCI). Initial u-boot support for this interface is available in https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/log/?h=wip/psci .

When running Xen under a FastModel (which typically have no firmware at all) it is sometimes necessary to use a boot-wrapper. See the Fast Model page for more information.

Xen previously included a workaround for firmware which lacked this functionality, however it was unmaintainable and interfered with proper support for other platforms and therefore as of September 2013 it has been removed.

Booting Natively

Before starting to load Xen it is highly recommended to get the kernel you intend to use as dom0 booting natively (i.e. without Xen underneath). This will let you iron out any driver issues and figure out the necessary kernel command line etc before adding Xen into the mix.

Booting Xen

Boot Protocol

Xen's boot requirements are spelled out in docs/misc/arm/booting.txt in the Xen tree, which references the Linux arm and arm64 booting documentation.

Device Trees

Xen needs the device trees to be in the flat device tree format (the device tree blob or DTB).

It is no longer necessary to build a specific DTB for use with Xen. The Device Tree files shipped with Linux or from the Split Device Tree Repository can be used.

Boot Modules

At boot time Xen must be provided with a dom0 kernel blob and an optional dom0 initramfs blob. The bootloader must load these into memory and describe their location in the Device Tree Blob using the bindings specified in docs/misc/arm/device-tree/booting.txt.

These nodes can either be added by hand (by editing and recompiling the .dts file) or by using u-boot's "fdt" command to add them dyanmically at boot time: 

   fdt addr ${fdt_addr}
   fdt resize
   
   fdt set /chosen \#address-cells <1>
   fdt set /chosen \#size-cells <1>
   
   fdt mknod /chosen module@0
   fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
   fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
   fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"

(this assumes the device tree blob is loaded at ${fdt_addr} and the kernel at ${kernel_addr_r}).

See e.g. Xen_ARM_with_Virtualization_Extensions/Allwinner for a more concrete example of this.

Command Lines

docs/misc/arm/device-tree/booting.txt describes where Xen looks for both its own command line and the command line to pass to domain 0.

Getting Xen output

To get output log on the UART, Xen needs to know which UART to use. This should be passed in the hypervisor command line using the "dtuart" parameter. e.g.: 

console=dtuart dtuart=myserial

where myserial is either an alias to the UART in the device tree (aliases are found in the aliases device tree node) or a full DTB path to the device. As Xen already uses it the UART will be disabled from the point of view of domain 0.

For instance, this is a dummy device tree (won't work) to use the uart0 in Xen: 

/ {
    choosen {
       bootargs = "console=dtuart dtuart=myserial";
    }
    aliases {
        myserial = &myserial_0;
    }
    myserial_0: uart0 {
       ... configuration of your UART ...
    }
  }

Here dtuart is configured using the myserial alias. Alternatively /uart0 (the full path to the device) could have been used.

Note: If you don't see output from Xen, you can enable early printk. This option will turn on platform specific UART and output information before the console is initialized.

Dom0 kernel

In general the same kernel configuration as used to boot natively, plus turning on the Xen specific options should work. A good starting point is often the "multi_v7_defconfig" + Xen options.

If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen. It is strongly recommended not to use APPENDED_DTB with your dom0 kernel.

DomU kernel and DTS

Unprivileged guests can be created using xl. A simple VM config file would look like this: 

kernel = "/root/image"
memory = 128
name = "guest"
vcpus = 1
disk = [ 'phy:/dev/loop0,xvda,w' ]
extra = "earlyprintk=xenboot console=hvc0 root=/dev/xvda debug rw init=/bin/sh"

where "/root/image" is a Linux zImage.

It is no longer recommended (or advised) to use an appended DTB. As of commit 0c64527e7fc9 "libxl: build a device tree for ARM guests" the toolstack will automatically generate a DTB for the guest based on the configuration file above. This change entered staging on 19 November and hit the master branch on 22 November 2013.

If you are using an older version of Xen then you should append a suitable dtb (e.g. xenvm-4.2.dtb) to your kernel with: 

   cat arch/arm/boot/zImage arch/arm/boot/dts/xenvm-4.2.dtb > /mnt/root/image

Building Xen on ARM

Cross Compiling

Cross compiling the hypervisor is relatively simple.

Linaro supply cross compilers for both arm32 (arm-linux-gnueabihf-) and arm64 (aarch64-linux-gnu-) via linaro-toolchain-binaries. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from kernel.org.

Once you have a suitable cross compiler you can compile Xen with: 

   $ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf-

or: 

   $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

This assumes that the command prefix for you cross compiler is arm-unknown-linux-gnueabihf- or aarch64-linux-gnu- and that the appropriate arm-unknown-linux-gnueabihf-gcc or aarch64-linux-gnu-gcc and friends are in your $PATH.

Cross compilation of the tools is described in Xen ARM with Virtualization Extensions/CrossCompiling.

Native Building

In order to build the tools a native build environment is required. For 32-bit the developers mainly use the armhf port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.

It may also be possible to build using a cross-architecture chroot running on an X86 system as described in this blog post, although this hasn't been tried yet.

Platform specific configuration

Open issues, known problems and workarounds

See this page for a full list of bugs and projects.

Developer FAQs

Use the Xen ARM Dev FAQ page to cover commonly asked questions.

Retrieved from "https://wiki.xenproject.org/index.php?title=Xen_ARM_with_Virtualization_Extensions&oldid=10894"