Xen ARM with Virtualization Extensions whitepaper
Xen on ARM
What is Xen?
Xen is a lightweight high performance Open Source hypervisor with a very small footprint. Xen on ARM amounts to less than 90K lines of code. Xen is licensed GPLv2 and has an healthy and diverse community to support it and fund the development. Xen is hosted by the LinuxFoundation, that provides stewardship for the project.
The Xen Architecture
Xen is type-1 hypervisor: it runs directly on the hardware, everything else in the system is running as a virtual machine on top of Xen, including Dom0 that is the first virtual machine created by Xen. Dom0 is privileged and drives the devices on the platform. Xen virtualizes CPU, memory, interrupts and timers, providing virtual machines with one or more virtual CPUs, a fraction of the memory of the system, a virtual interrupt controller (indistinguishable from a physical interrupt controller from the guest point of view) and a virtual timer. Xen assigns devices such as SATA controllers and network cards to Dom0, taking care of remapping MMIO regions and IRQs. Dom0 (typically Linux, but it could also be FreeBSD or other operating systems) runs the same device drivers for these devices that would be using on a native execution. Dom0 also runs a set of drivers called “paravirtualized backends” to give access to disk, network, etc, to the other unprivileged virtual machines. The operating system running as DomU (unprivileged guest in Xen terminology) gets access to a set of generic virtual devices by running the corresponding “paravirtualized frontend” drivers. A single backend services multiple frontends. A pair of paravirtualized drivers exist for all the most common classes of devices: disk, network, console, framebuffer, mouse, keyboard, etc. They usually live in the operating system kernel, i.e. Linux. A few PV backends can also run in userspace in QEMU. The frontends connect to the backends using a simple ring protocol over a shared page in memory. Xen provides all the tools for discovery and to setup the initial communication. Xen also provides a mechanism for the frontend and the backend to share additional pages and notify each other via software interrupts.
However there is no reasons to run all the device drivers and all the paravirtualized backends in Dom0. The Xen architecture allows “driver domains”: unprivileged virtual machines with the only purpose of running the driver and the paravirtualized backend for one device. For example you can have a disk driver domain, with the SATA controller assigned, running the driver for it and the disk paravirtualized backend. You can have a network driver domain with the network card assigned, running the driver for it and the network paravirtualized backend. As driver domains are regular unprivileged guests, they make the system more secure because they allow large pieces of code, such as the entire network stack, to run in unprivileged mode. Even if a malicious guest manages to take over the paravirtualized network backend and the network driver domain, it would not be able to take over the entire system. Driver domains also improve isolation and resilience: the network driver domain is fully isolated from the disk driver domains and Dom0. If the network driver crashes it would not be able to take down the entire system, only the network. It is possible to reboot just the network driver domains while everything else stays online. Finally driver domains allow Xen users to disaggregate and componentize the system in ways that would not be possible otherwise. For example they allow users to run a real-time operating system alongside the main OS to drive a device that has real time constraints. They allow users to run a legacy OS to drive old devices that do not have any new drivers in modern operating systems. They allow users to separate and isolate critical functionalities from less critical ones. For example they allow to run an OS such as QNX to drive most devices on the platform alongside Android for the user interface.
Xen on ARM: a cleaner architecture
Xen on ARM is not just a straight 1:1 port of x86 Xen. We exploited the opportunity offered by the new port to clean up the architecture and get rid of the cruft that we accumulated during the many years of x86 development. Firstly we removed any need for emulation. Emulated interfaces are slow and insecure. QEMU, used for emulation on x86 Xen, is a well maintained Open Source project but is big both in terms of binary size and lines of source code. In software the smaller, the simpler, the better. Xen on ARM does not need QEMU because it does not do any emulation. It accomplishes the goal by exploiting virtualization support in hardware as much as possible and using paravirtualized interfaces for IO. As a result Xen on ARM is faster and more secure.
On x86 two different kind of Xen guest coexist: PV guests, such as Linux and other Open Source OSes, and HVM guests, usually Microsoft Windows, but any OS can run as HVM guest. PV and HVM guests are quite different from the hypervisor point of view. The difference is exposed all the way up to the user, that needs to choose how to run the guest by setting a line in the VM config file. On ARM we do not want to introduce this differentiation that we feel is artificial and can confuse the user. Xen on ARM only supports one kind of guests that is the best of both worlds: it does not need any emulation and relies on paravirtualized interfaces for IO as early as possible in the boot sequence, like x86 PV guests. It exploits virtualization support in hardware as much as possible and does not require invasive changes to the guest operating system kernel in order to run, like x86 HVM guests.
The new architecture designed for Xen on ARM is much cleaner and simpler and it turned out to be a very good match for the hardware.
Xen on ARM: virtualization extensions
ARM provides 3 levels of execution: EL0, user mode, EL1, kernel mode, and EL2, hypervisor mode. ARM virtualization extensions introduce a new instruction, HVC, to switch between kernel mode and hypervisor mode. The MMU supports 2 stages of translation. The generic timers and the GIC interrupt controller are virtualization aware.
Xen runs entirely and only in hypervisor mode. It leaves kernel mode for the guest operating system kernel and EL0 for guest user space applications. Type-2 hypervisors need to frequently switch between hypervisor mode and kernel mode. By running entirely in EL2 Xen significantly reduces the number of context switches required. The new instruction, HVC, is used by the kernel to issue hypercalls to the hypervisor. Xen uses 2-stage translation in the MMU to assign memory to virtual machines. Xen also takes over the generic timers and the GIC interrupt controller. It uses the generic timers to receive timer interrupts as well as injecting timer interrupts and exposing the counter to virtual machines. It uses the GIC to receive interrupts as well as injecting interrupts into guests.
Xen discovers the hardware via device tree. It assigns all the devices that it does not use to Dom0 by remapping the corresponding MMIO regions and interrupts. It generates a flatten device tree binary for Dom0 that describes exactly the environment exposed to it: the number of virtual cpus that Xen created for it (maybe less than the number of physical cpus on the platform), the amount of memory that Xen gave to it (surely less than the amount of physical memory available) and only the devices that Xen re-assigned to it (not all devices are assigned to Dom0, at the very least one UART is not). Xen also adds a device tree node to advertise its own presence on the platform. Dom0 boots exactly the same way it would boot natively. By using device tree to discover the hardware, Dom0 finds out exactly what is available and loads the drivers for it. It does not try to access interfaces that are not present and therefore Xen does not need to do any emulation. By finding the Xen hypervisor node, Dom0 knows that it is running on Xen and therefore can initialize the paravirtualized backends. Other DomUs would load the paravirtualized frontends instead.
Xen on ARM: code size
We wrote previously that the new architecture turned out to be a very good match for the hardware. This is proven by the code size: the smaller the better. Xen on ARM is 1/6 of the code size of x86_64 Xen, while still providing a similar level of features. In Xen 4.4.0:
| Common | ARMv7 | ARMv8 | Total | |
| xen/arch/arm | 11,767 | 3,503 | 1,812 | 17,082 |
| C | 11,587 | 954 | 813 | 13,354 |
| ASM | 180 | 2,549 | 999 | 3,728 |
| xen/include/asm-arm | 4,786 | 984 | 1,050 | 6,820 |
| Total ARM | 16,553 | 4,487 | 2,862 | 23,902 |
| x86_64 | Total | |||
| xen/arch/x86 | 124,615 | |||
| xen/include/asm-x86 | 18,530 | |||
| Total x86_64 | 143,145 | |||
Porting Xen to a new SoC
Assuming that you already have a functional Dom0 kernel (usually Linux) for your SoC, porting Xen to it is a very simple task. In fact in terms of devices, Xen only uses:
- GIC
- generic timers
- SMMU
- one UART for debugging
Therefore the porting effort is limited to writing a new UART driver for Xen (if the SoC comes with an unsupported UART) and the code to bring up secondary CPUs (if the platform does not support PSCI, for which Xen has already a driver).
Porting an operating system to Xen on ARM
Porting an OS to Xen on ARM could not be easier: it does not require any changes to the operating system kernel, only few new drivers to get the paravirtualized frontends running and obtain access to network, disk, console, etc. The paravirtualized frontends rely on:
- grant table for page sharing
- xenbus for discovery
- event channels for notifications.
Once the OS has support for these basic building blocks, introducing the paravirtualized frontends is easy. You are likely to be able to reuse the existing frontends in Linux (GPLv2 licensed):
Or the ones in FreeBSD or NetBSD (BSD licensed).
