Xen ARM with Virtualization Extensions/Allwinner
Hardware
This wiki talks about Xen on Allwinner A20(Cortex-A7 dual core, code name sun7i).
The A31(Cortex-A7 quad core, code name sun6i) should work as well, but has not been as widely tested.
The following boards have been tested:
- A20
- Cubieboard2 (note that the original Cubieboard does not support virtualisation)
- Cubietruck (AKA Cubieboard 3)
Bootloader
As described in Firmware Requirements, Xen need boot from non-secure hyper mode.
Source Code
sunxi support is not currently merged into mainline u-boot. Support for this platform is available in the u-boot-sunxi.git tree
Hans de Goede maintains a branch which adds the necessary support for PSCI to enable SMP support and booting from non-secure hypervisor mode. See https://github.com/jwrdegoede/u-boot-sunxi, branch `sunxi-next`.
Compile
See [1] for more details on building U-boot for sunxi.
For sd card:
make Cubieboard2 CROSS_COMPILE=arm-linux-gnueabihf- -j8
For USB/FEL boot:
make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- -j8
Linux
Upstream Community
Support for the Allwinner processors in Linux is mainly a community effort, see the Linux Sunxi Wiki.
Source Code
Upstream Linux support for Allwinner processors is a work in progress. See Sunxi Linux mainlining effort for current status.
As of 3.13-rc6 there is a lot of useful support in the mainline kernel, however the following are not present:
- SATA AHCI support
- MMC support
- USB support
Support for these can be found in the sunxi linux tree at https://github.com/linux-sunxi/linux-sunxi. The `sunxi-next` contain stuff destined for the next Linux merge window, while `sunxi-devel` contains things targeting future versions. Both branches are rebasing.
Building
Start from the multi_v7_defconfig multiplatform kernel:
make sun7i_dom0_defconfig
From here you will want to enable the various Xen options (CONFIG_XEN, frontend and backend drivers etc, see Mainline_Linux_Kernel_Configs for more general details).
Then:
make zImage make dtbs
The final kernel binary will be in `arch/arm/boot/zImage` and the device tree will be located in in `arch/arm/boot/dts` e.g. sun7i-a20-cubieboard2.dtb, sun7i-a20-cubietruck.dtb etc.
The same kernel image should work for both dom0 and domU.
Building Xen
The latest xen should work. See Build Xen on ARM and Building Xen and Linux Dom0 for building xen and generating the xen-uImage.
Use the following build command for enable early printk for a20:
make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=sun7i
Booting Xen and Dom0
Load addresses
Due to Xen ARM TODO/Domain 0 memory limitation due to 1:1 mapping it is recommended to place the initial binary blobs towards the top of RAM, leaving 2MB free for Xen to relocate into.
The following table shows suggested load addresses for 1GB (e.g cubieboard2) and 2GB (e.g. cubietruck) systems. These can be used with the boot.scr boot script described in the next section.
| Binary | 1GB systems | 2GB systems | Size | boot.scr variable name |
|---|---|---|---|---|
| Top of RAM | 0x80000000 | 0xc0000000 | ||
| Xen relocation target address | 0x7fe00000 | 0xbfe00000 | 2MB | |
| Linux kernel (vmlinuz) | 0x7fa00000 | 0xbfa00000 | 4MB | kernel_addr_r |
| Device tree blob | 0x7f800000 | 0xbf800000 | 2MB | fdt_addr |
| Xen initial load address | 0x7f600000 | 0xbf600000 | 2MB | xen_addr_r |
This assumes that the kernel is <4MB, if it is larger then shift everything down in 2MB increments. If an initrd is required then it can be placed between the kernel and the device tree blob, with the dtb and Xen's initial address being shuffled down as appropriate.
Boot script
To boot Xen and Dom0, you can use USB boot or sd boot. Some runtime manipulation of the DTB is needed, but the basic outline is the same in each case (only the method to load the binary differs).
# SUNXI Xen Boot Script
# Addresses suitable for 1GB system, adjust as appropriate for a 2GB system.
# Top of RAM: 0x80000000
# Xen relocate addr 0x7fe00000
setenv kernel_addr_r 0x7fa00000 # 4M
setenv fdt_addr 0x7f800000 # 2M
setenv xen_addr_r 0x7f600000 # 2M
setenv fdt_high 0xffffffff # Load fdt in place instead of relocating
# Load xen/xen to ${xen_addr_r}
setenv bootargs "console=dtuart dtuart=/soc@01c00000/serial@01c28000 dom0_mem=128M"
# Load appropriate .dtb file to ${fdt_addr}
fdt addr ${fdt_addr} 0x40000
fdt resize
fdt chosen
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
# Load Linux arch/arm/boot/zImage to ${kernel_addr_r}
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "console=hvc0 ro root=/dev/sda1 clk_ignore_unused"
bootz ${xen_addr_r} - ${fdt_addr}
The easiest approach is to create a file (boot.xen) with this and then convert this into a u-boot boot.scr with:
mkimage -A arm -T script -d boot.xen boot.scr.
This can then be loaded to address 0x41000000 and then launched with:
source 0x41000000
Booting via PXE
To boot use the boot.scr above loading images with:
tftp ${address} /path/to/image
For example:
# Load xen/xen to ${xen_addr_r}
tftp ${xen_addr_r} /cubieboard/xen
# Load appropriate .dtb file to ${fdt_addr}
tftp ${fdt_addr} /cubieboard/sun7i-a20-cubieboard2.dtb
# Load Linux arch/arm/boot/zImage to ${kernel_addr_r}
tftp ${kernel_addr_r} /cubieboard/vmlinuz
To boot this set bootcmd:
setenv bootcmd setenv autoload no\;dhcp\;tftp 0x41000000 /boot.scr;source 0x41000000 saveenv boot
Booting via USBBoot(FEL)
reference: http://linux-sunxi.org/FEL/USBBoot
Arrange for all of the images to be loaded at the appropriate addresses via the FEL utility and then load a suitable boot.scr to launch them.
Booting directly on the SD card
reference Bootable SD Card for how to make a bootable sdcard.
Copy xen, zImage, the dtb and boot.scr to the root directory of you SD card.
To boot use the boot.scr above loading images with:
fatload mmc 0 ${address} /path/to/image
For example:
# Load xen/xen to ${xen_addr_r}
fatload mmc 0 ${xen_addr_r} /xen
# Load appropriate .dtb file to ${fdt_addr}
fatload mmc 0 ${fdt_addr} /sun7i-a20-cubieboard2.dtb
# Load Linux arch/arm/boot/zImage to ${kernel_addr_r}
fatload mmc 0 ${kernel_addr_r} /vmlinuz
To boot set the bootcmd:
setenv bootcmd "fatload mmc 0 0x41000000 boot.scr; source 0x41000000" savenv boot
This assume the SD card has only 1 partition with fat filesystem. Use e.g. ext2load etc if this is not the case.
Mount NFS after initrd boot
Using a distrbution is easily to compile xen tools.
Both opensuse arm 12.3 rootfs and Debian have been used by developers. Other distribution which support armhf is ok too.
Create domU
- setup the harddisk
losetup /dev/loop0 /root/domU/rootfs.ext3
- domU config example
> cat domU_test kernel = "/root/domU/zImage memory = "128" name = "domU" vcpus = 1 serial="pty" disk = [ 'phy:/dev/loop0,xvda,w' ] vif=[ 'mac=00:16:3e:56:af:69,bridge=virbr0,type=netfront', ]
- create
xl create domU_test
or enable all the debug message
xl -vvv create -d domU_testa
- connect to console
xl console domU_test
DomU kernel debug
Xen supports a variety of debug methods. You can use xl debug-keys for dumping information. Use `xen_raw_console_write` for output DomU kernel output.
`gdbsx` is not work for arm right now.
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index b4e8500..060c163 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1488,6 +1488,7 @@ static size_t cont_print_text(char *text, size_t size)
return textlen;
}
+#include <xen/hvc-console.h>
asmlinkage int vprintk_emit(int facility, int level,
const char *dict, size_t dictlen,
const char *fmt, va_list args)
@@ -1546,6 +1547,7 @@ asmlinkage int vprintk_emit(int facility, int level,
* prefix which might be passed-in as a parameter.
*/
text_len = vscnprintf(text, sizeof(textbuf), fmt, args);
+ xen_raw_console_write(text);
/* mark and strip a trailing newline */
if (text_len && text[text_len-1] == '\n') {
