Xen ARM with Virtualization Extensions/APMXGeneMustang

Xen ARM64 on APM XGENE Mustang

Xen is now supported upstream on the APM X-Gene Mustang platform.

Booting with U-Boot

The current approach requires tftp to download the software images to the board.

Basic IP address settings

setenv ipaddr 10.y.y.y #This is an IP address of your board.
setenv serverip 10.x.x.x #This is an IP address of your tftp server.
setenv ethaddr 00:zz:zz:zz:zz:zz #This is the MAC of your Ethernet.
setenv user_dir <SERVER TFTP DIRECTORY FOR GETTING XEN IMAGES> #See below
setenv xen_addr_r 0x4000080000 #This is the address where Xen kernel will be copied using tftp.
setenv gatewayip 10.x.x.x
setenv netmask 255.x.x.x
ping $serverip #This should work before proceeding further.
setenv kern_addr_r 0x4002000000 #This is the address where DOM0 kernel will be copied using tftp.
setenv fdt_addr_r 0x4003000000 #This is the address where DOM0 dtb will be copied using tftp. 
saveenv

Where:

• <SERVER TFTP DIRECTORY>: e.g. If your images are inside /tftpboot/xen and /tftpboot is the tftp server root then "setenv user_dir xen"

Setting up Xen booting environment

setenv xen_load 'tftp ${xen_addr_r} ${user_dir}/uXen'
setenv xen_dtb 'tftp ${fdt_addr_r} ${user_dir}/apm-mustang-xen.dtb;fdt addr  ${fdt_addr_r};fdt resize;'
setenv xen_boot 'setenv bootargs ${bootargs_xen};bootm ${xen_addr_r} - ${fdt_addr_r}' 
setenv dom0_node 'fdt set /chosen/module@0 compatible xen,linux-zimage xen,multiboot-module'
setenv xen_dom0 'tftp ${kern_addr_r} ${user_dir}/Image; run dom0_node;fdt set /chosen/module@0 reg < 0x40 ${kern_addr_r} 0x${filesize} >'
setenv xen_run 'run xen_dtb xen_dom0 xen_load;fdt print /chosen;run xen_boot'
setenv bootargs_xen 'conswitch=x sync_console dom0_mem=1G dom0_max_vcpus=1 console=dtuart dtuart=/soc/serial@1c020000'
saveenv

Preparing all the Images

Building Xen Kernel

First, pull the latest xen code from:

http://xenbits.xen.org/gitweb/?p=xen.git;a=summary

Then, build the Xen kernel:

# make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

TIP: To get early prints:

# make dist-xen XEN_TARGET_ARCH=arm64 debug=y   CONFIG_EARLY_PRINTK=xgene-storm  CROSS_COMPILE=aarch64-linux-gnu-

For Performance Testing:

# make dist-xen XEN_TARGET_ARCH=arm64 debug=n  CROSS_COMPILE=aarch64-linux-gnu-

To create the image, do:

# mkimage -A arm -C none -T kernel -a 0x0200000 -e 0x00200000 -n Xen -d xen/xen  uXen

Finally copy "uXen" to your tftp directory.

Building DOM0

For Dom0 rootfs we need SATA support to be enabled in the Dom0 kernel. At the time of writing (17/02/2014), SATA, CLOCK and PCIe Linux driver patches for Mustang are not yet merged and are under review. Hence please use Ian Campbell's linux tree which has un-merged SATA CLOCK and PCIe drivers for mustang. Once above mentioned drivers are made it to mainline then Dom0 can be build directly from mainline Linux.

To download and build Dom0 and DomU, do:

# git clone https://github.com/AppliedMicro/ENGLinuxLatest
# git checkout -b apm_linux_v3.15-rc8 origin/apm_linux_v3.15-rc8
# make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- xgene_xen_defconfig
# make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image

Once completed, copy arch/arm64/boot/Image to your tftp directory.

You also need to apply the following patch for the Dom0 DTB:

diff --git a/arch/arm64/boot/dts/apm-mustang-xen.dts b/arch/arm64/boot/dts /apm-mustang-xen.dts
index 848b34a..d6ce7ee 100644
--- a/arch/arm64/boot/dts/apm-mustang-xen.dts
+++ b/arch/arm64/boot/dts/apm-mustang-xen.dts
@@ -18,6 +18,8 @@
        compatible = "apm,mustang", "apm,xgene-storm";
 
        chosen {
+               #address-cells=<2>;
+               #size-cells=<1>;
                module@0 {
                        bootargs = "console=hvc0 root=/dev/sda2 rw earlyprintk=uart8250-32bit,0x1c020000 debug maxcpus=8"; /* initcall_debug*/
                };

NOTE: Above bootargs expects that rootfs will reside on /dev/sda2 of your SATA drive.

To build the device tree blob (DTB), do:

# make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- dtbs
# Copy arch/arm64/boot/dts/apm-mustang-xen.dtb to your tftpboot directory.

Preparing SATA Drive

Assuming we have a SATA disk with 2nd partition empty and formatted with ext3 or ext4, do:

# wget http://download.opensuse.org/repositories/devel:/ARM:/AArch64:/12.3/images/openSUSE-12.3-AArch64-JeOS-rootfs.aarch64-1.12.1-Build5.13.tbz

Then mount the 2nd SATA partition and untar the above rootfs on the 2nd SATA partition

Booting DOM0 Kernel

• Check your tftp directory has following files:
  • uXen
  • Image
  • apm-mustang-xen.dts
• Make sure SATA disk (with above mantioned rootfs) is attached to mustang board.
• Make sure PCIe NiC (like Intel's e1000) is attached to your board and it's driver is enabled in DOM0 config (for e1000 CONFIG_E1000=y CONFIG_E1000E=y should be enabled).

On u-boot prompt execute:

# run xen_run

This should reach to DOM0 login prompt!!

DomU

Make sure your e1000 network card is up in DOM0 and has internet connectivity.

Instal necessary build packages like gcc, libfdt etc... (This is one time configuration after first boot):

# zypper update
# zypper addrepo http://download.opensuse.org/repositories/devel:/ARM:/AArch64:/12.3/standard/devel:ARM:AArch64:12.3.repo
# zypper install gcc make patterns-openSUSE-devel_basis extra packages: git vim libyajl-devel python-devel wget libfdt1-devel
# cp libfdt_env.h /usr/include/

libfdt_env.h -> Can be found in DOM0 linux source at "scripts/dtc/libfdt/" folder.

Build xen tools

# git clone git://xenbits.xen.org/xen.git
# cd xen
# ./configure
# make tools
# make install

If you see error messages relating to redefinition of PSR_MODE_EL3h or other PSR_MODE_* symbols then see error: "PSR_MODE_EL3h" redefined.

DomU Images

• Image : Kernel Image (same DOM0 kenel Image can be used here)
• rootfs.ext3 : Create ext3 rootdisk using an opensuse or linaro arm64 rootfs.

Booting DomU

First, setup the disk:

# losetup /dev/loop0 /root/domU/rootfs.ext3
# cat d
   name = "d"
   uuid = "3fb78ba6-8182-484c-acf7-8faba9773f68"
   disk = ["phy:/dev/loop0,xvda,rw"]
   memory = 512
   kernel = "./Image"
   extra = "root=/dev/xvda rw console=hvc0 earlyprintk=xen"

# /etc/init.d/xencommons start || true
# echo 9 > /proc/sysrq-trigger
# export LIBXL_DEBUG_DUMP_DTB=guest.dtb
# xl -vvv create d
# xl console d

Author

• This guide was written by Pranavkumar Sawargaonkar (pranavkumar _at_ linaro _dot_ org).
• Modified to the Xen wiki by: Christoffer Dall (christoffer _dot_ dall _at_ linaro _dot_ org)