Xen ARM with Virtualization Extensions/Stout

This Wiki describes how to run Xen on Renesas Stout board with R-Car H2 SoC.

Main information how to deal with Stout board are located at: https://elinux.org/R-Car/Boards/Yocto https://elinux.org/R-Car/Boards/Stout

Prepare U-boot

Get U-Boot sources

Clone sources from upstream and checkout on commit 425c0a43fbbec36571f6a03b530695b8b16a841d “Prepare v2019.01-rc3”:

git clone git://git.denx.de/u-boot.git
cd u-boot
git checkout 425c0a43fbbec36571f6a03b530695b8b16a841d -b stout_uboot

Apply additional patches

Apply the following patch series (PSCI support for r8a7790 SoC): http://u-boot.10912.n7.nabble.com/PATCH-0-3-PSCI-support-for-r8a7790-SoC-Lager-Stout-boards-td357352.html

Configure U-Boot

export CROSS_COMPILE=<path_to_gcc>
make stout_defconfig
make menuconfig

Set maximum supported CPUs for PSCI to 8:

    ARM architecture  --->
        [*] Enable support for booting in non-secure mode
            (8) Maximum supported CPUs for PSCI

Build U-Boot

make

Copy resulting images to tftp directory for the future flashing:

sudo cp u-boot.img <path_to_tftp_dir>/
sudo cp spl/u-boot-spl.bin <path_to_tftp_dir>/

Prepare Xen

Get Xen sources

Clone sources from upstream and checkout on commit ca135a4863ffeedca451651835fcd9aef6c872ff “xen/arm: Add early printk support for SCIFA compatible UARTs”:

git clone git://xenbits.xenproject.org/xen.git
cd xen
git checkout ca135a4863ffeedca451651835fcd9aef6c872ff -b stout_xen

Apply additional patches

Apply the following small patch series (Support of handling nodes with “interrupts-extended” property): https://www.mail-archive.com/xen-devel@lists.xenproject.org/msg45951.html

Configure Xen

export CROSS_COMPILE=<path_to_gcc>
XEN_TARGET_ARCH=arm32 ./configure
cd xen
make menuconfig XEN_TARGET_ARCH=arm32 

Enable debugging option:

    Debugging Options  --->
        [*] Developer Checks

Build Xen

With earlyprintk support:

cd ../
make xen XEN_TARGET_ARCH=arm32 debug=y CONFIG_EARLY_PRINTK=scif,0xe6c40000,A

- or -

Without earlyprintk support:

cd ../
make xen XEN_TARGET_ARCH=arm32 debug=y

Make Xen uImage:

mkimage -A arm -C none -T kernel -a 0x48000000 -e 0x48000000 -n "XEN" -d xen/xen xen-uImage

Build xenpolicy:

make -C tools/flask/policy

Copy resulting images to micro SD card (prepared beforehand):

sudo cp xen-uImage <path_to_sd_card_partition>/boot/
sudo cp tools/flask/policy/xenpolicy* <path_to_sd_card_partition>/boot/xenpolicy

Prepare Dom0 Linux and Host DT

Get Linux sources

Clone sources from upstream and checkout on v5.1 tag:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout v5.1 -b stout_linux

Apply additional patches

Apply the following patch (minimal Host DT needed to run Xen):

diff --git a/arch/arm/boot/dts/Makefile b/arch/arm/boot/dts/Makefile
index f4f5aea..d7e0419 100644
--- a/arch/arm/boot/dts/Makefile
+++ b/arch/arm/boot/dts/Makefile
@@ -869,6 +869,7 @@ dtb-$(CONFIG_ARCH_RENESAS) += \
 	r8a7779-marzen.dtb \
 	r8a7790-lager.dtb \
 	r8a7790-stout.dtb \
+	r8a7790-stout-host.dtb \
 	r8a7791-koelsch.dtb \
 	r8a7791-porter.dtb \
 	r8a7792-blanche.dtb \
diff --git a/arch/arm/boot/dts/r8a7790-stout-host.dts b/arch/arm/boot/dts/r8a7790-stout-host.dts
new file mode 100644
index 0000000..8ef19dd
--- /dev/null
+++ b/arch/arm/boot/dts/r8a7790-stout-host.dts
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Device Tree Source for the Stout board running XEN hypervisor
+ *
+ * Copyright (C) 2018 EPAM Systems Inc.
+ */
+
+#include "r8a7790-stout.dts"
+
+/ {
+	model = "Stout board, running XEN hypervisor";
+	compatible = "renesas,stout", "renesas,r8a7790";
+
+	chosen {
+		bootargs = "dom0_mem=768M console=dtuart dtuart=serial0 dom0_vcpus_pin hmp-unsafe=true dom0_max_vcpus=8 bootscrub=0 loglvl=all";
+		/delete-property/stdout-path;
+		xen,dom0-bootargs = "console=hvc0 ignore_loglevel root=/dev/mmcblk0p1 rootfstype=ext3 rw rootwait clk_ignore_unused";
+
+		modules {
+			#address-cells = <2>;
+			#size-cells = <2>;
+			module@1 {
+				compatible = "xen,linux-zimage", "xen,multiboot-module";
+				reg = <0x0 0x72000000 0x0 0x02000000>;
+			};
+			module@2 {
+				compatible = "xen,xsm-policy", "xen,multiboot-module";
+				reg = <0x0 0x74000000 0x0 0x10000>;
+			};
+		};
+	};
+};
+
+&scifa0 {
+	/delete-property/clocks;
+};

Configure Linux

Use default "shmobile_defconfig" as the base config:

export CROSS_COMPILE=<path_to_gcc>
make ARCH=arm shmobile_defconfig

Add the following config entries in .config:

CONFIG_ARM_PSCI=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_XEN_WDT is not set
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y

Build Linux and DT

LOADADDR=0x48000000 make ARCH=arm uImage dtbs

Copy resulting images to SD card (prepared beforehand):

sudo cp arch/arm/boot/dts/r8a7790-stout-host.dtb <path_to_sd_card_partition>/boot/
sudo cp arch/arm/boot/uImage <path_to_sd_card_partition>/boot/

Run images

Flash U-Boot

Flash U-Boot according to the instruction described here: https://elinux.org/R-Car/Boards/U-Boot-Gen2#Flashing_U-Boot Please note, you need to setup TFTP for this purpose.

Setup U-Boot environment

Setup U-Boot environment to boot system via SD card. This is done using U-Boot CLI:

env default -f -a
setenv dtb_load_mmc 'ext4load mmc 0:1 0x70f00000 /boot/r8a7790-stout-host.dtb'
setenv xen_load_mmc 'ext4load mmc 0:1 0x70007fc0 /boot/xen-uImage'
setenv kernel_load_mmc 'ext4load mmc 0:1 0x72000000 /boot/uImage'
setenv xenpolicy_load_mmc 'ext4load mmc 0:1 0x74000000 /boot/xenpolicy'
setenv bootcmd_xen_mmc 'run dtb_load_mmc; run xen_load_mmc; run kernel_load_mmc; run xenpolicy_load_mmc; bootm 0x70007fc0 - 0x70f00000'
setenv bootargs
setenv bootcmd 'run bootcmd_xen_mmc'
saveenv

After rebooting the board the system (Xen + Dom0) will be started automatically if everything was done correctly.