Xen ARM with Virtualization Extensions/Stout

From Xen
Revision as of 12:22, 20 May 2019 by Otyshchenko (talk | contribs)

[WIP]

This Wiki describes how to run Xen on Renesas Stout board with R-Car H2 SoC.

Main information how to deal with Stout board are located at: https://elinux.org/R-Car/Boards/Yocto https://elinux.org/R-Car/Boards/Stout

Prepare U-boot

Get U-Boot sources

Clone sources from upstream and checkout on commit 425c0a43fbbec36571f6a03b530695b8b16a841d “Prepare v2019.01-rc3”:

git clone git://git.denx.de/u-boot.git
cd u-boot
git checkout 425c0a43fbbec36571f6a03b530695b8b16a841d -b stout_uboot

Apply additional patches

Apply the following patch series (PSCI support for r8a7790 SoC): http://u-boot.10912.n7.nabble.com/PATCH-0-3-PSCI-support-for-r8a7790-SoC-Lager-Stout-boards-td357352.html

Configure U-Boot

export CROSS_COMPILE=<path_to_gcc>
make stout_defconfig
make menuconfig

Set maximum supported CPUs for PSCI to 8:

    ARM architecture  --->
        [*] Enable support for booting in non-secure mode
            (8) Maximum supported CPUs for PSCI

Build U-Boot

make

Copy resulting images to tftp directory for the future flashing:

sudo cp u-boot.img <path_to_tftp_dir>/
sudo cp spl/u-boot-spl.bin <path_to_tftp_dir>/

Prepare Xen

Get Xen sources

Clone sources from upstream and checkout on commit ca135a4863ffeedca451651835fcd9aef6c872ff “xen/arm: Add early printk support for SCIFA compatible UARTs”:

git clone git://xenbits.xenproject.org/xen.git
cd xen
git checkout ca135a4863ffeedca451651835fcd9aef6c872ff -b stout_xen

Apply additional patches

Apply the following small patch series (Support of handling nodes with “interrupts-extended” property): https://www.mail-archive.com/xen-devel@lists.xenproject.org/msg44364.html

Please note, if you can’t apply that patch series to Xen for some reason, then you should apply the following patch to Linux when preparing it (to not use “interrupts-extended” property for ARCH timer node): "0003-r8a7790.dtsi-temp-Use-interrupt-parent-interrupts-fo.patch"

Configure Xen

export CROSS_COMPILE=<path_to_gcc>
XEN_TARGET_ARCH=arm32 ./configure
cd xen
make menuconfig XEN_TARGET_ARCH=arm32 

Enable debugging option:

    Debugging Options  --->
        [*] Developer Checks

Build Xen

With earlyprintk support:

cd ../
make xen XEN_TARGET_ARCH=arm32 debug=y CONFIG_EARLY_PRINTK=scif,0xe6c40000,A

- or -

Without earlyprintk support:

cd ../
make xen XEN_TARGET_ARCH=arm32 debug=y

Make Xen uImage:

mkimage -A arm -C none -T kernel -a 0x48000000 -e 0x48000000 -n "XEN" -d xen/xen xen-uImage

Build xenpolicy:

make -C tools/flask/policy

Copy resulting images to micro SD card (prepared beforehand):

sudo cp xen-uImage <path_to_sd_card_partition>/boot/
sudo cp tools/flask/policy/xenpolicy* <path_to_sd_card_partition>/boot/xenpolicy

Prepare Dom0 Linux and Host DT

Get Linux sources

Clone sources from upstream and checkout on v5.1 tag:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout v5.1 -b stout_linux

Apply additional patches

1. Apply the following patch (to not initialize CNTVOFF/counter module): https://marc.info/?l=linux-kernel&m=155808712932351&w=2

2. Apply the following patch (minimal Host DT needed to run Xen):

diff --git a/arch/arm/boot/dts/Makefile b/arch/arm/boot/dts/Makefile
index f4f5aea..d7e0419 100644
--- a/arch/arm/boot/dts/Makefile
+++ b/arch/arm/boot/dts/Makefile
@@ -869,6 +869,7 @@ dtb-$(CONFIG_ARCH_RENESAS) += \
 	r8a7779-marzen.dtb \
 	r8a7790-lager.dtb \
 	r8a7790-stout.dtb \
+	r8a7790-stout-host.dtb \
 	r8a7791-koelsch.dtb \
 	r8a7791-porter.dtb \
 	r8a7792-blanche.dtb \
diff --git a/arch/arm/boot/dts/r8a7790-stout-host.dts b/arch/arm/boot/dts/r8a7790-stout-host.dts
new file mode 100644
index 0000000..8ef19dd
--- /dev/null
+++ b/arch/arm/boot/dts/r8a7790-stout-host.dts
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Device Tree Source for the Stout board running XEN hypervisor
+ *
+ * Copyright (C) 2018 EPAM Systems Inc.
+ */
+
+#include "r8a7790-stout.dts"
+
+/ {
+	model = "Stout board, running XEN hypervisor";
+	compatible = "renesas,stout", "renesas,r8a7790";
+
+	chosen {
+		bootargs = "dom0_mem=768M console=dtuart dtuart=serial0 dom0_vcpus_pin hmp-unsafe=true dom0_max_vcpus=8 bootscrub=0 loglvl=all";
+		/delete-property/stdout-path;
+		xen,dom0-bootargs = "console=hvc0 ignore_loglevel root=/dev/mmcblk0p1 rootfstype=ext3 rw rootwait clk_ignore_unused";
+
+		modules {
+			#address-cells = <2>;
+			#size-cells = <2>;
+			module@1 {
+				compatible = "xen,linux-zimage", "xen,multiboot-module";
+				reg = <0x0 0x72000000 0x0 0x02000000>;
+			};
+			module@2 {
+				compatible = "xen,xsm-policy", "xen,multiboot-module";
+				reg = <0x0 0x74000000 0x0 0x10000>;
+			};
+		};
+	};
+};
+
+&scifa0 {
+	/delete-property/clocks;
+};

3. Apply the following patch (to not use “interrupts-extended” property for ARCH timer node) if needed:

diff --git a/arch/arm/boot/dts/r8a7790.dtsi b/arch/arm/boot/dts/r8a7790.dtsi
index 5a27477..8063d8b 100644
--- a/arch/arm/boot/dts/r8a7790.dtsi
+++ b/arch/arm/boot/dts/r8a7790.dtsi
@@ -1855,10 +1855,11 @@
 
 	timer {
 		compatible = "arm,armv7-timer";
-		interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
-				      <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
-				      <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
-				      <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>;
+		interrupt-parent = <&gic>;
+		interrupts = <GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
+				     <GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
+				     <GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,
+				     <GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>;
 	};
 
 	/* External USB clock - can be overridden by the board */

Configure Linux

Use default "shmobile_defconfig" as the base config:

export CROSS_COMPILE=<path_to_gcc>
make ARCH=arm shmobile_defconfig

Add the following config entries in .config:

CONFIG_ARM_PSCI=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_XEN_WDT is not set
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y

Build Linux and DT

LOADADDR=0x48000000 make ARCH=arm uImage dtbs

Copy resulting images to SD card (prepared beforehand):

sudo cp arch/arm/boot/dts/r8a7790-stout-host.dtb <path_to_sd_card_partition>/boot/
sudo cp arch/arm/boot/uImage <path_to_sd_card_partition>/boot/

Run images

Flash U-Boot

Flash U-Boot according to the instruction described here: https://elinux.org/R-Car/Boards/U-Boot-Gen2#Flashing_U-Boot Please note, you need to setup TFTP for this purpose.

Setup U-Boot environment

Setup U-Boot environment to boot system via SD card. This is done using U-Boot CLI:

env default -f -a
setenv dtb_load_mmc 'ext4load mmc 0:1 0x70f00000 /boot/r8a7790-stout-host.dtb'
setenv xen_load_mmc 'ext4load mmc 0:1 0x70007fc0 /boot/xen-uImage'
setenv kernel_load_mmc 'ext4load mmc 0:1 0x72000000 /boot/uImage'
setenv xenpolicy_load_mmc 'ext4load mmc 0:1 0x74000000 /boot/xenpolicy'
setenv bootcmd_xen_mmc 'run dtb_load_mmc; run xen_load_mmc; run kernel_load_mmc; run xenpolicy_load_mmc; bootm 0x70007fc0 - 0x70f00000'
setenv bootargs
setenv bootcmd 'run bootcmd_xen_mmc'
saveenv

After rebooting the board the system (Xen + Dom0) will be started automatically if everything was done correctly.