Xen ARM with Virtualization Extensions/Salvator-XS

General information

This Wiki describes how to run system with Xen and Dom0 on Renesas Salvator-XS board with R-Car H3 ES3.0 SoC (Arm64).

Main information how to deal with Salvator-XS board located at:

https://elinux.org/R-Car/Boards/Salvator-XS

Please note, although Wiki only covers Salvator-XS + H3 ES3.0 configuration, the R-Car M3 ES3.0 (M3-W+) SoC is also supported. It is possible to run Xen on other "board & SoC" configurations with updating platform specific parts (device-tree, etc), the hypervisor part will remain the same. For example, the following configurations will work as well:

https://elinux.org/R-Car/Boards/M3SK
https://elinux.org/R-Car/Boards/H3SK

Due to the fact that Renesas provides their reference software in form of Yocto BSP build instruction, this Wiki provides additional steps in order to build and run system with Xen and Dom0, it relies on Renesas Yocto BSP v5.1.0

Build BSP

• Follow the build instruction in Manual steps mode up to the build step #4, including it.
• Clone meta-renesas layer from xen-troops and cherry-pick last 4 patches from branch v5.1.0_xen, this is the minimum set of patches needed to run system with Xen and Dom0. Please note, all local changes need to be stashed or committed:

cd ${WORK}/meta-renesas
git remote add troops git@github.com:xen-troops/meta-renesas.git
git fetch troops
git stash
git cherry-pick f806ddaf40c465378c483004359404a9f141dc98^..405b9b98e5831c16e6b968a2cddce0a5fdac1856
git stash apply

• Consider changes to r8a77951-salvator-xs-xen.dts introduced by one of the previous patches in case you want system configuration changes (Dom0 command line, Xen command line, IPMMU settings, etc)
• Skip build steps #5, #6 and continue from build step #7 till the end
• In build step #8 change the path to local.conf file:

cp $WORK/meta-renesas/meta-rcar-gen3/docs/sample/conf/salvator-x/poky-gcc/bsp/*.conf ./conf/
cd $WORK/build
cp conf/local-wayland.conf conf/local.conf

Build Xen

Get Xen sources

There is no need in additional patches here to bring up the system with Xen and Dom0, the mainline Xen works out of the box. Clone sources from the upstream and checkout on commit 73c932d0ea43ddf904db9429811788480c4cb816 “tools/libxc: use uint32_t for pirq in xc_domain_irq_permission”. Please note, most likely the more recent version of Xen will work as well, but the proposed commit is known to work:

git clone git://xenbits.xenproject.org/xen.git
cd xen
git checkout 73c932d0ea43ddf904db9429811788480c4cb816 -b v4.16_xen

Configure Xen

export CROSS_COMPILE=<path_to_gcc>
XEN_TARGET_ARCH=arm64 ./configure
cd xen
make menuconfig XEN_TARGET_ARCH=arm64

Enable IPMMU-VMSA support. Please note, the DMA devices in Dom0 will work even with IPMMU being disabled, as Dom0 is a direct mapped domain on Arm for now, therefore the IPMMU here is for protection purpose. But, if you are going to launch other domains (which are not direct mapped) and assign DMA devices to them, the IPMMU must be enabled to perform proper address translations for these devices to work:

Device Drivers  --->
 [*] Renesas IPMMU-VMSA found in R-Car Gen3 SoCs

Enable earlyprintk support (if needed):

Debugging Options  --->
 [*] Early printk (Early printk via SCIF UART)  --->
  (X) Early printk via SCIF UART
  (0xe6e88000) Early printk, physical base address of debug UART
  Early printk UART SCIF interface version (default SCIF UART interface)  --->
   (X) default SCIF UART interface

Make Xen

cd ..
make xen XEN_TARGET_ARCH=arm64
mkimage -A arm64 -C none -T kernel -a 0x78080000 -e 0x78080000 -n "XEN" -d xen/xen xen-uImage
make -C tools/flask/policy

Copy resulting images to TFTP directory (prepared beforehand) for the future use:

sudo cp xen-uImage <path_to_tftp_dir>
sudo cp tools/flask/policy/xenpolicy* <path_to_tftp_dir>/xenpolicy

Run the system

• Setup TFTP and NFS (including the U-Boot environment) according to the instruction. Please note, the default expected NFS settings are the following: "nfsroot=192.168.1.100:/srv/salvator-x,vers=3 ip=dhcp". Also make sure that TFTP directory contains all target images, in addition to "r8a77951-salvator-xs-xen.dtb" and "Image" two new images need to be loaded: "xen-uImage" and "xenpolicy".

• Update bootloaders on the board. It is crucial to update ArmTF in order to get Xen start in Non-secure EL2 mode. Follow your board manual and Yocto BSP documentation to find out the bootloaders update procedure.

• For the automatic boot add the following commands from the U-Boot CLI:

setenv bootcmd run bootcmd_tftp
setenv bootcmd_tftp 'run tftp_xen_load; run tftp_dtb_load; run tftp_kernel_load; run tftp_xenpolicy_load; bootm 0x48080000 - 0x48000000'
setenv tftp_dtb_load tftp 0x48000000 r8a77951-salvator-xs-xen.dtb
setenv tftp_kernel_load tftp 0x8a000000 Image
setenv tftp_xen_load tftp 0x48080000 xen-uImage
setenv tftp_xenpolicy_load tftp 0x8c000000 xenpolicy
setenv bootargs
saveenv

After rebooting the board, the system with Xen and Dom0 will start automatically if everything was done correctly.