Difference between revisions of "Xen ARM with Virtualization Extensions/Salvator-XS"

From Xen
(Build Xen)
(Run the system)
 
(22 intermediate revisions by the same user not shown)
Line 1: Line 1:
== (NOT COMPLETE! in the process of creating) ==
 
 
= General information =
 
= General information =
This Wiki describes how to run Xen on Renesas Salvator-XS board with R-Car H3 ES3.0 SoC.
+
This Wiki describes how to run system with Xen and Dom0 on Renesas Salvator-XS board with R-Car H3 ES3.0 SoC (Arm64).
   
 
Main information how to deal with Salvator-XS board located at:
 
Main information how to deal with Salvator-XS board located at:
 
https://elinux.org/R-Car/Boards/Salvator-XS
 
https://elinux.org/R-Car/Boards/Salvator-XS
   
Please note, although Wiki only covers Salvator-XS + H3 ES3.0 configuration, the R-Car M3 ES3.0 (M3-W+) SoC is also supported. It is possible run Xen on other "board & SoC" configurations with updating platform specific parts (device-tree, etc), the hypervisor part will remain the same. For example, the following configurations will work as well:
+
Please note, although Wiki only covers Salvator-XS + H3 ES3.0 configuration, the R-Car M3 ES3.0 (M3-W+) SoC is also supported. It is possible to run Xen on other "board & SoC" configurations with updating platform specific parts (device-tree, etc), the hypervisor part will remain the same. For example, the following configurations will work as well:
 
https://elinux.org/R-Car/Boards/M3SK
 
https://elinux.org/R-Car/Boards/M3SK
 
https://elinux.org/R-Car/Boards/H3SK
 
https://elinux.org/R-Car/Boards/H3SK
   
Due to the fact that Renesas provides their reference software in form of Yocto build instruction, it is provided additional steps in order to build and run system with Xen and Dom0. This Wiki relies on Renesas Yocto BSP [https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.1.0 v5.1.0]
+
Due to the fact that Renesas provides their reference software in form of Yocto BSP build instruction, this Wiki provides additional steps in order to build and run system with Xen and Dom0, it relies on Renesas Yocto BSP [https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.1.0 v5.1.0]
   
= BSP build =
+
= Build BSP =
* Follow the build [https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.1.0 instruction] in '''Manual steps''' mode up to the build step #4, including it.
+
* Follow the build [https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.1.0 instruction] in '''Manual steps''' mode up to the build step #4, including it.
* Clone [https://github.com/xen-troops/meta-renesas meta-renesas] layer from xen-troops and cherry-pick last 4 patches from branch [https://github.com/xen-troops/meta-renesas/commits/v5.1.0_xen v5.1.0_xen], this is the minimum set of patches needed to run system with Xen and Dom0:
+
* Clone [https://github.com/xen-troops/meta-renesas meta-renesas] layer from xen-troops and cherry-pick last 4 patches from branch [https://github.com/xen-troops/meta-renesas/commits/v5.1.0_xen v5.1.0_xen], this is the minimum set of patches needed to run system with Xen and Dom0. Please note, all local changes need to be stashed or committed:
 
cd ${WORK}/meta-renesas
 
cd ${WORK}/meta-renesas
 
git remote add troops git@github.com:xen-troops/meta-renesas.git
 
git remote add troops git@github.com:xen-troops/meta-renesas.git
 
git fetch troops
 
git fetch troops
  +
git stash
 
git cherry-pick f806ddaf40c465378c483004359404a9f141dc98^..405b9b98e5831c16e6b968a2cddce0a5fdac1856
 
git cherry-pick f806ddaf40c465378c483004359404a9f141dc98^..405b9b98e5831c16e6b968a2cddce0a5fdac1856
  +
git stash apply
* Consider changes to r8a77951-salvator-xs-xen.dts added by one of the previous patches in case you want system configuration changes (i.e. NFS server ip, Dom0 root option, IPMMU settings, Xen command line, etc)
 
  +
* Consider changes to '''r8a77951-salvator-xs-xen.dts''' introduced by one of the previous patches in case you want system configuration changes (Dom0 command line, Xen command line, IPMMU settings, etc)
 
* Skip build steps #5, #6 and continue from build step #7 till the end
 
* Skip build steps #5, #6 and continue from build step #7 till the end
 
* In build step #8 change the path to local.conf file:
 
* In build step #8 change the path to local.conf file:
Line 26: Line 27:
 
cp conf/local-wayland.conf conf/local.conf
 
cp conf/local-wayland.conf conf/local.conf
   
= Xen build =
+
= Build Xen =
 
== Get Xen sources ==
 
== Get Xen sources ==
There is no need in additional patches here to bring up test system, the mainline [http://xenbits.xen.org/gitweb/?p=xen.git Xen] works out of the box. Clone sources from the upstream and checkout on commit 73c932d0ea43ddf904db9429811788480c4cb816 “tools/libxc: use uint32_t for pirq in xc_domain_irq_permission”, likely the more recent version will work as well, but the proposed commit is known to work:
+
There is no need in additional patches here to bring up the system with Xen and Dom0, the mainline [http://xenbits.xen.org/gitweb/?p=xen.git Xen] works out of the box. Clone sources from the upstream and checkout on commit 73c932d0ea43ddf904db9429811788480c4cb816 “tools/libxc: use uint32_t for pirq in xc_domain_irq_permission”. Please note, most likely the more recent version of Xen will work as well, but the proposed commit is known to work:
 
git clone git://xenbits.xenproject.org/xen.git
 
git clone git://xenbits.xenproject.org/xen.git
 
cd xen
 
cd xen
Line 51: Line 52:
 
(X) default SCIF UART interface
 
(X) default SCIF UART interface
   
== Build Xen ==
+
== Make Xen ==
 
cd ..
 
cd ..
 
make xen XEN_TARGET_ARCH=arm64
 
make xen XEN_TARGET_ARCH=arm64
Line 57: Line 58:
 
make -C tools/flask/policy
 
make -C tools/flask/policy
   
Copy resulting images to tftp directory (prepared beforehand) for the future use:
+
Copy resulting images to TFTP directory (prepared beforehand) for the future use:
 
sudo cp xen-uImage <path_to_tftp_dir>
 
sudo cp xen-uImage <path_to_tftp_dir>
 
sudo cp tools/flask/policy/xenpolicy* <path_to_tftp_dir>/xenpolicy
 
sudo cp tools/flask/policy/xenpolicy* <path_to_tftp_dir>/xenpolicy
  +
  +
= Run the system =
  +
  +
* Setup TFTP and NFS (including the U-Boot environment) according to the [https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.1.0#Loading_kernel_via_TFTP_and_rootfs_via_NFS instruction]. Please note, the default expected NFS settings are the following: "nfsroot=192.168.1.100:/srv/salvator-x,vers=3 ip=dhcp". Also make sure that TFTP directory contains all target images, in addition to "r8a77951-salvator-xs-xen.dtb" and "Image" two new images need to be loaded: "xen-uImage" and "xenpolicy".
  +
  +
* Update bootloaders on the board. It is crucial to update ArmTF in order to get Xen start in Non-secure EL2 mode. Follow your board manual and Yocto BSP documentation to find out the bootloaders update procedure.
  +
  +
* For the automatic boot add the following commands from the U-Boot CLI:
  +
setenv bootcmd run bootcmd_tftp
  +
setenv bootcmd_tftp 'run tftp_xen_load; run tftp_dtb_load; run tftp_kernel_load; run tftp_xenpolicy_load; bootm 0x48080000 - 0x48000000'
  +
setenv tftp_dtb_load tftp 0x48000000 r8a77951-salvator-xs-xen.dtb
  +
setenv tftp_kernel_load tftp 0x8a000000 Image
  +
setenv tftp_xen_load tftp 0x48080000 xen-uImage
  +
setenv tftp_xenpolicy_load tftp 0x8c000000 xenpolicy
  +
setenv bootargs
  +
saveenv
  +
  +
After rebooting the board, the system with Xen and Dom0 will start automatically if everything was done correctly.

Latest revision as of 11:27, 3 August 2021

General information

This Wiki describes how to run system with Xen and Dom0 on Renesas Salvator-XS board with R-Car H3 ES3.0 SoC (Arm64).

Main information how to deal with Salvator-XS board located at:

https://elinux.org/R-Car/Boards/Salvator-XS

Please note, although Wiki only covers Salvator-XS + H3 ES3.0 configuration, the R-Car M3 ES3.0 (M3-W+) SoC is also supported. It is possible to run Xen on other "board & SoC" configurations with updating platform specific parts (device-tree, etc), the hypervisor part will remain the same. For example, the following configurations will work as well:

https://elinux.org/R-Car/Boards/M3SK
https://elinux.org/R-Car/Boards/H3SK

Due to the fact that Renesas provides their reference software in form of Yocto BSP build instruction, this Wiki provides additional steps in order to build and run system with Xen and Dom0, it relies on Renesas Yocto BSP v5.1.0

Build BSP

  • Follow the build instruction in Manual steps mode up to the build step #4, including it.
  • Clone meta-renesas layer from xen-troops and cherry-pick last 4 patches from branch v5.1.0_xen, this is the minimum set of patches needed to run system with Xen and Dom0. Please note, all local changes need to be stashed or committed:
cd ${WORK}/meta-renesas
git remote add troops git@github.com:xen-troops/meta-renesas.git
git fetch troops
git stash
git cherry-pick f806ddaf40c465378c483004359404a9f141dc98^..405b9b98e5831c16e6b968a2cddce0a5fdac1856
git stash apply
  • Consider changes to r8a77951-salvator-xs-xen.dts introduced by one of the previous patches in case you want system configuration changes (Dom0 command line, Xen command line, IPMMU settings, etc)
  • Skip build steps #5, #6 and continue from build step #7 till the end
  • In build step #8 change the path to local.conf file:
cp $WORK/meta-renesas/meta-rcar-gen3/docs/sample/conf/salvator-x/poky-gcc/bsp/*.conf ./conf/
cd $WORK/build
cp conf/local-wayland.conf conf/local.conf

Build Xen

Get Xen sources

There is no need in additional patches here to bring up the system with Xen and Dom0, the mainline Xen works out of the box. Clone sources from the upstream and checkout on commit 73c932d0ea43ddf904db9429811788480c4cb816 “tools/libxc: use uint32_t for pirq in xc_domain_irq_permission”. Please note, most likely the more recent version of Xen will work as well, but the proposed commit is known to work:

git clone git://xenbits.xenproject.org/xen.git
cd xen
git checkout 73c932d0ea43ddf904db9429811788480c4cb816 -b v4.16_xen

Configure Xen

export CROSS_COMPILE=<path_to_gcc>
XEN_TARGET_ARCH=arm64 ./configure
cd xen
make menuconfig XEN_TARGET_ARCH=arm64

Enable IPMMU-VMSA support. Please note, the DMA devices in Dom0 will work even with IPMMU being disabled, as Dom0 is a direct mapped domain on Arm for now, therefore the IPMMU here is for protection purpose. But, if you are going to launch other domains (which are not direct mapped) and assign DMA devices to them, the IPMMU must be enabled to perform proper address translations for these devices to work:

Device Drivers  --->
 [*] Renesas IPMMU-VMSA found in R-Car Gen3 SoCs

Enable earlyprintk support (if needed):

Debugging Options  --->
 [*] Early printk (Early printk via SCIF UART)  --->
  (X) Early printk via SCIF UART
  (0xe6e88000) Early printk, physical base address of debug UART
  Early printk UART SCIF interface version (default SCIF UART interface)  --->
   (X) default SCIF UART interface

Make Xen

cd ..
make xen XEN_TARGET_ARCH=arm64
mkimage -A arm64 -C none -T kernel -a 0x78080000 -e 0x78080000 -n "XEN" -d xen/xen xen-uImage
make -C tools/flask/policy

Copy resulting images to TFTP directory (prepared beforehand) for the future use:

sudo cp xen-uImage <path_to_tftp_dir>
sudo cp tools/flask/policy/xenpolicy* <path_to_tftp_dir>/xenpolicy

Run the system

  • Setup TFTP and NFS (including the U-Boot environment) according to the instruction. Please note, the default expected NFS settings are the following: "nfsroot=192.168.1.100:/srv/salvator-x,vers=3 ip=dhcp". Also make sure that TFTP directory contains all target images, in addition to "r8a77951-salvator-xs-xen.dtb" and "Image" two new images need to be loaded: "xen-uImage" and "xenpolicy".
  • Update bootloaders on the board. It is crucial to update ArmTF in order to get Xen start in Non-secure EL2 mode. Follow your board manual and Yocto BSP documentation to find out the bootloaders update procedure.
  • For the automatic boot add the following commands from the U-Boot CLI:
setenv bootcmd run bootcmd_tftp
setenv bootcmd_tftp 'run tftp_xen_load; run tftp_dtb_load; run tftp_kernel_load; run tftp_xenpolicy_load; bootm 0x48080000 - 0x48000000'
setenv tftp_dtb_load tftp 0x48000000 r8a77951-salvator-xs-xen.dtb
setenv tftp_kernel_load tftp 0x8a000000 Image
setenv tftp_xen_load tftp 0x48080000 xen-uImage
setenv tftp_xenpolicy_load tftp 0x8c000000 xenpolicy
setenv bootargs
saveenv

After rebooting the board, the system with Xen and Dom0 will start automatically if everything was done correctly.