Book/HelloXenProject/3-Chapter
Chapter 3: Dom(s) and Grub
Welcome to Chapter 3. We want to speak about Dom0 and DomU. I'm sure that currently you know what is Xen Domain but we want to speak about it more.
Dom0 is the initial domain started by the Xen hypervisor on boot. Its mean “Domain0” and “Dom0” is an abbreviation for it. The Dom0 duty is a privilege domain that manages DomU domains. Without it our Hypervisor is useless and it is kind of “Host” OS, But have special and low level access to hardware. The Dom0 has Drivers for your Hardware but you must know something about the Backend Driver and Front-end Driver. These are two terms for Xen Project.
If DomU wants share hardware, then Dom0 must give it an interface for make requests for access to the hardware. This is done via Backend Driver. For access to the Hardware that shared between Domains, DomU must contact to Dom0. The Front-end Driver prepared for the OS in DomU and use XenBus, XenStore, Shared Pages and... for communicating with Backend Driver. I just want to tell you that “XenStore” is an information storage space that shred between domains and maintained by the Xenstored. The XenBus provides a bus abstraction for Paravirtualized drivers to communicate between domains.
For more information about “XenStore” and “XenBus” Please see XenStore and XenBus.
In below Figure, You can see a cute photo about Dom0 and DomU :
I guess you are impatiently waiting for how you can configure Dom0 and DomU. The dome is used for configuration our Guest OS and we will configure it in the next chapter but we will Play with Dom0 here. I know that your thinking, that we want to hack Dom0 here, but your wrong, We just want to show you a way to manipulate:
Change Dom0 memory
For change memory setting you must open /etc/default/grub with an editor like “nano” and apply your Arbitrary settings :
[root@localhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root00 rd.lvm.lv=fedora/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
Find “GRUB_CMDLINE_LINUX” and add “dom0_mem=512M, max: 512M” at the end of this line. After it, Your Dom0 will use 512 megabyte memory and “max” parameter means the maximum memory usage:
[root@localhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root00 rd.lvm.lv=fedora/swap rhgb quiet dom0_mem=512M,max:512M"
GRUB_DISABLE_RECOVERY="true"
Then you must run “grub2-mkconfig -o /boot/grub2/grub.cfg” and reboot your system. After it, The Dom0 will use 512 Megabyte of memory. You can use B for bytes, K for kilobytes, M for megabytes, or G for gigabytes.
If you do some research the you will find a “ballooned” term. The Xen Project from version 3+ supporting memory “ballooning” that allow you to hot, add and remove memory from a running System. It is a technique that used to dynamically adjust the physical memory in use by a guest. Let me show you an example. I have a VM and give it 1024MB memory, but I notice that it just uses %50 of its memory and 512M of my memory wasted and The Xen Project let me to re-size my VM memory without any Downtime. Now that you are familiar with this Term, I want to show you that you can config it via “XL” command.
Open “/etc/xen/xl.conf” file and find “#autoballoon="auto"” then remove “#” and change “auto” to “0”. After it, the “xl” can't adjust the amount of memory assigned to dom0 :
[root@localhost ~]# cat /etc/xen/xl.conf
## Global XL config file ##
# Control whether dom0 is ballooned down when xen doesn't have enough
# free memory to create a domain. "auto" means only balloon if dom0
# starts with all the host's memory.
#autoballoon="auto"
# full path of the lockfile used by xl during domain creation
#lockfile="/var/lock/xl"
...
Change :
[root@localhost ~]# cat /etc/xen/xl.conf
## Global XL config file ##
# Control whether dom0 is ballooned down when xen doesn't have enough
# free memory to create a domain. "auto" means only balloon if dom0
# starts with all the host's memory.
autoballoon="0"
# full path of the lockfile used by xl during domain creation
#lockfile="/var/lock/xl"
...
You may ask yourself, Why you must dedicate the fixed amount of memory for Dom0. It is a good idea because :
- The Linux Kernel calculates various network parameters at the boot time and based on this memory.
- For store the memory metadata, The Linux need some amount of memory and this feature based on the memory on the boot time.
For more information about “Ballooned”, Please look at “https://blog.xenproject.org/2014/02/14/ballooning-rebooting-and-the-feature-youve-never-heard-of/” and “http://xenbits.xen.org/people/liuw/xen-balloon-driver-improvement.pdf”.
Limit the number of vcpus
To dedicate “vcpu” to Dom0 you must use “dom0_max_vcpus=X” option and “X” is the number of vcpu that you need. You must open “grub.conf” via an editor like “nano” and add above option to it :
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root00 rd.lvm.lv=fedora/swap rhgb quiet dom0_max_vcpus=2"
GRUB_DISABLE_RECOVERY="true"
As you see, I dedicated two VCPUs to my Dom0.
Let me guess your question! I guess you are thinking about dedicate a CPU core to Dom0. Am I right? I must tell you that you can. It is possible and sometime is a good idea and guarantee that Dom0 always has Free CPU time and can process the I/O requests from DomUs. For doing it, You must use “ dom0_max_vcpus=X dom0_vcpus_pin” option in your “grub.conf” file :
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root00 rd.lvm.lv=fedora/swap rhgb quiet dom0_max_vcpus=2 dom0_vcpus_pin"
GRUB_DISABLE_RECOVERY="true"
After rebooting you can test it via “xl vcpu-list” command.
After it, The next step is to configure your Guests. You must configure them because they must not use those same physical CPUs. This can do via the guest configuration file under “/etc/xen/<guest>.cfg” file. We will talk about it in the future.
You can do more research and find some interested parameters that you like. We prepared a table that shows you some dom0 options :
| Option Description | |
|---|---|
| dom0_mem = X | Amount of Memory that allocated to Dom0. |
| apic=default | Via this option you can manipulate interrupt controllers but it is detected automatically. |
| nolapic | Don't use APIC even it is enabled by BIOS. |
| nosmp | Disable SMP support. |
| watchdog | It is using for debugging and can be helpful in reporting failures. |
| noreboot | Don't reboot machine automatically when an error happened. |
| Vga | Use VGA console. |
| com1 | Use serial Port COM1 |
To be honest, these options that are said are important, But you can find a list of these hacking at http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html and Tuning_Xen_for_Performance.
Next we would like to talk about the Xen Project and Grub.
Xen Project and Grub
If you remember, we briefly talked about PvGrub and PyGrub. The PyGrub enable you to start DomU with a Kernel that inside of DomU instead of Kernel that lies in the filesystem of the Dom0. As you see, this cause easier management and each DomU manage its own kernel and if you manipulate DomU kernel then it can't affect on Dom0 Kernel. The PyGrub have a behavior like Grub and read the standard Grub configuration file (grub.cfg).
The PyGrub, use libfsimage for support Ext2/3/4, FAT, ISO9660, REISERFS, UFS and ZFS File Systems. If you want your DomU use PyGrub then you must add “bootloader = "/usr/bin/pygrub"” in your DomU config file. The PyGrub just support Grub 1 & Grub 2, LILO and ExtLinux bootloaders. Don't forget, Use it is not as easy we said and some Linux Distros need special configuration for it. For example, For Configure it for Debian Please see “https://tipstricks.itmatrix.eu/using-pygrub-on-squeeze-to-boot-a-domu-kernel/”.
The PvGrub or ParaVirtual Grub is a boot manager for Xen Vms. It is a safer alternative for PyGrub to boot DomU images. We can say that the PvGrub is an adapted version of the grub boot loader. The PvGrub require Xen 3.3 or higher. It is provided by the Xen source and you don't need to install grub in DomU. PvGrub Compared with PyGrub has less bugs and is more secure. PvGrub is not scripted and is binary that has less dependencies. For a good How to about it, Please see “http://backdrift.org/xen-pvgrub-howto”.
Now, The time is come to speak about Grub bootloader.
Grub2
A boot loader is the first program that runs when a computer start. The boot loader is responsible for loading and transferring control to an operating system kernel like Linux and then the Kernel initialize the rest of the OS. Grub2 is a powerful boot loader that support modern OS. One the important feature of Grub is flexibility. GNU Grub can offer you a command-line interface and a menu interface. Via CLI you can type the drive and the kernel file manually, but via Menu you select an OS via arrow Keys.
Grub2 is a rewrite version of Grub and have some changes. For example, In the previous version of Grub, The config file name was “menu.lst” or “grub.conf” but in Grub2, It changed to “grub.cfg”. A good advantage of Grub2 is that it can support many file systems and not limited to ext4, HFS+, and NTFS. The “grub.conf” automatically generated by “grub-mkconfig” and it help you to handle various Kernel easier. I'm sure you can remember “grub-mkconfig” in recent chapter and worked with it for generating your Kernel.
If you like to have a good knowledge about Grub2 then IBM has a good article about it that can be found at “http://www.ibm.com/developerworks/library/l-grub2/”. We recommended to see “https://blog.xenproject.org/2015/01/07/using-grub-2-as-a-bootloader-for-xen-pv-guests/” for more information about Grub and Xen.
We showed you some important parameters that you can use them via Grub and the good point is that the GNU Grub is part of Xen by default.
In below Figure, You can see a diagram about The Linux Boot process:
What do think? We showed you some important information about Boot and Xen Project and recommended to read the articles that introduced. In the article from Xen project, You can see and Learn how to install “pvgrub2” but this article may be removed or moved to other places in the future and I just show you the installing process :
1- Git grub :
# git clone git://git.savannah.gnu.org/grub.git
2- Compile :
# ./autogen.sh
3- Configure Grub to use Xen :
# ./configure --target=amd64 –with-platform=xen
(For 64 bit)
# ./configure --target=i386 –with-platform=xen
(For 32 bit)
4- make and install :
# make && make install
Now, Grub installed and you must create “pvgrub2” image :
1- create a file with the name “grub.cfg” and open it via “nano” editor and write below text to it :
normal (xen/xvda,msdos1)/boot/grub/grub.cfg
2- Build grub image using “grub-mkimage” utility :
# grub-mkimage -O x86_64-xen -c grub.cfg -o grub-x86_64-xen.bin $prefix/lib/grub/x86_64-xen/*.mod
3- Now, You can start a guest via “grub-x86_64-xen.bin” as a Kernel by writing below line in your guest configuration :
# kernel = "grub-x86_64-xen.bin"
