Xen Project Best Practices

Best Practices for Xen

This wiki page lists various best practices for running Xen hypervisor.

Information here applies to opensource Xen from xen.org, not necessarily to Citrix XenServer or XCP.

Xen dom0 dedicated memory and preventing dom0 memory ballooning

You should always dedicate a fixed amount of RAM for Xen dom0.

This can be done by specifying "dom0_mem=512M,max:512M" option for the Xen hypervisor (usually xen.gz) in grub.conf/menu.lst. This makes sure that the initial amount of memory allocated for dom0 is 512 MB (note: Replace this value with the amount of memory you want to allocate to dom0) and leaves the rest of the host system's RAM available for other guests. See this grub.conf example for GRUB1:

title Xen 4.1.0 / pv_ops dom0 kernel 2.6.32.36
        root (hd0,0)
        kernel /xen-4.0.gz dom0_mem=512M,max:512M loglvl=all guest_loglvl=all
        module /vmlinuz-2.6.32.36 ro root=/dev/sda2 console=hvc0 earlyprintk=xen nomodeset
        module /initrd-2.6.32.36.img

The next step is to configure the toolstack to make sure dom0 memory is never ballooned down while starting new guests:

• If you are using the XL toolstack this can be done by editing /etc/xen/xl.conf and setting autoballoon=0. This will prevent XL from ever automatically adjusting the amount of memory assigned to dom0.

• If you are using the xend toolstack this can be done by editing /etc/xen/xend-config.sxp and changing the "dom0-min-mem" option (to "dom0-min-mem 512") and changing the "enable-dom0-ballooning" option (to "enable-dom0-ballooning no"). These options will make sure xend never takes any memory away from dom0.

After making these changes to grub.conf and to xend-config.sxp/xl.conf, reboot the system. After reboot you will notice dom0 has only 512 MB of memory, and the rest of the RAM is available in Xen hypervisor as a free memory. You can run "xl list" (or "xm list") to verify the amount of memory dom0 has, and "xl info" (or "xm info") to verify the amount of free memory in Xen hypervisor.

Why should I dedicate fixed amount of memory for Xen dom0?

Dedicating fixed amount of memory for dom0 is good for two reasons:

First of all (dom0) Linux kernel calculates various network related parameters based on the boot time amount of memory.

The second reason is Linux needs memory to store the memory metadata (per page info structures), and this allocation is also based on the boot time amount of memory.

Now, if you boot up the system with dom0 having all the memory visible to it, and then balloon down dom0 memory every time you start up a new guest, you end up having only a small amount of the original (boot time) amount of memory available in the dom0 in the end. This means the calculated parameters are not correct anymore, and you end up wasting a lot of memory for the metadata for a memory you don't have anymore. Also ballooning down busy dom0 might have bad side effects.

Xen credit scheduler domain weights and making sure dom0 gets enough CPU time to serve IO requests (disk/net)

For smooth operation and good guest performance you need to make sure that dom0 always gets enough CPU time to process and serve the IO requests for guest VMs. This can be done by setting up the Xen credit scheduler domain weights and caps. See CreditScheduler wiki page for more information.

Some background: As a default Xen gives every guest (including dom0) the default weight of 256. This means all guests (including dom0) are equal, and get the same amount of CPU time. This can be bad for dom0, since it needs to be able to serve and process the IO requests for other guests. You should give dom0 more weight, so it will get more CPU time than the guests, when it needs it.

Example commands:

• use "xm sched-credit -d Domain-0" to check the current Xen credit scheduler parameters for dom0.
• use "xm sched-credit -d Domain-0 -w 512" to give dom0 weight of 512, giving it more (up to twice as much) CPU time than the guests.

Note that you need to apply this setting after every reboot! It's not persistent setting. You can place the "xm sched-credit" to rc.local or other script that is executed late in the boot process. Make sure the command is executed after xend is started, since the xm command needs to talk to xend.

For domUs just use the default weight (256), unless you have a reason to tweak them.

Dedicating a CPU core(s) only for dom0

If you're running IO intensive guests or workloads in the VMs it might be a good idea to dedicate (pin) a CPU core only for dom0 use. Please see XenCommonProblems wiki page section "Can I dedicate a cpu core (or cores) only for dom0?" for more information.

Dom0 network configuration

You should configure and set up networking on Xen dom0 using the networking scripts provided by your dom0 distribution, ie. "/etc/network/interfaces" on Debian and Ubuntu, and "/etc/sysconfig/network-scripts/ifcfg*" in RHEL/CentOS/Fedora. The "distro-method" is much better than using the xen "network-bridge" script, which is troublesome on many configurations due to renaming tricks it uses.

If using XL / LIBXL toolstack (Xen 4.1+) you're actually required to set up the networking using distro network scripts!

To use distro networking scripts/tools:

• If using xm/xend toolstack disable the Xen network-script in "/etc/xen/xend-config.sxp", ie. comment out the "network-script" line, or make it be "/bin/true".
• Configure network settings (bridges etc) using the networking scripts available on your dom0 distribution. See the documentation of your distro for more help.
• Reboot.

You can still use the default Xen vif-bridge script to attach VM vifs to the bridges you have configured using the distro networking tools, so no changes are required to the domU configuration files in "/etc/xen/<domU>".

See Network Configuration Examples (Xen 4.1+) wiki page for examples how to use the distro network scripts.

Common problems with Xen

Also check the XenCommonProblems wiki page for answers to many common problems related to using/running Xen.