Archived/Kernel.org Linux on Xen

To Do: Suggestion: some of this page should be merged with: XenParavirtOps Alternatively, one of the pages could be used for the high level overview, theory, and quick status and the other could be used for the "howto"-style using it.

Running kernel.org "mainline" Linux on Xen

Since release 2.6.23, Linus's "mainline" Linux releases at kernel.org have included support for running as a Xen domU via the "paravirt-ops" framework. This is a different framework to that used by the XenLinux supplied by the Xen project and so the process of building and running it is slightly different. The paravirt-ops framework is likely to be used for XenLinux in future but for now this page is intended to provide additional guidance for those using the kernel.org version of Xen support.

This document has been developed against Linux 2.6.24, however it will probably work (perhaps with minor changes) for 2.6.23 and also for subsequent releases. Please report any mistakes or omissions to mark.williamson@cl.cam.ac.uk and / or make fixes to this page as appropriate.

kernel.org vs XenLinux

UPDATE 2010-01-14: This page is somewhat outdated now. Please see [[[XenParavirtOps]]] and [[[XenDom0Kernels]]] wiki pages for more information.

Because they are based on different frameworks, kernel.org Linux and XenLinux currently have somewhat different featuresets. In general, XenLinux has more powerful and complete Xen-related functionality but kernel.org Linux supports more recent Linux functionality due to being based on a more recent Linux release. Some key differences are:

• dom0 support - only XenLinux currently supports use as dom0 on a Xen system. kernel.org pv_ops Linux may only be used within a domU. Pv_ops dom0 work is underway.
• 64-bit support - Both XenLinux and kernel.org pv_ops Linux support running as a 64-bit Xen guest. Recent versions of Xen can run paravirtualized 32-bit (PAE) guests under a 64-bit hypervisor.
• framebuffer support - Both XenLinux and kernel.org pv_ops Linux support the virtual framebuffer device
• suspend/resume and live migration - Both XenLinux and kernel.org pv_ops Linux support these operations
• kernel version - XenLinux is based on Linux 2.6.18, kernel.org supports Xen domU operation since version 2.6.23 and will continue to do in subsequent releases
• native boot - kernel.org Linux may boot directly on hardware or within a Xen domU. XenLinux can only boot on top of Xen and cannot be used as a native kernel.
• support for other virtualization platforms - kernel.org Linux's paravirt-ops framework allows the same kernel to also boot on the lguest hypervisor and using VMware's VMI interface. XenLinux does not support these platforms.

XenLinux and kernel.org Linux are configured and built differently and require different Xen configuration file entries in order to work correctly. The procedure for building and booting a kernel.org Linux on Xen is described below.

Xen dom0 support / available dom0 kernels

Please see http://wiki.xensource.com/xenwiki/XenDom0Kernels for a list of available Xen dom0 capable Linux kernels.

Building kernel.org Linux with Xen domU support

Your kernel config should enable the following options to get Xen domU support:

CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_HVC_XEN=y

The CONFIG_XEN option depends on a sufficiently new x86 processor architecture. Configure Linux with a Pentium Pro or newer CPU if you want to use Xen; Xen does not support older CPUs anyway.

nb you will probably also want to enabled CONFIG_HIGHMEM_64G, which enables PAE addressing support. This is likely to be needed, since PAE mode has been the default for 32-bit guests under Xen for some time. Unless you have a specific reason not to choose this, you probably want it switched on.

If you are using an kernel configurator, such as menuconfig or xconfig then you will find "CONFIG_PARAVIRT_GUEST" as Processor type and features -> Paravirtualized guest support and "CONFIG_XEN" as Processor type and features -> Paravirtualized guest support -> Xen guest support. Activating Xen support will automatically select the other Xen-related config items. "CONFIG_HIGHMEM_64G" can be found under Processor type and features -> High Memory Support -> 64GB.

When building a kernel to run as a domU it is not necessary to include any device drivers for the host system's real hardware. The real hardware is hidden by the Xen abstraction layer. However, you may still wish to select some device drivers if you also want to boot this kernel directly on your machine (not under Xen). If the kernel is only for use under Xen it is possible to slim down the build process considerably by removing unwanted device drivers.

Having configured the kernel for use on Xen, you should start the build process in the usual way:

make

this will build the kernel from the configuration you specified. A bzImage will be generated, which you can use if you want to boot this kernel directly on the host system. If you are running Xen 3.3 or earlier version, you can not use the bzImage to boot the kernel under Xen as it is not currently a supported format for the Xen tools. Instead, you need to use the vmlinux image, which should be in the root of your kernel tree. This contains the same kernel as the bzImage but with less packaging around it. However, the vmlinux is initially very large because it contains all the kernel symbols. To reduce it to a manageable size, you can run:

strip vmlinux -o vmlinux-stripped

Which should make a much smaller image called vmlinux-stripped. You can use this image in your Xen config file, or you can make it still smaller by compressing it:

gzip vmlinux-stripped -c > vmlinuz

the vmlinuz file is the same as vmlinux but is gzipped compressed. This is also suitable for use in your Xen config file.

If you have Xen 3.4 or newer version, then you can use bzImage dom0 kernels without problems.

Running kernel.org Linux in a Xen domU

Once you have built a suitable kernel image to boot under Xen, you may want to build a Xen configuration file in order to boot that kernel. This file has the same basic syntax as any other Xen configuration file but there are a few points to note when filling out the values of some parameters:

• kernel.org uses xvd* device nodes to specify Xen block devices, so your disks= and root= parameters should use this format too (*unless* you are using the LABEL= or UUID= format for root devices, in which case your root= option should work-as is).
• kernel.org uses /dev/hvc0 as the Xen console device, so include "console=hvc0" in the extra= section of your Xen config in order to get boot output
• If you wish to log in on the Xen console, you should also modify inittab to start a getty on /dev/hvc0 and modify /etc/securettys to refer to it, so that you can log in as root.

If you are booting the domain using pygrub then you still need to make sure that the disks are specified using the correct device names in the domain config file. The root= parameter on your kernel command line needs to be set appropriately. You'll need to add "console=hvc0" to the kernel command line in order to get boot output from the domain. Once this is done, you should simply be able to choose the relevant grub entry at boot time and watch your domain start up as usual.

Other notes

• The ability to boot a bzImage directly in Xen has not been available thusfar because it required some tricky alterations to the bzImage format. Patches to do this are now being tested and are expected to be included in a future release of Linux. Support for loading bzImage kernels should be available in Xen releases by this time. http://lists.xensource.com/archives/html/xen-devel/2008-03/msg01022.html .
• Xen 3.4.x and newer versions support booting bzImage dom0 kernels.
• Xen 3.2.x and newer version support loading bzImage domU kernels.
• Xen 3.4.2 and newer versions do have support for BZIP2 and LZMA compressed PV domU/guest kernels, assuming the required bzip2 and lzma (xz) libraries and development headers have been available at the Xen compilation time.
• Developers from Red Hat are helping in a push to add pv-ops support for dom0. This will enable the same kernel to be booted on bare metal, as Xen dom0 and as a Xen domU. The plan is to use this kernel for Xen support in Fedora. http://fedoraproject.org/wiki/Features/XenPvops