PvGrub2

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0) administrator to explicitly supply a kernel (and perhaps initial ramdisk) from the domain 0 filesystem in order to start a new guest.

This mostly worked and for some use cases, i.e. those where the host admin wants very strict control over what each guest runs, was desirable and remains so today.

However for other use cases it was rather inflexible since it meant that the host administrator needed to be involved in what many considered to be a guest administrator, or even distribution level, decision i.e. the selection of which kernel to run, with what parameters etc.

The first solution to this problem to come along was pygrub. pygrub is an application (written in Python) which can be used by the Xen toolstack in domain 0 as a kind of pseudo-bootloader. pygrub will open the guest file system (using a userspace filesystem access library), extract a configuration file, parse it and extract the referenced kernel, (optional) initial ramdisk and kernel command line to be booted almost as if they were provided from the domain 0 filesystem.

pygrub initially supported configuration files in the GNU grub (known as "grub-legacy" by upstream today) menu.lst syntax, which was a common option for distributions at the time. pygrub even supported an optional curses frontend menu similar to the native grub legacy interface, allowing boot time selection between multiple kernels, as well as editing of the command line etc.

This allowed host admins configure a guest to use pygrub and thereby delegate the management and selection of the guest kernel to the guest administrator. Guest administrators could use the usual tools which they expect (i.e. distribution packaging and grub integration) and configuration file syntax from their non-Xen systems.

Since it was introduced pygrub has gone from supporting the grub-legacy menu.lst configuration files to supporting a variety of configuration files including the syntaxes of GNU grub 2, [syslinux][syslinux] (which includes pxelinux, syslinux, and isolinux) and LiLo. pygrub remains part of the Xen releases today (and will be for the foreseeable future) however it has some short comings:

• The configuration file parsers are simplistic and cannot cope with all of the constructs which can be used in practice. In particular the grub 2 syntax is a particularly expressive shell-like language which pygrub can only cope with basic elements of and that support is fragile requiring updates as grub-upstream and distributions find new ways to make use of the flexibility allowed. This is particularly troublesome now that grub 2 is the default in many distributions.
• It ultimately boots a potentially untrusted guest kernel as if it was a kernel supplied from the domain 0 filesystem (which would normally be somewhat implicitly trustworthy). This means that the code to build a domain now must take special care to treat the kernel image as hostile.

As a result of these shortcomings pvgrub was created (it's an unfortunate source of much confusion that pygrub and pvgrub differ only in the descender on a single letter). pvgrub was a port of GNU grub (AKA "grub-legacy") to run as a Xen PV kernel. This had several advantages:

• Since the pvgrub kernel is supplied by the host administrator it is trusted and can be expected not to deliberately attack the domain builder. All handling of the untrusted guest inputs now happens within guest context where the harm which can be done is contained.
• Since the code is actually real grub-legacy it has the same configuration file parser and features as grub running on a native system.

One minor downside of pvgrub is that it did not support syslinux or LiLo configuration files (that would need to be achived via a PV port of those respective bootloaders), although in practice they are not so widely used so this was a minor shortcoming.

A second shortcoming was that it is not possible for a PV guest to switch between 32- and 64-bit operation. This means that the user needs to know a-priori the type of kernel which they will be booting and the host administrator needs to provide the ability to select between 32- and 64-bit builds of pvgrub.

The most serious problem today though is the move of most distributions from grub-legacy to grub 2, with its radically different architecture and configuration file syntax. This meant that admins could no longer simply reuse their existing grub 2 based workflows and distribution integration. Some workarounds have evolved such as pv-grub-menu but something better was needed...

PV Grub 2

In November 2013 upstream grub maintainer Vladimir 'phcoder' Serbinenko announced that:

pvgrub2 has just became part of upstream grub as ports i386-xen and x86_64-xen.

This meant that it was now possible to compile the upstream grub 2 code base to run as a pvgrub2 Xen PV guest, in much the same way as the original pvgrub-legacy port of Grub legacy.

This has the same advantages as the pvgrub-legacy port originally had, except using the more modern grub 2 code base. In addition since this support was part of upstream grub there was no fork to maintain and therefore no risk that the Xen PV support would languish.

In the remainder of this blog post I'm going to explain how to install and use pvgrub2 on your Xen hosts and guests.

Guest Setup

This guide assumes that the PV guest is already setup with a grub2 configuration file, as if it were a native system (i.e. usually /boot/grub/grub.cfg). Many distribution installers (at least those for distributions which use grub2) will do this automatically even within a PV guest. If not then you may need to install manually, e.g. on Debian by installing the grub-pc package.

Building and installing pvgrub2

Getting the source

The last release of grub was 2.00, released in June 2012, which is before PV Xen support was added. Since then the grub development team have released beta versions of grub 2.02. In particular the latest, 2.02~beta2, contains support for Xen. (Don't be scared off by the beta tag, in reality several distributions are shipping this version as their primary bootloader).

Grub 2.02~beta2 can be downloaded from http://alpha.gnu.org/gnu/grub/grub-2.02~beta2.tar.gz.

Alternatively you can fetch the code from the grub git repository:

   $ git clone git://git.savannah.gnu.org/grub.git

Compiling

The file INSTALL in the code tree contains detailed information on how to build grub, including a full list of dependencies.

Other than the obvious things, such as make and gcc and slightly less obvious things such as flex and bison it is worth noting that:

• compiling 64-bit grub on a 64-bit platform needs to be able to build some 32-bit components and therefore requires a biarch capable gcc (i.e. one which accepts the -m32 option, pretty much all 64-bit distribution gcc packages today include this feature in the default compiler) as well as a 32-bit libc (e.g. from the libc6-dev-i386 package on Debian).
• the Xen headers must be installed, if you've installed Xen from distribution packages this might require you to install the relevant -dev package (e.g. libxen-dev in Debian). If you've installed Xen from source with make install then you should have the headers already.

The process is pretty much the standard configure/make/make install routine associated with all autoconf based projects.

If you are compiling from git then you will need to start by generating the configure script. This can be skipped if you are using the tarball.

   $ ./autogen.sh

Next, configure grub2 for use on an x86_64-xen platform:

   $ ./configure --target=amd64 --with-platform=xen

To target 32-bit/i386 Xen domains instead use:

   $ ./configure --target=i386 --with-platform=xen

The rest of this guide will assume x86_64-xen, but you can substitute i386-xen throughout if you want to boot a 32-bit guest.

If you want to avoid the possibility of messing with the native bootloader on the system then add --prefix=/opt/grub2 to the configure parameters in order to install pvgrub2 in an out of the way location (if you do this then you may want to add /opt/grub2/sbin:/opt/grub2/bin to your $PATH or else remember to give an explicit path to the relevant commands).

Once things are configured then:

   $ make

Finally, as root:

   # make install

Creating a basic pvgrub2 image

Now that grub is installed on the host the next step is to build the actual pvgrub2 image which will be used to boot the guest. Grub is highly modular and allows you to construct images with various features embedded. It also supports loading additional modules at runtime.

To create a basic image first create a file named grub.cfg which contains:

   normal (xen/xvda,msdos1)/boot/grub/grub.cfg

This tells grub to read the /boot/grub/grub.cfg configuration file from the first partition of the xvda device (which must be using MSDOS style partitioning) and run it using the normal command interpreter.

Finally we can build the grub image using the grub-mkimage utility:

    grub-mkimage -O x86_64-xen -c grub.cfg \
        -o grub-x86_64-xen.bin $prefix/lib/grub/x86_64-xen/*.mod

Where:

• -O x86_64-xen: Is the target platform
• -c grub.cfg: Includes the configuration file which created above
• -o grub-x86_64-xen.bin: Is the output file
• $prefix/lib/grub/x86_64-xen/*.mod: Includes all loadable modules in the image. ($prefix is wherever you installed grub to, which is /usr/local by default, or /opt/grub2 if you added the --prefix=/opt/grub2 option as discussed above).

The reason for the include *.mod is that since this image is going to running in guest context it is not going to be able to load additional modules from domain 0 at runtime, since it will only see the guest filesystem.

Now you can start a guest using grub-x86_64-xen.bin as the kernel. e.g. by writing in your guest configuration file:

   kernel = "grub-x86_64-xen.bin"

This is all that is required. There is no need to give any other ramdisk, bootloader, cmdline, root, extra etc options. You will still need to configure the name, disks, network devices etc in the normal way.

Then, assuming your in-guest grub.cfg is indeed located at (xen/xvda,msdos1)/boot/grub/grub.cfg, you should be presented with the usual grub menu when you connect to the guest console.

Loading grub.cfg from any partition

The above simple example is all well and good, but it is rather inflexible and if the guest uses a separate /boot partition or otherwise differs from the expectations encoded in the initial configuration then it won't find the real configuration file and you will be dumped to a grub prompt.

Luckily the grub configuration file syntax is far richer than we've used so far, so lets try something more flexible.

As well embedding a bootstrap configuration grub-mkimage is also able to embed a memdisk (essentially an initramfs for the bootloader) into the pvgrub2 image, we are going to use this in order to bootstrap into a more capable grub shell.

First we need to create two configuration files which will be embedded into the pvgrub2 image:

grub-bootstrap.cfg:

   normal (memdisk)/grub.cfg

grub.cfg:

   if search -s -f /boot/grub/grub.cfg ; then
       echo "Reading (${root})/boot/grub/grub.cfg"
       configfile /boot/grub/grub.cfg
   fi
   
   if search -s -f /grub/grub.cfg ; then
       echo "Reading (${root})/grub/grub.cfg"
       configfile /grub/grub.cfg
   fi

The reason for using two configuration files in this manner is that the inbuilt grub interpreter is rather simple, whereas features such as if and search are only available from the more feature complete normal interpreter. Bootstrapping into this more complex grub.cfg allows us to cope with guests which have a separate /boot partition by searching for a file named /boot/grub/grub.cfg or /grub/grub.cfg on any partition. Note that ${root} is set by the search command and should be entered literally, not substituted while creating this file.

Next we need to embed the grub.cfg into a memdisk, which is really just a tarball:

   $ tar cf memdisk.tar grub.cfg

Finally we can build the grub image using the grub-mkimage utility:

   $grub-mkimage -O x86_64-xen \
       -c grub-bootstrap.cfg \
       -m memdisk.tar \
       -o grub-x86_64-xen.bin \
       $prefix/lib/grub/x86_64-xen/*.mod

Where in addition to the example above we now use:

• -c grub-bootstrap.cfg: Includes the bootstrap configuration
• -m memdisk.tar: Includes our memdisk, which includes our more featureful configuration snippet.

On booting this grub image will now try much harder to find a grub.cfg to run and will work on far more guests without special tweaking.

Chainloading guest pvgrub2 from domain 0 pvgrub2

The above works well in many situations but has one major drawback which is that the bootloader is still ultimately under the control of the host admin. This may present a problem for example if the guest admin wishes to run a guest which makes use of new features found in a newer version of grub, or if there is a need to load a module which is not included in the host grub image from the guest filesystem, since the module may not be compatible with the running grub.

To address this issue the Xen team has published the Xen x86 PV Bootloader Protocol specification which describes the in-guest path where a PV bootloader, such as pvgrub2, should be installed in order that a host grub can chainload it. Specifically the bootloader should be installed to either /boot/xen/pvboot-i386.elf or /boot/xen/pvboot-x86_64.elf depending on the guest bit size.

Support for this protocol can be implemented in the host grub using the same grub-bootstrap.cfg + memdisk.tar method as above and a grub.cfg which contains:

   if search -s -f /boot/xen/pvboot-x86_64.elf ; then
       echo "Chainloading (${root})/boot/xen/pvboot-x86_64.elf"
       multiboot "/boot/xen/pvboot-x86_64.elf"
       boot
   fi
   
   if search -s -f /xen/pvboot-x86_64.elf ; then
       echo "Chainloading (${root})/xen/pvboot-x86_64.elf"
       multiboot "/xen/pvboot-x86_64.elf"
       boot
   fi

On the guest side you can build and install a grub within the guest just like for the host. Then run, as root, within the guest:

   # grub-install --target=x86_64-xen
   Installing for x86_64-xen platform.
   grub-install: warning: no hints available for your platform. Expect reduced performance.
   grub-install: warning: WARNING: no platform-specific install was performed.
   Installation finished. No error reported.

(the two warnings can safely be ignored, there is no actual performance impact)

Then the Grub image needs to be moved to the standardised location:

   # mkdir /boot/xen/
   # cp /boot/grub/x86_64-xen/core.elf /boot/xen/pvboot-x86_64.elf

Alternatively you can apply a patch (submitted upstream but not yet applied) which causes grub-install to do the right thing by default and rebuild grub-mkimage on your host.

For maximum flexibility you can combine both the chainloading and loading the guest grub.cfg directly by putting both options in the host grub's memdisk grub.cfg, I recommend trying to chainload first and only then falling back to loading a grub.cfg from the guest with:

   if search -s -f /boot/xen/pvboot-x86_64.elf ; then
       echo "Chainloading (${root})/boot/xen/pvboot-x86_64.elf"
       multiboot "/boot/xen/pvboot-x86_64.elf"
       boot
   fi
   
   if search -s -f /xen/pvboot-x86_64.elf ; then
       echo "Chainloading (${root})/xen/pvboot-x86_64.elf"
       multiboot "/xen/pvboot-x86_64.elf"
       boot
   fi
    
   if search -s -f /boot/grub/grub.cfg ; then
       echo "Reading (${root})/boot/grub/grub.cfg"
       configfile /boot/grub/grub.cfg
   fi

   if search -s -f /grub/grub.cfg ; then
       echo "Reading (${root})/grub/grub.cfg"
       configfile /grub/grub.cfg
   fi

Chainloading from pvgrub-legacy

What about all those host systems which provide only the legacy PV grub by default, as is common in some cloud environments? Fortunately it is possible to chainload pvgrub2 from pvgrub-legacy. Simply create a grub-legacy configuration file in your guest at /boot/grub/menu.lst which contains:

   default 0
   timeout 1
   title Chainload grub2
   kernel (hd0,0)/boot/xen/pvboot-x86_64.elf

You will need to adjust (hd0,0)/boot/xen/pvboot-x86_64.elf to suit your actual partition layout (sadly I don't know of any useful tricks to find it automatically with grub-legacy).

This will cause the host provided pvgrub-legacy to chainload into a guest supplied pvgrub2.

Distribution Integration

The above describes how to go about using pvgrub2 manually. Ideally this would all happen automatically as part of the standard distribution setup.

Debian

The Debian 8.0 (Jessie) release contains support for both host and guest pvgrub2. This was added in version 2.02~beta2-17 of the package (bits were present before then, but -17 ties it all together as described below).

The package grub-xen-host contains grub binaries configured for the host, these will attempt to chainload an in-guest grub image (following the specification and falling back to search for a grub.cfg in the guest filesystems as described above). grub-xen-host is Recommended by the Xen meta-packages in Debian or can be installed by hand.

The package grub-xen-bin contains the grub binaries for both the i386-xen and x86_64-xen platforms, while the grub-xen package integrates this into the running system by providing the actual pvgrub2 image (i.e. running grub-install at the appropriate times to create an image tailored to the system) and integration with the kernel packages (i.e. running update-grub at the right times), so it is the grub-xen which should be installed in Debian guests.

At this time the grub-xen package is not installed automatically so it will need to be done manually (something which perhaps could be addressed for Stretch).

Others

I'm not aware of any other distributions which have integrated grub2 support for Xen. I'd be glad to be corrected or equally happy to help advise on any integration efforts.

References

• Using Grub 2 as a bootloader for Xen PV guests