PvGrub

PV-GRUB (ParaVirtual Grub) is a boot manager for Xen virtual machines. It is a safer (as in more reliable but specifically as in more secure - see the technical details section) and more efficient alternative to PyGrub to boot domU images: unlike pygrub it runs an adapted version of the grub boot loader inside the created domain itself, and uses the regular domU facilities to read the disk mounted as root directory, fetch files from network, etc.; it also eventually loads the PV kernel and chain-boots it.

This page talks mainly about the PV port of Grub 1.0 (sometimes called grub-legacy) which is part of the Xen source tree. On modern systems it is likely to be be more appropriate/convenient to use the PV port of Grub 2 which is part of the main grub2 source since grub2 is used by many of the common distros.

Notes

• PvGrub requires Xen 3.3 or greater.
• You won’t actually need to install grub in the domU, it is implemented as a stub domain and provided with the Xen sources.
• On certain GNU/Linux distributions (Debian, for example), the pvgrub.gz file isn't packaged. As a result, it is not possible to directly use the PvGrub loader in these GNU/Linux distributions. As a workaround, you can build it from the sources (see build subsection).

Technical details

PVGRUB, as mentionned in the introduction, is an adapted Grub bootloader built for Xen. Since it uses the codebase of a widely used software (i.e. GRUB), it should be less prone to bugs than its older alternative, PyGrub. Also, since it is not scripted but a complied binary, it is probably more efficient and has less dependencies (at least for python). Lastly; since the bootloader is designed to run as a ParaVirtualized loader in the DomU environment rather than to perform risky operations (such as copying a file from a potentially non-trusted virtual disk on the Dom0) as PyGrub does; it is also way more secure.

How To

Inspired from backdrift.org/xen-pvgrub-howto:

So if you’re like me you are probably somewhat lazy. I imagine most sysadmins are. Which must be why we put so much effort into scripting and automating repeated tasks. If you’ve managed a Xen VM before you are likely to notice that all your kernel images live outside the VM itself. This makes it rather difficult to patch and upgrade among other things, especially as the number and type of guests you support grows. Pvgrub can help solve this problem. It implements grub in a stub domain which in turn loads your guest domain. It works quite well and is rather easy to set up. Read on to learn more.

From the xen wiki: “PV-GRUB replaces PyGrub to boot domU images safely: it runs the regular grub inside the created domain itself and uses regular domU facilities to read the disk / fetch files from network etc.; it eventually loads the PV kernel and chain-boots it.”

This means that your domU guests can install and manage their own kernels as if they were running on regular hardware. Yum, apt-get and other package management software will let you patch your virtual machine’s kernel without the need to alter any Xen configuration. Here’s how it works.

Build

The following example is for building pv-grub on a stock GNU/Linux Debian 6 (squeeze)

#!/bin/sh

apt-get install git
#for the configure checks
apt-get install python-dev gettext bin86 bcc iasl uuid-dev libncurses5-dev pkg-config libglib2.0-dev libyajl-dev
#not checked by the configure script
apt-get install libpixman-1-dev bzip2
git clone git://xenbits.xen.org/xen
cd xen
./configure
make

you might also lack the following package (dependency for some versions of Xen, for the bits/predefs.h file):

libc6-dev-i386

Then you can find the pv-grub in two architectures (32 and 64 bits) as gzip archives in the xen/dist/install/usr/lib/xen/boot/ directory

Configure

Boot domU from local disk

kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0,0)/grub/menu.lst"

This example is assuming that you have a boot partition (mounted at /boot in your /etc/fstab) as the first partition of the first hard drive; and that this partition contains a grub directory containing a menu.lst file.

Boot domU from network

kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(nd)/grub/menu.lst"

You should setup the DHCP server and TFTP server correctly. Create ${tftproot}/grub/menu.lst; copy necessary files (vmlinuz, initrd, etc.).

Notes

See Troubleshooting/Tips.

Modify /etc/modprobe.conf in the guest

We need to make sure that a couple of lines exist in /etc/modprobe.conf before we’re ready to install the kernel packages. We add these so that when the kernel install process goes to build a new initrd it will include the correct driver modules for your hardware, so we want to be certain that the Xen block and net drivers are included.

Note: If /etc/modprobe.conf doesn’t already exist you should create it.

#/etc/modprobe.conf
alias eth0 xennet
alias scsi_hostadapter xenblk

Update /etc/inittab and /etc/securetty in the guest

Now we update /etc/inittab and /etc/securetty to make certain that we attach to and permit logins from the various xen consoles.

/etc/inittab

#/etc/inittab
hvc0:2345:respawn:/sbin/getty 38400 hvc0
xvc0:2345:respawn:/sbin/getty 38400 xvc0

/etc/securetty

#/etc/securetty
# hypervisor virtual consoles
hvc0
xvc0

Install a stub grub.conf in the guest so that the grub update scripts won’t fail

Configuration file locations:

Depending on the Linux distribution you use you’re likely to see the grub config in either /boot/grub/grub.conf or /boot/grub/menu.lst. To keep things simple I usually use /boot/grub/grub.conf as a config file and symlink /boot/grub/menu.lst to it. This should ensure that scripts used to automatically update the grub config when a new kernel is installed will work correctly.

Reasonable stub for Debian Lenny

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.26-2-xen-686
   root (hd0)
   kernel /boot/vmlinuz-2.6.26-2-xen-686 console=hvc0 root=/dev/xvda1 ro
   initrd /boot/initrd.img-2.6.26-2-xen-686

Reasonable stub for CentOS/RHEL

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.18-128.4.1.el5xen
	root (hd0)
	kernel /boot/vmlinuz-2.6.18-128.4.1.el5xen root=LABEL=root ro
	initrd /boot/initrd-2.6.18-128.4.1.el5xen.img

Install kernel packages in the guest

Debian

# apt-get install linux-image-2.6.26-1-686-bigmem

CentOS/RHEL

# yum install kernel-xen

Modify Xen guest config file in the dom0

Now that the VM has been prepped for booting with pvgrub we need to update its configuration file to load the pvgrub kernel instead of a linux kernel.

Note: be certain to remove the old “root=” line, this seems to confuse pvgrub. Also, make certain that the “extra=” line points to the location of your grub config file inside the guest. My example does not use partitions, if you do use partitions in your guests you’ll need so use something like (hd0,0), etc.

name    = 'superchunk'
 
vcpus =2;
 
memory 	= 512;
maxmem  = 4096;
 
#pvgrub here
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0)/boot/grub/grub.conf"
#end pvgrub
 
disk = [ 'phy:/dev/vg0/vm_superchunk-root,xvda1,w', \
         'phy:/dev/vg0/vm_superchunk-swap,xvda2,w' ];
 
vif = [ 'bridge=eth1, vifname=superchunk, mac=B0:00:00:00:B1:35' ];

Shutdown and then create the domain

My Xen guest is named superchunk. You should be familiar with shutdown and create at this point. Here’s how, just in case…

xm shutdown superchunk
#(wait until it has shut down)
xm create -c /path/to/superchunk-config

At this point you should see some kernel output and then either a grub menu or the grub command line. If you get the command line, no worries. This usually means that it couldn’t find a grub configuration file. Make sure that the “extra=” line of the Xen guest domU configuration file references the right location and that you have pointed grub at the correct disk, i.e. (hd0) vs. (hd0,0), etc…

Good luck! And feel free to drop me a line at soundsoldier@backdrift.org, I always appreciate feedback and do my best to help.

Troubleshooting

Tips

• There shall be no root parameter in the DomU configuration file if you use pvgrub. If there is one, it will likely be prepended to the cmdline; rendering the extra inoperant. In this case, you will be dropped to a grub shell instead of having a grub menu.
• The content of the usual root parameter of the DomU configuration file should be appended to the kernel line in the menu.lst file; in the DomU file system.

Links

Since pvgrub corresponds to GRUB legacy, it is not supported anymore by the GRUB developers. Until there is a pvgrub2, one should use the following resources to get help and solve problems:

• GNU GRUB Manual 0.97
• The list of command-line and menu entry commands

References

• Xen PVGRUB Howto: http://backdrift.org/xen-pvgrub-howto
• GNU GRUB Manual.
• Xen PXE Boot Howto.
• Repair Linux Boot failures in Grub 2 rescue mode