Virtualization Spectrum

To Do: Merge with post that will be published later this week. Redefine Glossary to point to definitions on this page

At XenSummit 2012 in San Diego, Mukesh Rathor from Oracle presented his work on a new virtualization mode, called “PVH”. Adding this mode, there are now a rather dizzying array of different terms thrown about — “HVM”, “PV”, “PVHVM”, “PVH” — what do they all mean? And why do we have so many?

The reason we have all these terms is that virtualization is no longer binary; there is a spectrum of virtualization, and the different terms are different points along the spectrum. Part of the reason the terminology is a little unclear is the history; any language and terminology evolves over time in response to the changing situation. However, changing the terminology after-the-fact once certain usages become common is difficult.

So in this series of articles I will introduce just enough history to understand how the current situation came about, and (hopefully) introduce a consistent set of terminology which may help clear things up, while balancing this against the fact that people will still continue to use existing terminology.

This article will be divided into two parts. The first part will give a general introduction to virtualization, and to paravirtualization, Xen’s unique contribution to the field, as well as the advent of hardware virtualization extensions (HVM). It will also introduce the idea of adding paravirtualized drivers for disk and network. The second mode will cover the motivation and technical descriptions of two more modes which further mix elements of full virtualization and paravirtualization.

Full virtualization

In the early days of virtualization (at least in the x86 world), the assumption was that you needed your hypervisor to provide a virtual machine that was functionally nearly identical to a real machine. This included the following aspects:

• Disk and network devices
• Interrupts and timers
• Emulated platform: motherboard, device buses, BIOS
• “Legacy” boot: i.e., starting in 16-bit mode and bootstrapping up to 64-bit mode
• Privileged instructions
• Pagetables (memory access)

In the early days of x86 virtualization, all of this needed to be virtualized: disk and network devices needed to be emulated, as did interrupts and timers, the motherboard and PCI buses, and so on. Guests needed to start in 16-bit mode and run a BIOS which loaded the guest kernel, which (again) ran in 16-bit mode, then bootstrapped its way up to 32-bit mode, and possibly then to 64-bit mode. All privileged instructions executed by the guest kernel needed to be emulated somehow; and the pagetables needed to be emulated in software.

This is mode, where all of the aspects the virtual machine must be functionally identical to real hardware, is what I will call fully virtualized mode.

Xen and paravirtualization

Unfortunately, particularly for x86, virtualizing privileged instructions is very complicated. Many instructions for x86 behave differently in kernel and user mode without generating a trap, meaning that your options for running kernel code were to do full software emulation (incredibly slow) or binary translation (incredibly complicated, and still very slow).

The key question of the original Xen research project at Cambridge University was, “What if instead of trying to fool the guest kernel into thinking it’s running on real hardware, you just let the guest know that it was running in a virtual machine, and changed the interface you provide to make it easier to implement?” To answer that question, they started from the ground up designing a new interface designed for virtualization. Working together with researchers at both the Intel and Microsoft labs, they took both Linux and Windows XP, and ripped out anything that was slow or difficult to virtualize, replacing it with calls into the hypervisor (hypercalls) or other virtualization-friendly techniques. (The Windows XP port to Xen 1.0, as you might imagine, never left Microsoft Research; but it was benchmarked in the original paper.)

The result was impressive — by getting rid of all the difficult legacy interfaces, they were able to make a fast, very lightweight hypervisor in under 70,000 lines of code.

This technique of changing the interface to make it easy to virtualize they called paravirtualization. In a paravirtualized VM, guests run with fully paravirtualized disk and network interfaces; interrupts and timers are paravirtualized; there is no emulated motherboard or device bus; guests boot directly into the kernel in the mode the kernel wishes to run in (32-bit or 64-bit), without needing to start in 16-bit mode or go through a BIOS; all privileged instructions are replaced with paravirtualized equivalents (hypercalls), and access to the page tables was paravirtualized as well.

Xen and full virtualization

In early versions of Xen, paravirtualization was the only mode available. Although Windows XP had been ported to the Xen platform, it was pretty clear that such a port was never going to see the light of day outside Microsoft Research. This meant, essentially, that only open-source operating systems were going to be able to run on Xen.

At the same time the Xen team was coming up with paravirtualization, the engineers at Intel and AMD were working to try to make full virtualization easier. The result was something we now call HVM — which stands for “hardware virtual machine”. Rather than needing to do software emulation or binary translation, the HVM extensions do what might be called “hardware emulation”.

Technically speaking, HVM refers to a set of extensions that make it much simpler to virtualize one component: the processor. To run a fully virtualized guest, may other components still need to be virtualized. To accomplish this, the Xen project integrated qemu to emulate disk, network, motherboard, and PCI devices; wrote the shadow code, to virtualize the pagetables; wrote emulated interrupt controllers in Xen; and integrated ROMBIOS to provide a virtual BIOS to the guest.

Even though the HVM extensions are only one component of making a fully virtualized VM, the “fully virtualized” mode in the hypervisor was called HVM mode, distinguishing it from PV mode. This usage spread throughout the toolstack and into the user interface; to this day, users generally speak of running a VM in “PV mode” or in “HVM mode”.

Once you have a fully-virtualized system, the first thing you notice is that the interface you provide for network and disks — that is, emulating a full PCI card with MMIO registers and so on — is really unnecessarily complicated. Because nearly all modern kernels have ways to load third-party device drivers, it’s a fairly obvious step to create disk and network drivers that can use the paravirtualized interfaces. Running in this mode can be called fully virtualized with PV drivers.

Summary

I have introduced the concepts of full virtualization and paravirtualization (PV), as well as the hardware virtualization (HVM) feature used by Xen (among other things) to implement full virtualization. I have also introduced the concept of installing paravirtualized drivers on a fully virtualized system.

This small step, from full virtualization towards paravirtualization, begins to hint at the idea of a spectrum of paravirtualization. In part 2, I will cover the historical reasons for the development of PVHVM, and finally of the newest mode, PVH.