Difference between revisions of "Xen Project Beginners Guide"

From Xen
m (Creating a Windows HVM Guest: expand acronym in title)
(Changes to Configuring Grub, Connecting to a GUI guest, eliminated some dated referneces to xm and qemu-traditional, other minor edits)
 
(149 intermediate revisions by 15 users not shown)
Line 1: Line 1:
 
Welcome!
 
Welcome!
   
This guide was written to introduce beginners to basic Xen concepts and allow you to get started with Xen with no prior knowledge. Some prior Linux experience is required however, knowledge of networking, lvm and grub will go a long way!
+
This guide was written to introduce beginners to basic Xen Project concepts and allow you to get started with Xen Project with no prior knowledge. Some prior Linux experience is required however, and some knowledge of networking, lvm and grub will go a long way! You will need to use root-level access at various points, so some experience with that (likely through sudo) is necessary.
   
By completing this guide you will have installed a fully functional Xen hypervisor and started your first guest operating systems, connected them to your network and have been introduced to fundamental concepts such as virtual machine storage and virtual networking.
+
By completing this guide you will have installed a fully functional Xen Project hypervisor and started your first guest operating systems, connected them to your network and have been introduced to fundamental concepts such as virtual machine storage and virtual networking.
To make this process easy we will be using a Linux distribution called Debian. Debian’s current stable release “Squeeze” ships with support for Xen 4 and everything you need to get started!
 
   
  +
To make this process easy we will be using a Linux distribution called ''Debian''. This document was originally written for use with Debian 6/7 (called "Squeeze" and “Wheezy”) and has been tested with newer releases through Debian 10 ("Buster"). It should continue work with future releases as well. Debian ships with support for Xen Project release 4.x, providing everything you need to get started!
Though this guide looks long at first don’t be daunted, it is very indepth and comprehensive but doesn’t expect you to know all that much beforehand. The goal instead is to teach you all the things you need to know to build a functioning Xen hypervisor. :)
 
   
  +
Though this guide looks long at first, don’t let it daunt you. It is very in-depth and comprehensive, and doesn’t expect you to know all that much beforehand. The goal instead is to teach you all the things you need to know to build a functioning Xen Project Hypervisor. :)
=What is Xen all about?=
 
  +
<!---
  +
Not everything in this guide applies to XCP (Xen Cloud Platform). If you are looking for help with XCP, also see [[XCP]] and [[XCP_Command_Line_Interface]].
  +
--->
  +
=What is this Xen Project software all about?=
   
Xen is a Virtual Machine Monitor (VMM) also known as a hypervisor, this is a software system that allows the execution of multiple virtual guest operating systems simultaneously on a single physical machine. Xen is known as a Type 1 or “bare-metal” hypervisor, meaning that it runs directly ontop of the physical machine as opposed to within an operating system.
+
Xen Project creates a Virtual Machine Monitor (VMM) also known as a hypervisor: a software system that allows the execution of multiple virtual guest operating systems simultaneously on a single physical machine. In particular, the project creates a Type 1 or “bare-metal” hypervisor, meaning that it runs directly on top of the physical machine as opposed to within an operating system.
   
Guest virtual machines running on Xen are known as “domains” and a special domain known as dom0 is responsible for controlling the hypervisor and starting other guest operating systems. These other guest operating systems are called domUs, this is because these domains are “unprivileged” in the sense they cannot control Xen or start/stop other domains.
+
Guest virtual machines running on a Xen Project Hypervisor are known as “domains”. A special domain known as domain0 (or dom0) is responsible for controlling the hypervisor and starting other guest operating systems. These other guest operating systems are called domUs. This is because these domains are “unprivileged” in the sense they cannot control the hypervisor or start/stop other domains.
   
Xen supports 2 primary types of virtualization, para-virtualization and hardware virtual machine (HVM) also known as “full virtualization”. Para-virtualization uses modified guest operating systems that we refer to as enlightened guests. These operating systems are aware that they are being virtualized and as such don’t require virtual “hardware” devices, instead they make special calls to Xen that allow them to access CPUs, storage and network resources.
+
Our hypervisor supports two primary types of virtualization: paravirtualization (PV) and hardware virtualized machine (HVM) also known as “full virtualization”. Paravirtualization uses modified guest operating systems that we refer to as "enlightened" guests. These operating systems are aware that they are being virtualized and as such don’t require virtual hardware devices. Instead they make special calls to the hypervisor that allow them to access CPUs, storage and network resources.
   
In contrast HVM guests need not be modified as Xen will create a fully virtual set of hardware devices for this machine that resemble a physical x86 computer. This emulation requires much more overhead than the paravirtualisation approach but allows unmodified guest operating systems like Microsoft Windows to run ontop of Xen. HVM support requires special CPU extensions - VT-x for Intel processors and AMD-V for AMD based machines. This technology is now prevalent and all recent servers and desktop systems should be equipped with them.
+
In contrast, HVM guests need not be modified, as the hypervisor will create a fully virtual set of hardware devices for the machine resembling a physical x86 computer. This emulation requires more overhead than the paravirtualization approach but allows unmodified guest operating systems like Microsoft Windows to run on top of the hypervisor. HVMs are supported through virtualization extensions in the CPU. Several iterations of these extensions have been introduced in the last decade or so, collectively known as Intel VT and AMD-V and development continues. The technology is now prevalent; all recent servers, many desktops and some mobile systems should be equipped with at least some extensions.
   
  +
Xen virtualization is now seen as on a spectrum, with PV at one end and HVM at the other. In between are various enhancements to improve performance: HVM with PV drivers, PVHVM or “Paravirtualization on HVM”, and most recently PVH. Each strives to provide the best of both worlds by reducing expensive emulation. To keep things simple for the purposes of this guide, we will create a generic PV guest and optionally, an HVM guest. For a closer look at how the various modes (PV, HVM, PVHVM, PVH etc.) fit into the picture, see
A third type of virtualization though not discussed in this guide is called PVHVM or “Para-virtualisation on HVM” which is a HVM domain with paravirtualized storage, network and other devices. This provides the best of both worlds by reducing expensive emulation but providing hardware accelerated CPU and memory access.
 
A brief look at Xen architecture
 
To understand how storage, networking and other resources are delivered to guest systems we need to quickly delve into how the different bits of Xen interact.
 
   
  +
* [[Understanding the Virtualization Spectrum]]
'''<todo diagram>'''
 
   
  +
=A brief look at Xen Project architecture=
'''<todo explain dom0 role and datapath for guest IO/network>'''
 
   
  +
To understand how storage, networking and other resources are delivered to guest systems we need to quickly delve into how the different bits of the software interact.
The dom0 forms the interface to the hypervisor, through special instructions the dom0 communicates to Xen and changes the configuration of the hypervisor. This includes instantiating new domains and related tasks.
 
   
  +
<!--[[File:Xen-hypervisor-for-the-cloud-from-frontier-meetup-mountain-view-ca-20131014-31-638.jpg]]-->
Another crucial part of the dom0’s role is that it is the primary interface to the hardware. Xen doesn’t contain device drivers, instead the devices are attached to dom0 and you can use standard Linux drivers. Dom0 then shares these resources with guest operating systems through a number of “backend” deamons.
 
   
  +
<!--[[File:XenProjectHypervisorArchitecture.jpg]]-->
For each para-virtualized subsystem in Xen consists of 2 parts, the aforementioned “backend” that lives in dom0 and the “frontend” driver within the guest domain. The backend is effectively a deamon which uses special buffer based interfaces called “eventchannel” to transfer data to guests, be it to provide a virtual hard-disk, Ethernet adapter of even a generic SCSI device.
 
  +
[[File:XenArch1.png]]
The frontend driver then takes this stream of data and converts it back into a device within the guest operating system.
 
   
  +
This is the basic architecture of the Xen Project Hypervisor. We see that the hypervisor sits on the bare metal (the actual computer hardware). The guest VMs all sit on the hypervisor layer, as does dom0, the "Control Domain". The Control Domain is a VM like the guest VMs, except that it has two basic functional differences:
The 2 important paravirtualisation systems are known as net-back/net-front and blk-back/blk-front which are the paravirtualized networking and storage systems respectively.
 
   
  +
1. The Control Domain has the ability to talk to the hypervisor to instruct it to start and stop guest VMs.
You can read more about how Xen is architected, paravirtualization and the benefits of such here:
 
   
  +
2. The Control Domain by default contains the device drivers needed to address the hardware. This stops the problem that often plagued Linux users in the 1990s: You install your software on a new piece of hardware, only to find that you lack the drivers to use it. Since those early days, Linux and the BSDs have become quite good at supporting more pieces of hardware fairly quickly after they are birthed. Xen Project leverages that support by using the drivers in the Control Domain's operating system to access many types of hardware.
'''<todo find xen article on wiki with deep explaination of PV>'''
 
  +
  +
<!--[[File:XenProjectHypervisorArchitecture2.jpg]]-->
  +
[[File:XenArch2.png]]
  +
  +
<!---'''<todo datapath for guest IO/network>'''--->
  +
  +
Dom0 forms the interface to the hypervisor. Through special instructions dom0 communicates to the Xen Project software and changes the configuration of the hypervisor. This includes instantiating new domains and related tasks.
  +
  +
Another crucial part of dom0’s role is as the primary interface to the hardware. The hypervisor doesn’t contain device drivers. Instead the devices are attached to dom0 and use standard Linux drivers. Dom0 then shares these resources with guest operating systems.
  +
  +
To implement paravirtualization, each paravirtualized datapath consists of two parts: 1) a “backend” that lives in dom0, which provides the virtual device and 2) a “frontend” driver within the guest domain, which allows the guest OS to access the virtual device. The backend and frontend use a high-speed software interface based on shared memory to transfer data between the guest and dom0.
  +
  +
The two important paravirtualized datapaths are: '''net-back/net-front''', and '''blk-back/blk-front''' - which are the paravirtualized networking and storage systems, respectively. There are also paravirtualized interrupts, timers, page-tables and more.
  +
  +
You can read more about how the Xen Project system is architected, paravirtualization and the benefits of such here:
  +
  +
* Details of [[Paravirtualization (PV)]] and how it is used on Xen Project
  +
  +
In the case of HVM guests, dom0 uses hardware virtualization extensions provided by the CPU. The most basic of these is virtualization of the CPU itself. Support was later added for page-table management (MMU) and I/O virtualization (IOMMU). Dom0 also emulates some hardware using components of qemu ([https://en.wikipedia.org/wiki/QEMU#Xen-HVM the Quick Emulator]). Emulation in software requires the most overhead, however, so performance is reduced.
   
 
=Preparation=
 
=Preparation=
Line 43: Line 63:
 
This guide requires a number of items, this checklist is what you will need:
 
This guide requires a number of items, this checklist is what you will need:
   
* 64bit x86 computer with atleast 1GB of RAM (this can be a server,desktop or laptop!)
+
* 64bit x86 computer with at least 1GB of RAM (this can be a server, desktop or laptop)
* (Optional) VT-d or AMD-V support
+
* Intel VT or AMD-V support (optional for PV, required for HVM and some PV optimizations)
 
* Sufficient storage space for your dom0 and whatever guests you want to install
 
* Sufficient storage space for your dom0 and whatever guests you want to install
* A CD burner + blank CD (you can use a USB but this is not covered here)
+
* A CD/DVD burner + blank CD/DVD (you can use USB but that's not covered here - see [https://www.debian.org/CD/ the Debian site] for details)
* Internet access for downloading and installing Debian
+
* Internet access and some way to download Debian and burn it to the CD/DVD (e.g. another computer)
  +
* installation ISO for a licensed copy of Windows or a trial copy of Windows Server 2008R2 (only if you want a Windows HVM)
* (Optional) Windows Server 2008R2 installation ISO, a trial copy is sufficient
 
* (Optional) VNC client for installing HVM domain
+
* VNC client (optional for PV, required for HVM)
  +
  +
A word about VT/AMD-V: If you want to be sure you can use the hardware extensions, it is important to check that both the CPU chipset and the motherboard support virtualization. It is quite possible to have virtualization features in the chipset that cannot be enabled because the mobo isn't designed for it. Also, if you plan to use an HVM instance for more that demonstration purposes, the underlying hardware should support at least VT-d and VT-i or AMD-V and AMD-Vi. Having said all of that, sometimes the easiest (or only way) to see what is supported is to check the BIOS.
   
 
==Enable virtualization support in BIOS==
 
==Enable virtualization support in BIOS==
  +
  +
''NOTE: This is not strictly required for PV guests. However, it is highly recommended so that you have the widest number of options for virtualization modes once you get underway.''
   
  +
In order to support HVM guests we need to ensure that virtualization extensions are enabled in the BIOS. If you find your system doesn’t support these extensions you cannot use the hypervisor to virtualize unmodified operating systems. Paravirtualization will work fine though.
''NOTE: This is optional and not required for PV guests''
 
   
  +
The virtualization options appear differently in different BIOS builds but often are referred to as “Enable Intel VT” for Intel chipsets, "Enable AMD-V" for AMD or simply “Enable Virtualization Technology”. Oftentimes this option can be found under the “Advanced Chipset Features” menu in the BIOS or by using Search if the BIOS supports that. It is worthwhile digging around on this a bit. The options may be specified individually, for example: VT-x and VT-d or AMD-V and AMD-IOMMU (aka AMD-Vi or AMD-RVI). You may even find one is enabled by default but the other is not!
In order to support HVM guests we need to ensure that virtualization extensions are enabled in the BIOS. If you don’t wish to start a HVM guest you can skip this step but it is still highly recommended. If your system doesn’t support these extensions you cannot use Xen to virtualize unmodified operating systems, however para-virtualization will work fine.
 
 
The virtualization option appears differently in different BIOS builds but generally it is refered to as “Enable Virtualisation Technology” or “Enable Intel VT” for Intel chipsets, however in some cases it can be listed as “Vanderpool Technology”. Oftentimes this option can be found under the “Advanced Chipset Features” menu in the BIOS. Similar also for AMD.
 
   
 
Consult your motherboard documentation for more assistance in enabling virtualization extensions on your system.
 
Consult your motherboard documentation for more assistance in enabling virtualization extensions on your system.
   
==Download and burn Debian Squeeze Installer CD==
+
==Download and Burn the Debian Installer CD==
  +
  +
You can find the most recent Debian ISO images at this URL:
   
  +
http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/
Download the ISO image from this URL:
 
   
  +
The netinst image is sufficient for our purposes.
http://cdimage.debian.org/debian-cd/6.0.3/amd64/iso-cd/debian-6.0.3-amd64-netinst.iso
 
   
Burn the ISO to disk using your computers standard utilities. I recommend wodim on Linux or the built in ISO burning feature in Windows.
+
Burn the ISO to disk using your computer's standard utilities. Linux has wodim (among others) or use the built in ISO burning feature in Windows.
   
 
==Quick intro to Debian==
 
==Quick intro to Debian==
   
Debian is a simple, stable and well supported Linux distribution. It has included Xen support since Debian 3.1 “Sarge” released in 2005. The current stable release Debian 6.0 “Squeeze” ships with Xen 4.0.1 and a Xen enabled Linux 2.6.32 kernel.
+
Debian is a simple, stable and well supported Linux distribution. It has included Xen Project Hypervisor support since Debian 3.1 “Sarge” released in 2005.
   
 
Debian uses the simple Apt package management system which is both powerful and simple to use.
 
Debian uses the simple Apt package management system which is both powerful and simple to use.
 
Installing a package is as simple as the following example:
 
Installing a package is as simple as the following example:
   
aptitude install htop
+
apt-get install htop
   
Where htop was the application desired to install.
+
where htop is the application desired to install.
   
 
Simple tasks such as configuring startup scripts, setting up the network etc are covered by this tutorial so don’t worry if you haven’t used Debian before!
 
Simple tasks such as configuring startup scripts, setting up the network etc are covered by this tutorial so don’t worry if you haven’t used Debian before!
Line 84: Line 108:
 
Many popular distributions are based off of Debian and also use the Apt package manager, if you have used Ubuntu, Linux Mint or Damn Small Linux you will feel right at home.
 
Many popular distributions are based off of Debian and also use the Apt package manager, if you have used Ubuntu, Linux Mint or Damn Small Linux you will feel right at home.
   
=Installing Debian Squeeze=
+
=Installing Debian =
   
Boot the Debian Squeeze Installer CD
+
Boot the Debian Installer CD/DVD
  +
Insert the Debian CD and configure the CDROM drive as your default boot device in the BIOS or use the system bootmenu if your BIOS supports it (usually F12).
 
  +
Insert the Debian CD/DVD and configure the CDROM drive as your default boot device in the BIOS or use the system boot menu if your BIOS supports it (usually F12).
   
 
You should see a menu, choose the default “Install” option to begin the installation process.
 
You should see a menu, choose the default “Install” option to begin the installation process.
Line 110: Line 135:
 
sda4 - reserved for LVM
 
sda4 - reserved for LVM
   
When you reach the package selection stage only install the base system, we won’t require any GUI or other packages for this guide.
+
When you reach the package selection stage only install the base system. We won’t require any GUI or other packages for this guide. (If you want to set up a graphical desktop environment in dom0, that's not a problem, but you may want to wait until after you've completed this guide to avoid complicating things.)
   
'''<todo: extend to cover the rest of the installation process>'''
+
You can find out details of the Debian installation process from [https://www.debian.org/releases/stable/amd64/ the Debian documentation].
   
  +
<!--'''<todo: extend to cover the rest of the installation process>'''
  +
-->
 
Continue through the installer then reboot and login at the prompt as root.
 
Continue through the installer then reboot and login at the prompt as root.
  +
  +
If you've got any hardware [http://forums.debian.net/viewtopic.php?f=5&t=104202 you're not sure open source drivers are available for], you may want to install non-free firmware files via:
  +
  +
apt-get install firmware-linux-nonfree
  +
  +
If this does not work straight away make sure your <code>/etc/apt/sources.list</code> has entries including <code>non-free</code> and perhaps <code>contrib</code> while you're at it, e.g. like this:
  +
  +
deb <nowiki>http://some.debian.server.org/debian</nowiki> stretch main contrib non-free
  +
  +
Add the same to <code>deb-src</code> and the <code>stretch/updates</code> lines (changing 'stretch' to the name of the current Debian release if you're using some newer release).
  +
  +
=Installing the Xen Project Software=
  +
  +
We've still got a few more steps to complete before we're ready to launch a domU, but let's install the Xen Project software now and use it to check the BIOS settings.
  +
  +
The Debian Xen Project packages consist primarily of a Xen Project-enabled Linux kernel, the hypervisor itself, a modified version of QEMU that support the hypervisor’s HVM mode and a set of userland tools.
  +
  +
All of this can be installed via an Apt meta-package called xen-linux-system. A meta-package is basically a way of installing a group of packages automatically. Apt will of course resolve all dependencies and bring in all the extra libraries we need.
  +
  +
Let's install the xen-linux-system meta-package:
  +
  +
apt-get install xen-system-amd64
  +
  +
Now we have a Xen Project hypervisor, a Xen Project kernel and the userland tools installed. When you next boot the system, the boot menu should include entries for starting Debian with the Xen hypervisor. One of them should be highlighted, to start Xen by default. Do that now, logging in as root again.
  +
  +
Next, let's check to see if virtualization is enabled in the BIOS. There are a few ways to do that.
  +
  +
The most comprehensive is to review the Xen section of dmesg created during the boot process. This will be your first use of xl, the very versatile Xen tool, which we will come back to shortly to create and manage domUs:
  +
  +
xl dmesg
  +
  +
Included in the output will be references to the CPU flags set in the BIOS to enable virtualization: 'vmx' for Intel, 'svm' for AMD. It will also detail other hardware virtualization extensions: VT-d features, Hardware Assisted Paging (HAP), I/O Virtualization and so on.
  +
  +
Another way is to check the flags set in the CPU on boot:
  +
  +
egrep '(vmx|svm|hypervisor)' /proc/cpuinfo
  +
  +
egrep will return any line containing one or more of those same text fragments (vmx/svm or more recently, just 'hypervisor'). If nothing comes back and you think it should, you may wish to look through the flags yourself:
  +
  +
cat /proc/cpuinfo
  +
  +
If the virtualization extensions don't appear, take a closer look at the BIOS settings. A few round-trips through the BIOS are often required to get all the bits working right.
   
 
=Setup LVM storage for guests=
 
=Setup LVM storage for guests=
   
LVM is the Linux Logical Volume manager. It is a technology that allows Linux to manage block devices in a more abstract manner.
+
LVM is the Linux Logical Volume Manager. It is a technology that allows Linux to manage block devices in a more abstract manner.
   
LVM introduces the concept of a “logical volume”, effectively a virtualized block device composed of blocks written to 1 or more physical devices. These blocks don’t need to be contiguous unlike proper disk partitions.
+
LVM introduces the concept of a “logical volume”, effectively a virtualized block device composed of blocks written to one or more physical devices. Unlike proper disk partitions, these blocks don’t need to be contiguous.
   
 
Because of this abstraction logical volumes can be created, deleted, resized and even snapshotted without affecting other logical volumes.
 
Because of this abstraction logical volumes can be created, deleted, resized and even snapshotted without affecting other logical volumes.
Line 126: Line 195:
 
LVM creates logical volumes within what is called a volume group, which is simply a set of logical volumes that share the same physical storage, known as physical volumes.
 
LVM creates logical volumes within what is called a volume group, which is simply a set of logical volumes that share the same physical storage, known as physical volumes.
   
The process of setting up LVM can be sumarized as allocating a physical volume, creating a volume group ontop of this then creating logical volumes to store data.
+
The process of setting up LVM can be summarized as allocating a physical volume, creating a volume group on top of this, then creating logical volumes to store data.
   
Because of these features and superior performance over file backed virtual machines I recommend the use of LVM if you are going to store VM data locally.
+
Because of these features and superior performance over file backed virtual machines we recommend the use of LVM if you are going to store VM data locally.
   
 
Now lets install LVM and get started!
 
Now lets install LVM and get started!
   
Install LVM with aptitude by running this command:
+
Install LVM:
   
aptitude install lvm2
+
apt-get install lvm2
   
Now that we have LVM installed lets configure it to use /dev/sda4 as it’s physical volume
+
Now that we have LVM installed let's configure it to use /dev/sda4 as its physical volume
   
 
pvcreate /dev/sda4
 
pvcreate /dev/sda4
   
Ok, now LVM has somewhere to store it’s blocks (known as extents for future reference).
+
Ok, now LVM has somewhere to store its blocks (known as extents for future reference).
Lets create a volume group called ‘vg0’ using this physical volume:
+
Let's create a volume group called ‘vg0’ using this physical volume:
   
 
vgcreate vg0 /dev/sda4
 
vgcreate vg0 /dev/sda4
Line 147: Line 216:
 
Now LVM is setup and initialized so that we can later create logical volumes for our virtual machines.
 
Now LVM is setup and initialized so that we can later create logical volumes for our virtual machines.
   
  +
For the interested below is a number of useful commands and tricks when using LVM.
'''<todo: write advanced lvm storage guide and link to it here>'''
 
   
  +
Create a new logical volume:
=Setup Linux Bridge for guest networking=
 
   
  +
lvcreate -n <name of the volume> -L <size, you can use G and M here> <volume group>
Next we need to setup our system so that we can attach virtual machines to the external network. This is done by creating a virtual switch within dom0 that takes packets from the virtual machines and forwards them onto the physical network so they can see the internet and other machines on your network.
 
   
  +
For example, creating a 100 gigabyte volume called database-data on a volume group called vg0.
The piece of software we use to do this is called the Linux bridge and it’s core components reside inside the Linux kernel. In this case the “bridge” is effectively our virtual switch. Our Debian kernels are compiled with the Linux bridging module so all we need to do is install the control utilities.
 
   
  +
lvcreate -n database-data -L 100G vg0
aptitude install bridge-utils
 
   
  +
You can then remove this volume with the following:
With bridge-utils installed we now have a utility called “brctl” this utility talks to the Linux bridging module to setup new bridges and attach physical or virtual interfaces to these bridges.
 
   
  +
lvremove /dev/vg0/database-data
brctl can be used to create a bridge as such, where <bridgename> is the name of the bridge:
 
   
  +
Note that you have to provide the path to the volume here.
brctl addbr <bridgename>
 
   
  +
[https://wiki.debian.org/LVM#List_of_VG_commands More on LVM on Debian here].
And interfaces can be added to that bridge by running the following:
 
   
  +
If you already have a volume setup that you would like to copy, LVM has a cool feature that allows you to create a CoW (copy on write) clone called a snapshot.
brctl addif <bridgename> <interface>
 
  +
This means that you can make an "instant" copy that will only store the changes compared to the original. There are a number of caveats to this that will be discussed in a yet unwritten article.
  +
The most important thing to note is that the "size" of the snapshot is only the amount of space allocated to store changes. So you can make the snapshot "size" a lot smaller than the source volume.
   
  +
To create a snapshot use the following command:
Instead of calling brctl directly we are instead going to configure our bridge through Debian’s networking infrastructure which can be configured through
 
   
  +
lvcreate -s /dev/vg0/database-data -ndatabase-backup -L5G
/etc/network/interfaces
 
   
  +
Once again note the use of the full path.
Open this file with the editor of your choice (more editors can be installed with aptitude)
 
  +
Nano is installed by default if you selected the minimal install
 
  +
=Setup Linux Bridge for guest networking=
  +
  +
Next we need to set up our system so that we can attach virtual machines to the external network. This is done by creating a virtual switch within dom0. The switch will take packets from the virtual machines and forward them on to the physical network so they can see the internet and other machines on your network.
  +
  +
The piece of software we use to do this is called the Linux bridge and its core components reside inside the Linux kernel. In this case, the bridge acts as our virtual switch. The Debian kernel is compiled with the Linux bridging module so all we need to do is install the control utilities:
  +
  +
apt-get install bridge-utils
  +
  +
Management of the bridge is usually done using the <code>brctl</code> command. The initial setup for our Xen bridge, though, is a "set it once and forget it" kind of thing, so we are instead going to configure our bridge through Debian’s networking infrastructure. It can be configured via '''/etc/network/interfaces'''.
  +
  +
Open this file with the editor of your choice. If you selected a minimal installation, the '''nano''' text editor should already be installed. Open the file:
   
 
nano /etc/network/interfaces
 
nano /etc/network/interfaces
  +
  +
(If you get <code>nano: command not found</code>, install it with <code>apt-get install nano</code>.)
   
 
Depending on your hardware you probably see a file pretty similar to this:
 
Depending on your hardware you probably see a file pretty similar to this:
Line 187: Line 271:
   
 
This file is very simple. Each stanza represents a single interface.
 
This file is very simple. Each stanza represents a single interface.
  +
Breaking it down “auto eth0” means that eth0 will be configured when ifup -a is run (which is run a boot time) what this means is that the interface with automatically be started/stopped for you.
 
  +
Breaking it down, “auto eth0” means that eth0 will be configured when ifup -a is run (which happens at boot time). This means that the interface will automatically be started/stopped for you. ("eth0 is its traditional name - you'll probably see something more current like "ens1", "en0sp2" or even "enx78e7d1ea46da")
“iface eth0” then describes the interface itself, in this case it merely specifies that it should be configured by DHCP - we are going to assume that you have DHCP running on your network for this guide. If you are using static addressing you probably know how to set that up.
 
  +
  +
“iface eth0” then describes the interface itself. In this case, it specifies that it should be configured by DHCP - we are going to assume that you have DHCP running on your network for this guide. If you are using static addressing you probably know how to set that up.
  +
 
We are going to edit this file so it resembles such:
 
We are going to edit this file so it resembles such:
   
Line 195: Line 282:
 
iface lo inet loopback
 
iface lo inet loopback
   
  +
auto eth0
 
iface eth0 inet manual
 
iface eth0 inet manual
  +
 
auto xenbr0
 
auto xenbr0
 
 
iface xenbr0 inet dhcp
 
iface xenbr0 inet dhcp
bridge_ports eth0
+
bridge_ports eth0
 
</pre>
 
</pre>
   
  +
As well as adding the bridge stanza, be sure to change '''dhcp''' to '''manual''' in the <code>iface eth0 inet manual</code> line, so that IP (Layer 3) is assigned to the bridge, not the interface. The interface will provide the physical and data-link layers (Layers 1 & 2) only.
This will setup the bridge xenbr0 and add eth0 to the bridge for us.
 
The equivalent commands would be:
 
   
  +
Now restart networking (for a remote machine, make sure you have a backup way to access the host if this fails):
brctl addbr xenbr0
 
brctl addif xenbr0 eth0
 
dhclient xenbr0
 
   
  +
<pre>
Except this will now be done automatically on boot and completely managed by Debian.
 
  +
service networking restart
  +
</pre>
   
  +
and check to make sure that it worked:
This method of network configuration is best practice and the automated scripts called by xend are now deprecated, we will discuss this later once we have Xen installed.
 
   
  +
<pre>
=Installing Xen=
 
  +
brctl show
  +
</pre>
   
  +
If all is well, the bridge will be listed and your interface will appear in the interfaces column:
The Debian Xen packages consists primarily of a Xen enabled Linux kernel, the Xen hypervisor itself, a modified version of QEMU that support Xen’s HVM mode and a set of userland tools.
 
   
  +
<pre>
All of this except QEMU can be installed via an Apt meta-package callled xen-linux-system. A meta-package is basically a way of installing a group of packages automatically. Apt will ofcourse resolve all dependencies and bring in all the extra libraries we need.
 
  +
bridge name bridge id STP enabled interfaces
  +
xenbr0 8000.4ccc6ad1847d no enp2s0
  +
</pre>
   
  +
Bridged networking will now start automatically every boot.
Lets install the xen-linux-system metapackage:
 
   
  +
If the bridge isn't operating correctly, go back and check the edits to the <code>interfaces</code> file very carefully.
aptitude -P install xen-linux-system
 
   
  +
Reboot before continuing. During the reboot, note the list of OS choices and check to see what the default start-up choice is. "Debian GNU/Linux, with Xen hypervisor" (or equivalent) means Xen is loading by default. If both the start-up default is fine, skip the next section and go directly to Basic Xen Project Commands.
Next we will install the Xen QEMU package so that we can boot HVM guests later (this is optional but highly recommended)
 
   
  +
=Configure GRUB to start Xen Project=
aptitude install xen-qemu-dm
 
   
  +
GRUB, the bootloader installed during installation, tells the computer which operating system to start and how. To use the hypervisor, Xen must be started before the operating system. Depending on your use-case for Xen, you may or may not want GRUB to default to loading Xen.
Now have Xen, a Xen kernel and the userland tools installed, almost ready to go.
 
   
  +
GRUB2's configuration is stored in the file /boot/grub/grub.cfg
=Configure GRUB to start Xen=
 
   
  +
We aren’t going to edit this file directly, as it changes every time we update our kernel.
Because Xen starts before your operating system we need to change how your systems boot process is setup. The bootloader installed during installation called GRUB is what tells your computer which operating system to start and how.
 
  +
Debian configures GRUB for us using a number of automated scripts that handle upgrades etc, these scripts are stored in /etc/grub.d/* and can be configured via
 
GRUB2 configuration is stored in the file /boot/grub/grub.cfg
 
However we aren’t going to edit this file directly, as it changes every time we update our kernel.
 
Debian configures GRUB for us using a number of automated scripts that handle upgrades etc, these scripts are stored in /etc/grub.d/* and are configured via
 
   
 
/etc/default/grub
 
/etc/default/grub
   
  +
To change the default operating system change the GRUB_DEFAULT line in that file (or add it if it's not already there).
We are going to change the order of the operating systems so that our Xen system is the default option. By executing the below command we are moving Xen to a higher priority than default Linux so that it gets the first position in the boot menu.
 
  +
  +
GRUB_DEFAULT=0
  +
  +
causes the first OS in the boot list to be the default, and so on. So, if Xen was, say, the third OS choice, change the line to
  +
  +
GRUB_DEFAULT=2
   
  +
to have Xen load by default.
mv -i /etc/grub.d/20_linux_xen /etc/grub.d/09_linux_xen
 
   
We then generate the /boot/grub/grub.cfg file by running the command below:
+
Then regenerate the /boot/grub/grub.cfg file by running:
   
 
update-grub
 
update-grub
   
We can now reboot and the default boot option will be our Xen dom0 running ontop of Xen!
+
At the next reboot confirm the correct default boot option is selected.
   
'''See also'''
 
* [[Xen Hypervisor Boot Options|Xen GRUB Boot Options]]
 
   
=Basic Xen commands=
+
=Basic Xen Project Commands=
   
Before we dive into creating some guest domains we will quickly cover some basic Xen commands using the “xm” utiltiy. Going forward it is likely that xm will be deprecated as previously mentioned but once xl is stable and part of a supported release this guide will be rewritten to reflect that.
+
Before we dive into creating some guest domains we will quickly cover some basic commands. In the examples below, we use '''[[XL|xl]]''' command line tool. Older versions of the Xen Project software used the '''[[XM|xm]]''' command line tool. '''xl''' and '''xm''' are command line compatible (the format of the output may be slightly different). If, for example, you come across "xm" while reading old documentation, say, just substitute "xl".
   
So lets start with simple stuff!
+
Lets start with simple stuff!
   
xm info
+
xl info
   
This returns the information about the Xen hypervisor and dom0 including version, free memory etc.
+
returns the information about the hypervisor and dom0 including version, free memory etc.
   
xm list
+
xl list
   
Lists running domains, their IDs. memory, state and CPU time consumed
+
lists running domains, their IDs, memory, state and CPU time consumed
   
xm top
+
xl top
   
Shows running domains in real time and is similar to the “top” command under Linux. This can be used to visualize CPU, memory usage and block device access.
+
shows running domains in real time and is similar to the “top” command under Linux. This can be used to visualize CPU, memory usage and block device access.
   
 
We will cover some more commands during the creation of our guest domains.
 
We will cover some more commands during the creation of our guest domains.
   
'''See also'''
+
'''See also:'''
* [[Xen Man Pages]]
+
* [[Xen_4.x_Manuals|Xen Project 4.x Man Pages]]
   
 
=Creating a Debian PV (Paravirtualized) Guest=
 
=Creating a Debian PV (Paravirtualized) Guest=
   
PV guests are notoriously “different” to install. Due to the nature of enlightened systems they don’t have the usual concepts of a CD-ROM drive installer analagous to their physical counterparts. However, luckly enough there does exist some tools that help us prepare “images” or effectively snapshots of the operating systems that are able to run inside of Xen domains.
+
PV guests are notoriously “different” to install. Fortunately, though, there are tools that help us prepare “images” or snapshots of the operating systems to run inside guest domains.
   
Debian contains a number of tools for creating Xen guests. The easiest of which is known as xen-tools. This software suite manages the downloading and installing of guest operating systems including both Debian and RHEL based DomUs. In this guide we are going to use xen-tools to prepare a Debian Squeeze paravirtualized domU.
+
Debian contains a number of tools for creating Xen Project guests, the easiest of which is known as [[xen-tools]]. This software suite manages the downloading and installing of guest operating systems including both Debian and RHEL based DomUs. In this guide we are going to use xen-tools to prepare a Debian paravirtualized domU.
   
Xen-tools can use LVM storage for storing the guest operating systems, in this guide we created the volume group “vg0” in the Setting up LVM Storage section.
+
xen-tools can use LVM storage for storing the guest operating systems. In this guide we created the volume group “vg0” in the Setting up LVM Storage section.
   
 
When guests are paravirtualized there is no “BIOS” or bootloader resident within the guest filesystem and for a long time guests were provided with kernels external to the guest image. This however is bad for maintainability (guests cannot upgrade their kernels without access to the dom0) and is not as flexible in terms of boot options as they must be passed via the config file.
 
When guests are paravirtualized there is no “BIOS” or bootloader resident within the guest filesystem and for a long time guests were provided with kernels external to the guest image. This however is bad for maintainability (guests cannot upgrade their kernels without access to the dom0) and is not as flexible in terms of boot options as they must be passed via the config file.
   
The Xen community wrote a utility known as pygrub which is a python application for PV guests that enables the dom0 to parse the GRUB configuration of the domU and extract it’s kernel, initrd and boot parameters. This allows for kernel upgrades etc inside of our guest machines along with a GRUB menu. Using pygrub or the stub-dom implementation known as pv-grub is best practice for starting PV guests. In some cases pv-grub is arguably more secure but as it is not included with Debian we won’t use it here though it is recommended in production environments where guests cannot be trusted.
+
The Xen Project community wrote a utility known as pygrub which is a python application for PV guests that enables the dom0 to parse the GRUB configuration of the domU and extract its kernel, initrd and boot parameters. This allows for kernel upgrades etc inside of our guest machines along with a GRUB menu. Using pygrub or the stub-dom implementation known as pv-grub is best practice for starting PV guests. In some cases pv-grub is arguably more secure but as it is not included with Debian we won’t use it here though it is recommended in production environments where guests cannot be trusted.
   
 
Apart from this PV guests are very similar to their HVM and physical OS counterparts.
 
Apart from this PV guests are very similar to their HVM and physical OS counterparts.
  +
Configuring xen-tools and building our guest
 
  +
==Configuring xen-tools and building our guest==
  +
 
First lets install the xen-tools package:
 
First lets install the xen-tools package:
   
aptitude install xen-tools
+
apt-get install xen-tools
   
We can now create a guest operating system with this tool. It effectly automates the process of setting up a PV guest from scratch right to the point of creating config files and starting the guest. The process can be summarized as follows:
+
We can now create a guest operating system with this tool. It effectively automates the process of setting up a PV guest from scratch right to the point of creating config files and starting the guest. The process can be summarized as follows:
   
 
* Create logical volume for rootfs
 
* Create logical volume for rootfs
Line 301: Line 396:
 
* Install operating system using debootstrap (or rinse etc, only debootstrap covered here)
 
* Install operating system using debootstrap (or rinse etc, only debootstrap covered here)
 
* Run a series of scripts to generate guest config files like fstab/inittab/menu.lst
 
* Run a series of scripts to generate guest config files like fstab/inittab/menu.lst
* Create a Xen config file for the guest
+
* Create a VM config file for the guest
 
* Generate a root password for the guest system
 
* Generate a root password for the guest system
 
* Unmount the guest filesystem
 
* Unmount the guest filesystem
   
These 9 steps can be carried out manually but the manual process is outside the scope of this guide. We instead will execute the below command:
+
These 9 steps can be carried out manually but the manual process is outside the scope of this guide. We instead will execute the below command (for --dist you could in place of Wheezy e.g. use Squeeze, or even Precise or Quantal for a Ubuntu install):
   
 
xen-create-image --hostname=tutorial-pv-guest \
 
xen-create-image --hostname=tutorial-pv-guest \
Line 313: Line 408:
 
--dhcp \
 
--dhcp \
 
--pygrub \
 
--pygrub \
--dist=squeeze
+
--dist=wheezy
   
This command instructs xen-create-image (the primary binary of the xen-tools toolkit) to create a guest domain with 512MB of memory, 2 vcpus, using storage from the vg0 volume group we created, use DHCP for networking, pygrub to extract the kernel from the image when booted and lastly we specify that we want to deploy a Debian Squeeze operating system.
+
This command instructs xen-create-image (the primary binary of the xen-tools toolkit) to create a guest domain with 512MB of memory, 2 vcpus, using storage from the vg0 volume group we created, use DHCP for networking, pygrub to extract the kernel from the image when booted and lastly we specify that we want to deploy a Debian Wheezy operating system.
   
This process will take a few minutes after which you can start the guest with:
+
This process will take a few minutes. Once it is complete, it will provide a summary of the installation. Take note of the root password for the guest.
   
  +
Also see
xm create -c /etc/xen/tutorial-pv-guest.cfg
 
  +
* [http://manpages.debian.org/cgi-bin/man.cgi?query=xen-create-image&apropos=0&sektion=0&manpath=Debian+6.0+squeeze&format=html&locale=en xen-create-image(8) man page]
  +
* [[Xen-tools|Further articles on xen-tools]]
   
  +
=Creating a PV (Paravirtualized) Guest manually=
The -c in this command tells xm that we wish to connect to the guest virtual console. Which is a paravirtualised serial port within the domain that xen-create-image configured to listen with a getty. This is analgous to running:
 
   
  +
Not every distribution provides the xen-tools package for an automated PV creation and configuration, and some Xen users prefer more control of the setup process.
xm create /etc/xen/tutorial-pv-guest.cfg && xm console tutorial-pv-guest
 
  +
  +
Alpine Linux is such a distro. It provides detailed instructions [http://wiki.alpinelinux.org/wiki/Create_Alpine_Linux_PV_DomU on installing and starting a PV domU manually], using PVGRUB2 rather than pygrub.
   
  +
It also provides valuable information regarding the startup options available.
You can leave the guest virtual console by pressing ctrl+] and re-enter it by running the “xm console <domain>” command.
 
  +
  +
=Starting a console guest=
  +
  +
To start the guest, run (as root or using sudo):
  +
  +
xl create -c /etc/xen/tutorial-pv-guest.cfg
  +
  +
The -c in this command tells '''xl''' that we wish to connect to the guest virtual console, a paravirtualized serial port within the domain that xen-create-image configured to listen with a getty. This is analogous to running:
  +
  +
xl create /etc/xen/tutorial-pv-guest.cfg && xl console tutorial-pv-guest
  +
  +
You can leave the guest virtual console by pressing ctrl+] and re-enter it by running the “xl console <domain>” command.
   
 
You can later shutdown this guest either from within the domain or from dom0 with the following:
 
You can later shutdown this guest either from within the domain or from dom0 with the following:
   
xm shutdown tutorial-pv-guest
+
xl shutdown tutorial-pv-guest
   
That completes our section on setting up your first paravirtualized domain! If you don’t have any interest in setting up a HVM domain then no need to read any further but it is highly recommended!
+
That completes our section on setting up your first paravirtualized domain! If you don’t have any interest in setting up a HVM domain, skip ahead to Starting a GUI guest, below.
   
 
=Creating a Windows HVM (Hardware Virtualized) Guest=
 
=Creating a Windows HVM (Hardware Virtualized) Guest=
   
HVM guests are quite a bit different to their PV counterparts. Because they require the emulation of hardware there is more moving pieces that need to be configured etc.
+
HVM guests are quite a bit different to their PV counterparts. Because they require the emulation of hardware there are more moving pieces that need to be configured etc.
   
 
The main point worth mentioning here is that HVM requires the emulation of ATA, Ethernet and other devices, while virtualized CPU and Memory access is performed in hardware to achieve good performance. Because of this the default emulated devices are very slow and we generally try to use PV drivers within HVM domains. We will be installing a set of Windows PV drivers that greatly increase performance once we have our Windows guest running.
 
The main point worth mentioning here is that HVM requires the emulation of ATA, Ethernet and other devices, while virtualized CPU and Memory access is performed in hardware to achieve good performance. Because of this the default emulated devices are very slow and we generally try to use PV drivers within HVM domains. We will be installing a set of Windows PV drivers that greatly increase performance once we have our Windows guest running.
   
This extra emulation is provided by a Xen modified version of QEMU we should have installed this earlier but in case you skipped that step install the Xen QEMU package now:
+
This extra emulation is provided by QEMU which will have been installed along with the Xen software.
  +
  +
To set up the HVM domU, we need to create a logical volume to store our Windows VM hard disk, create a config file that tells the hypervisor to start the domain in HVM mode and boot from the DVD in order to install Windows.
  +
  +
First, create the new logical volume - name the volume "windows", set the size to 20GB and use the volume group vg0 we created earlier.
  +
  +
lvcreate -nwindows -L20G vg0
  +
  +
Next open a new file with your text editor of choice:
  +
  +
nano windows.cfg
  +
  +
Paste the config below into the file and save it, NOTE this assumes your Windows iso is located in /root/ with the filename windows.iso. In the <code>kernel =</code> line below, be sure the xen version number matches your installation (e.g. xen-4.11, not 4.0):
  +
  +
kernel = "/usr/lib/xen-4.0/boot/hvmloader"
  +
type='hvm'
  +
memory = 4096
  +
vcpus=4
  +
name = "ovm-1734"
  +
vif = ['bridge=xenbr0']
  +
disk = ['phy:/dev/vg0/windows,hda,w','file:/root/windows.iso,hdc:cdrom,r']
  +
acpi = 1
  +
<!-- device_model = 'qemu-dm' -->
  +
device_model_version = 'qemu-xen'
  +
boot="d"
  +
sdl=0
  +
serial='pty'
  +
vnc=1
  +
vnclisten=""
  +
vncpasswd=""
  +
  +
The <code>vnclisten=</code> line specifies valid VNC connection addresses. <code>vnclisten="127.0.0.1"</code> will limit connections to the local machine. <code>vnclisten="0.0.0.0"</code> will accept unauthenticated remote connections from anywhere so is not suitable except in a secure network.
  +
  +
Start the guest as described below in Starting a GUI guest and proceed with Windows' installation.
  +
  +
Once you have installed Windows by formatting the disk and by following the prompts the domain will restart - however this time we want to prevent it booting from DVD so destroy the domain with
  +
  +
xl destroy windows
  +
  +
Then change the boot line in the config file to read boot="c"' restart the domain with
  +
  +
xl create windows.cfg
  +
  +
Reconnect with VNC and finish the installation.
  +
When this process is complete you should then proceed to download the GPLPV drivers for Windows by James Harper.
  +
  +
==Installing PV drivers for HVM guests==
  +
  +
Signed drivers can be obtained from [http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers Univention's website].
  +
  +
Many thanks for Univention for making signed drivers available to the Xen Project community and of course a massive thanks to James for all his work on making Windows in guest VMs such a smooth experience.
  +
  +
On finalizing the installation and rebooting you should notice much improved disk and network performance and the hypervisor will now be able to gracefully shutdown your Windows domains.
  +
  +
Another slightly different version of James Harper's drivers can be found [https://github.com/spurious/win-pvdrivers-mirror here].
  +
  +
=Starting a GUI guest (with VNC server)=
  +
  +
Here is the command to start the domain and connect to it via VNC from your graphical machine.
  +
  +
xl create windows.cfg
  +
  +
The VNC display should be available on port 5900 of your dom0 IP, for instance using gvncviewer:
  +
  +
gvncviewer <dom0-ip-address>:5900
  +
  +
If this does not work try it without the port number and if you are trying from a GUI on dom0, try specifying localhost instead of the dom0 ip:
  +
  +
gvncviewer localhost
  +
  +
= Conclusion =
  +
  +
That concludes our introduction to the Xen Project software, by now you can setup both PV and HVM domains on a bare dom0 hypervisor!
  +
  +
You can now move onto building your own guest images or try out some prebuilt [[Guest VM Images]].
   
aptitude install xen-qemu-dm
 
   
 
[[Category:Xen]]
 
[[Category:Xen]]
 
[[Category:Beginners]]
 
[[Category:Beginners]]
 
[[Category:Tutorial]]
 
[[Category:Tutorial]]
  +
[[Category:Debian]]
  +
[[Category:Fundamentals]]

Latest revision as of 23:41, 28 July 2020

Welcome!

This guide was written to introduce beginners to basic Xen Project concepts and allow you to get started with Xen Project with no prior knowledge. Some prior Linux experience is required however, and some knowledge of networking, lvm and grub will go a long way! You will need to use root-level access at various points, so some experience with that (likely through sudo) is necessary.

By completing this guide you will have installed a fully functional Xen Project hypervisor and started your first guest operating systems, connected them to your network and have been introduced to fundamental concepts such as virtual machine storage and virtual networking.

To make this process easy we will be using a Linux distribution called Debian. This document was originally written for use with Debian 6/7 (called "Squeeze" and “Wheezy”) and has been tested with newer releases through Debian 10 ("Buster"). It should continue work with future releases as well. Debian ships with support for Xen Project release 4.x, providing everything you need to get started!

Though this guide looks long at first, don’t let it daunt you. It is very in-depth and comprehensive, and doesn’t expect you to know all that much beforehand. The goal instead is to teach you all the things you need to know to build a functioning Xen Project Hypervisor. :)

What is this Xen Project software all about?

Xen Project creates a Virtual Machine Monitor (VMM) also known as a hypervisor: a software system that allows the execution of multiple virtual guest operating systems simultaneously on a single physical machine. In particular, the project creates a Type 1 or “bare-metal” hypervisor, meaning that it runs directly on top of the physical machine as opposed to within an operating system.

Guest virtual machines running on a Xen Project Hypervisor are known as “domains”. A special domain known as domain0 (or dom0) is responsible for controlling the hypervisor and starting other guest operating systems. These other guest operating systems are called domUs. This is because these domains are “unprivileged” in the sense they cannot control the hypervisor or start/stop other domains.

Our hypervisor supports two primary types of virtualization: paravirtualization (PV) and hardware virtualized machine (HVM) also known as “full virtualization”. Paravirtualization uses modified guest operating systems that we refer to as "enlightened" guests. These operating systems are aware that they are being virtualized and as such don’t require virtual hardware devices. Instead they make special calls to the hypervisor that allow them to access CPUs, storage and network resources.

In contrast, HVM guests need not be modified, as the hypervisor will create a fully virtual set of hardware devices for the machine resembling a physical x86 computer. This emulation requires more overhead than the paravirtualization approach but allows unmodified guest operating systems like Microsoft Windows to run on top of the hypervisor. HVMs are supported through virtualization extensions in the CPU. Several iterations of these extensions have been introduced in the last decade or so, collectively known as Intel VT and AMD-V and development continues. The technology is now prevalent; all recent servers, many desktops and some mobile systems should be equipped with at least some extensions.

Xen virtualization is now seen as on a spectrum, with PV at one end and HVM at the other. In between are various enhancements to improve performance: HVM with PV drivers, PVHVM or “Paravirtualization on HVM”, and most recently PVH. Each strives to provide the best of both worlds by reducing expensive emulation. To keep things simple for the purposes of this guide, we will create a generic PV guest and optionally, an HVM guest. For a closer look at how the various modes (PV, HVM, PVHVM, PVH etc.) fit into the picture, see

A brief look at Xen Project architecture

To understand how storage, networking and other resources are delivered to guest systems we need to quickly delve into how the different bits of the software interact.


XenArch1.png

This is the basic architecture of the Xen Project Hypervisor. We see that the hypervisor sits on the bare metal (the actual computer hardware). The guest VMs all sit on the hypervisor layer, as does dom0, the "Control Domain". The Control Domain is a VM like the guest VMs, except that it has two basic functional differences:

1. The Control Domain has the ability to talk to the hypervisor to instruct it to start and stop guest VMs.

2. The Control Domain by default contains the device drivers needed to address the hardware. This stops the problem that often plagued Linux users in the 1990s: You install your software on a new piece of hardware, only to find that you lack the drivers to use it. Since those early days, Linux and the BSDs have become quite good at supporting more pieces of hardware fairly quickly after they are birthed. Xen Project leverages that support by using the drivers in the Control Domain's operating system to access many types of hardware.

XenArch2.png


Dom0 forms the interface to the hypervisor. Through special instructions dom0 communicates to the Xen Project software and changes the configuration of the hypervisor. This includes instantiating new domains and related tasks.

Another crucial part of dom0’s role is as the primary interface to the hardware. The hypervisor doesn’t contain device drivers. Instead the devices are attached to dom0 and use standard Linux drivers. Dom0 then shares these resources with guest operating systems.

To implement paravirtualization, each paravirtualized datapath consists of two parts: 1) a “backend” that lives in dom0, which provides the virtual device and 2) a “frontend” driver within the guest domain, which allows the guest OS to access the virtual device. The backend and frontend use a high-speed software interface based on shared memory to transfer data between the guest and dom0.

The two important paravirtualized datapaths are: net-back/net-front, and blk-back/blk-front - which are the paravirtualized networking and storage systems, respectively. There are also paravirtualized interrupts, timers, page-tables and more.

You can read more about how the Xen Project system is architected, paravirtualization and the benefits of such here:

In the case of HVM guests, dom0 uses hardware virtualization extensions provided by the CPU. The most basic of these is virtualization of the CPU itself. Support was later added for page-table management (MMU) and I/O virtualization (IOMMU). Dom0 also emulates some hardware using components of qemu (the Quick Emulator). Emulation in software requires the most overhead, however, so performance is reduced.

Preparation

This guide requires a number of items, this checklist is what you will need:

  • 64bit x86 computer with at least 1GB of RAM (this can be a server, desktop or laptop)
  • Intel VT or AMD-V support (optional for PV, required for HVM and some PV optimizations)
  • Sufficient storage space for your dom0 and whatever guests you want to install
  • A CD/DVD burner + blank CD/DVD (you can use USB but that's not covered here - see the Debian site for details)
  • Internet access and some way to download Debian and burn it to the CD/DVD (e.g. another computer)
  • installation ISO for a licensed copy of Windows or a trial copy of Windows Server 2008R2 (only if you want a Windows HVM)
  • VNC client (optional for PV, required for HVM)

A word about VT/AMD-V: If you want to be sure you can use the hardware extensions, it is important to check that both the CPU chipset and the motherboard support virtualization. It is quite possible to have virtualization features in the chipset that cannot be enabled because the mobo isn't designed for it. Also, if you plan to use an HVM instance for more that demonstration purposes, the underlying hardware should support at least VT-d and VT-i or AMD-V and AMD-Vi. Having said all of that, sometimes the easiest (or only way) to see what is supported is to check the BIOS.

Enable virtualization support in BIOS

NOTE: This is not strictly required for PV guests. However, it is highly recommended so that you have the widest number of options for virtualization modes once you get underway.

In order to support HVM guests we need to ensure that virtualization extensions are enabled in the BIOS. If you find your system doesn’t support these extensions you cannot use the hypervisor to virtualize unmodified operating systems. Paravirtualization will work fine though.

The virtualization options appear differently in different BIOS builds but often are referred to as “Enable Intel VT” for Intel chipsets, "Enable AMD-V" for AMD or simply “Enable Virtualization Technology”. Oftentimes this option can be found under the “Advanced Chipset Features” menu in the BIOS or by using Search if the BIOS supports that. It is worthwhile digging around on this a bit. The options may be specified individually, for example: VT-x and VT-d or AMD-V and AMD-IOMMU (aka AMD-Vi or AMD-RVI). You may even find one is enabled by default but the other is not!

Consult your motherboard documentation for more assistance in enabling virtualization extensions on your system.

Download and Burn the Debian Installer CD

You can find the most recent Debian ISO images at this URL:

   http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/

The netinst image is sufficient for our purposes.

Burn the ISO to disk using your computer's standard utilities. Linux has wodim (among others) or use the built in ISO burning feature in Windows.

Quick intro to Debian

Debian is a simple, stable and well supported Linux distribution. It has included Xen Project Hypervisor support since Debian 3.1 “Sarge” released in 2005.

Debian uses the simple Apt package management system which is both powerful and simple to use. Installing a package is as simple as the following example:

   apt-get install htop

where htop is the application desired to install.

Simple tasks such as configuring startup scripts, setting up the network etc are covered by this tutorial so don’t worry if you haven’t used Debian before!

Many popular distributions are based off of Debian and also use the Apt package manager, if you have used Ubuntu, Linux Mint or Damn Small Linux you will feel right at home.

Installing Debian

Boot the Debian Installer CD/DVD

Insert the Debian CD/DVD and configure the CDROM drive as your default boot device in the BIOS or use the system boot menu if your BIOS supports it (usually F12).

You should see a menu, choose the default “Install” option to begin the installation process. Install the system The Debian installer is very straight forward. Follow the prompts until you reach the disk partitioning section.

Choose advanced/custom, we are going to configure a few partitions here, one for /boot another for /, one more for swap and a final partition to setup as an LVM volume group for our guest machines.

First create the /boot partition by choosing the disk and hitting enter, make the partition 300MB and format it as ext2, choose /boot as the mountpoint.

Repeat the process for / but of course changing the mountpoint to / and making it 15GB or so large. Format it as ext3.

Create another partition approximately 1.5x the amount of RAM you have in size and elect to have it used as a swap volume.

Finally create a partition that consumes the rest of the diskspace but don’t format it or assign a mount point.

We should now have a layout that looks like this assuming your disk device is /dev/sda :

   sda1 - /boot 200MB
   sda2 - / 15GB
   sda3 - swap
   sda4 - reserved for LVM

When you reach the package selection stage only install the base system. We won’t require any GUI or other packages for this guide. (If you want to set up a graphical desktop environment in dom0, that's not a problem, but you may want to wait until after you've completed this guide to avoid complicating things.)

You can find out details of the Debian installation process from the Debian documentation.

Continue through the installer then reboot and login at the prompt as root.

If you've got any hardware you're not sure open source drivers are available for, you may want to install non-free firmware files via:

   apt-get install firmware-linux-nonfree

If this does not work straight away make sure your /etc/apt/sources.list has entries including non-free and perhaps contrib while you're at it, e.g. like this:

   deb http://some.debian.server.org/debian stretch main contrib non-free

Add the same to deb-src and the stretch/updates lines (changing 'stretch' to the name of the current Debian release if you're using some newer release).

Installing the Xen Project Software

We've still got a few more steps to complete before we're ready to launch a domU, but let's install the Xen Project software now and use it to check the BIOS settings.

The Debian Xen Project packages consist primarily of a Xen Project-enabled Linux kernel, the hypervisor itself, a modified version of QEMU that support the hypervisor’s HVM mode and a set of userland tools.

All of this can be installed via an Apt meta-package called xen-linux-system. A meta-package is basically a way of installing a group of packages automatically. Apt will of course resolve all dependencies and bring in all the extra libraries we need.

Let's install the xen-linux-system meta-package:

   apt-get install xen-system-amd64

Now we have a Xen Project hypervisor, a Xen Project kernel and the userland tools installed. When you next boot the system, the boot menu should include entries for starting Debian with the Xen hypervisor. One of them should be highlighted, to start Xen by default. Do that now, logging in as root again.

Next, let's check to see if virtualization is enabled in the BIOS. There are a few ways to do that.

The most comprehensive is to review the Xen section of dmesg created during the boot process. This will be your first use of xl, the very versatile Xen tool, which we will come back to shortly to create and manage domUs:

   xl dmesg

Included in the output will be references to the CPU flags set in the BIOS to enable virtualization: 'vmx' for Intel, 'svm' for AMD. It will also detail other hardware virtualization extensions: VT-d features, Hardware Assisted Paging (HAP), I/O Virtualization and so on.

Another way is to check the flags set in the CPU on boot:

   egrep '(vmx|svm|hypervisor)' /proc/cpuinfo 

egrep will return any line containing one or more of those same text fragments (vmx/svm or more recently, just 'hypervisor'). If nothing comes back and you think it should, you may wish to look through the flags yourself:

   cat /proc/cpuinfo

If the virtualization extensions don't appear, take a closer look at the BIOS settings. A few round-trips through the BIOS are often required to get all the bits working right.

Setup LVM storage for guests

LVM is the Linux Logical Volume Manager. It is a technology that allows Linux to manage block devices in a more abstract manner.

LVM introduces the concept of a “logical volume”, effectively a virtualized block device composed of blocks written to one or more physical devices. Unlike proper disk partitions, these blocks don’t need to be contiguous.

Because of this abstraction logical volumes can be created, deleted, resized and even snapshotted without affecting other logical volumes.

LVM creates logical volumes within what is called a volume group, which is simply a set of logical volumes that share the same physical storage, known as physical volumes.

The process of setting up LVM can be summarized as allocating a physical volume, creating a volume group on top of this, then creating logical volumes to store data.

Because of these features and superior performance over file backed virtual machines we recommend the use of LVM if you are going to store VM data locally.

Now lets install LVM and get started!

Install LVM:

   apt-get install lvm2

Now that we have LVM installed let's configure it to use /dev/sda4 as its physical volume

   pvcreate /dev/sda4

Ok, now LVM has somewhere to store its blocks (known as extents for future reference). Let's create a volume group called ‘vg0’ using this physical volume:

   vgcreate vg0 /dev/sda4

Now LVM is setup and initialized so that we can later create logical volumes for our virtual machines.

For the interested below is a number of useful commands and tricks when using LVM.

Create a new logical volume:

   lvcreate -n <name of the volume> -L <size, you can use G and M here> <volume group>

For example, creating a 100 gigabyte volume called database-data on a volume group called vg0.

   lvcreate -n database-data -L 100G vg0

You can then remove this volume with the following:

   lvremove /dev/vg0/database-data

Note that you have to provide the path to the volume here.

More on LVM on Debian here.

If you already have a volume setup that you would like to copy, LVM has a cool feature that allows you to create a CoW (copy on write) clone called a snapshot. This means that you can make an "instant" copy that will only store the changes compared to the original. There are a number of caveats to this that will be discussed in a yet unwritten article. The most important thing to note is that the "size" of the snapshot is only the amount of space allocated to store changes. So you can make the snapshot "size" a lot smaller than the source volume.

To create a snapshot use the following command:

   lvcreate -s /dev/vg0/database-data -ndatabase-backup -L5G

Once again note the use of the full path.

Setup Linux Bridge for guest networking

Next we need to set up our system so that we can attach virtual machines to the external network. This is done by creating a virtual switch within dom0. The switch will take packets from the virtual machines and forward them on to the physical network so they can see the internet and other machines on your network.

The piece of software we use to do this is called the Linux bridge and its core components reside inside the Linux kernel. In this case, the bridge acts as our virtual switch. The Debian kernel is compiled with the Linux bridging module so all we need to do is install the control utilities:

   apt-get install bridge-utils

Management of the bridge is usually done using the brctl command. The initial setup for our Xen bridge, though, is a "set it once and forget it" kind of thing, so we are instead going to configure our bridge through Debian’s networking infrastructure. It can be configured via /etc/network/interfaces.

Open this file with the editor of your choice. If you selected a minimal installation, the nano text editor should already be installed. Open the file:

   nano /etc/network/interfaces

(If you get nano: command not found, install it with apt-get install nano.)

Depending on your hardware you probably see a file pretty similar to this:

    auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet dhcp

This file is very simple. Each stanza represents a single interface.

Breaking it down, “auto eth0” means that eth0 will be configured when ifup -a is run (which happens at boot time). This means that the interface will automatically be started/stopped for you. ("eth0 is its traditional name - you'll probably see something more current like "ens1", "en0sp2" or even "enx78e7d1ea46da")

“iface eth0” then describes the interface itself. In this case, it specifies that it should be configured by DHCP - we are going to assume that you have DHCP running on your network for this guide. If you are using static addressing you probably know how to set that up.

We are going to edit this file so it resembles such:

    auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet manual
    
    auto xenbr0
    iface xenbr0 inet dhcp
         bridge_ports eth0

As well as adding the bridge stanza, be sure to change dhcp to manual in the iface eth0 inet manual line, so that IP (Layer 3) is assigned to the bridge, not the interface. The interface will provide the physical and data-link layers (Layers 1 & 2) only.

Now restart networking (for a remote machine, make sure you have a backup way to access the host if this fails):

    service networking restart

and check to make sure that it worked:

    brctl show

If all is well, the bridge will be listed and your interface will appear in the interfaces column:

    bridge name     bridge id               STP enabled     interfaces
    xenbr0          8000.4ccc6ad1847d       no              enp2s0

Bridged networking will now start automatically every boot.

If the bridge isn't operating correctly, go back and check the edits to the interfaces file very carefully.

Reboot before continuing. During the reboot, note the list of OS choices and check to see what the default start-up choice is. "Debian GNU/Linux, with Xen hypervisor" (or equivalent) means Xen is loading by default. If both the start-up default is fine, skip the next section and go directly to Basic Xen Project Commands.

Configure GRUB to start Xen Project

GRUB, the bootloader installed during installation, tells the computer which operating system to start and how. To use the hypervisor, Xen must be started before the operating system. Depending on your use-case for Xen, you may or may not want GRUB to default to loading Xen.

GRUB2's configuration is stored in the file /boot/grub/grub.cfg

We aren’t going to edit this file directly, as it changes every time we update our kernel. Debian configures GRUB for us using a number of automated scripts that handle upgrades etc, these scripts are stored in /etc/grub.d/* and can be configured via

   /etc/default/grub

To change the default operating system change the GRUB_DEFAULT line in that file (or add it if it's not already there).

   GRUB_DEFAULT=0

causes the first OS in the boot list to be the default, and so on. So, if Xen was, say, the third OS choice, change the line to

   GRUB_DEFAULT=2

to have Xen load by default.

Then regenerate the /boot/grub/grub.cfg file by running:

   update-grub

At the next reboot confirm the correct default boot option is selected.


Basic Xen Project Commands

Before we dive into creating some guest domains we will quickly cover some basic commands. In the examples below, we use xl command line tool. Older versions of the Xen Project software used the xm command line tool. xl and xm are command line compatible (the format of the output may be slightly different). If, for example, you come across "xm" while reading old documentation, say, just substitute "xl".

Lets start with simple stuff!

   xl info

returns the information about the hypervisor and dom0 including version, free memory etc.

   xl list

lists running domains, their IDs, memory, state and CPU time consumed

   xl top

shows running domains in real time and is similar to the “top” command under Linux. This can be used to visualize CPU, memory usage and block device access.

We will cover some more commands during the creation of our guest domains.

See also:

Creating a Debian PV (Paravirtualized) Guest

PV guests are notoriously “different” to install. Fortunately, though, there are tools that help us prepare “images” or snapshots of the operating systems to run inside guest domains.

Debian contains a number of tools for creating Xen Project guests, the easiest of which is known as xen-tools. This software suite manages the downloading and installing of guest operating systems including both Debian and RHEL based DomUs. In this guide we are going to use xen-tools to prepare a Debian paravirtualized domU.

xen-tools can use LVM storage for storing the guest operating systems. In this guide we created the volume group “vg0” in the Setting up LVM Storage section.

When guests are paravirtualized there is no “BIOS” or bootloader resident within the guest filesystem and for a long time guests were provided with kernels external to the guest image. This however is bad for maintainability (guests cannot upgrade their kernels without access to the dom0) and is not as flexible in terms of boot options as they must be passed via the config file.

The Xen Project community wrote a utility known as pygrub which is a python application for PV guests that enables the dom0 to parse the GRUB configuration of the domU and extract its kernel, initrd and boot parameters. This allows for kernel upgrades etc inside of our guest machines along with a GRUB menu. Using pygrub or the stub-dom implementation known as pv-grub is best practice for starting PV guests. In some cases pv-grub is arguably more secure but as it is not included with Debian we won’t use it here though it is recommended in production environments where guests cannot be trusted.

Apart from this PV guests are very similar to their HVM and physical OS counterparts.

Configuring xen-tools and building our guest

First lets install the xen-tools package:

   apt-get install xen-tools

We can now create a guest operating system with this tool. It effectively automates the process of setting up a PV guest from scratch right to the point of creating config files and starting the guest. The process can be summarized as follows:

  • Create logical volume for rootfs
  • Create logical volume for swap
  • Create filesystem for rootfs
  • Mount rootfs
  • Install operating system using debootstrap (or rinse etc, only debootstrap covered here)
  • Run a series of scripts to generate guest config files like fstab/inittab/menu.lst
  • Create a VM config file for the guest
  • Generate a root password for the guest system
  • Unmount the guest filesystem

These 9 steps can be carried out manually but the manual process is outside the scope of this guide. We instead will execute the below command (for --dist you could in place of Wheezy e.g. use Squeeze, or even Precise or Quantal for a Ubuntu install):

  xen-create-image --hostname=tutorial-pv-guest \
  --memory=512mb \
  --vcpus=2 \
  --lvm=vg0 \
  --dhcp \
  --pygrub \
  --dist=wheezy

This command instructs xen-create-image (the primary binary of the xen-tools toolkit) to create a guest domain with 512MB of memory, 2 vcpus, using storage from the vg0 volume group we created, use DHCP for networking, pygrub to extract the kernel from the image when booted and lastly we specify that we want to deploy a Debian Wheezy operating system.

This process will take a few minutes. Once it is complete, it will provide a summary of the installation. Take note of the root password for the guest.

Also see

Creating a PV (Paravirtualized) Guest manually

Not every distribution provides the xen-tools package for an automated PV creation and configuration, and some Xen users prefer more control of the setup process.

Alpine Linux is such a distro. It provides detailed instructions on installing and starting a PV domU manually, using PVGRUB2 rather than pygrub.

It also provides valuable information regarding the startup options available.

Starting a console guest

To start the guest, run (as root or using sudo):

  xl create -c /etc/xen/tutorial-pv-guest.cfg

The -c in this command tells xl that we wish to connect to the guest virtual console, a paravirtualized serial port within the domain that xen-create-image configured to listen with a getty. This is analogous to running:

  xl create /etc/xen/tutorial-pv-guest.cfg && xl console tutorial-pv-guest

You can leave the guest virtual console by pressing ctrl+] and re-enter it by running the “xl console <domain>” command.

You can later shutdown this guest either from within the domain or from dom0 with the following:

  xl shutdown tutorial-pv-guest

That completes our section on setting up your first paravirtualized domain! If you don’t have any interest in setting up a HVM domain, skip ahead to Starting a GUI guest, below.

Creating a Windows HVM (Hardware Virtualized) Guest

HVM guests are quite a bit different to their PV counterparts. Because they require the emulation of hardware there are more moving pieces that need to be configured etc.

The main point worth mentioning here is that HVM requires the emulation of ATA, Ethernet and other devices, while virtualized CPU and Memory access is performed in hardware to achieve good performance. Because of this the default emulated devices are very slow and we generally try to use PV drivers within HVM domains. We will be installing a set of Windows PV drivers that greatly increase performance once we have our Windows guest running.

This extra emulation is provided by QEMU which will have been installed along with the Xen software.

To set up the HVM domU, we need to create a logical volume to store our Windows VM hard disk, create a config file that tells the hypervisor to start the domain in HVM mode and boot from the DVD in order to install Windows.

First, create the new logical volume - name the volume "windows", set the size to 20GB and use the volume group vg0 we created earlier.

   lvcreate -nwindows -L20G vg0

Next open a new file with your text editor of choice:

   nano windows.cfg

Paste the config below into the file and save it, NOTE this assumes your Windows iso is located in /root/ with the filename windows.iso. In the kernel = line below, be sure the xen version number matches your installation (e.g. xen-4.11, not 4.0):

   kernel = "/usr/lib/xen-4.0/boot/hvmloader"
   type='hvm'
   memory = 4096
   vcpus=4
   name = "ovm-1734"
   vif = ['bridge=xenbr0']
   disk = ['phy:/dev/vg0/windows,hda,w','file:/root/windows.iso,hdc:cdrom,r']
   acpi = 1
   device_model_version = 'qemu-xen'
   boot="d"
   sdl=0
   serial='pty'
   vnc=1
   vnclisten=""
   vncpasswd=""

The vnclisten= line specifies valid VNC connection addresses. vnclisten="127.0.0.1" will limit connections to the local machine. vnclisten="0.0.0.0" will accept unauthenticated remote connections from anywhere so is not suitable except in a secure network.

Start the guest as described below in Starting a GUI guest and proceed with Windows' installation.

Once you have installed Windows by formatting the disk and by following the prompts the domain will restart - however this time we want to prevent it booting from DVD so destroy the domain with

   xl destroy windows

Then change the boot line in the config file to read boot="c"' restart the domain with

   xl create windows.cfg

Reconnect with VNC and finish the installation. When this process is complete you should then proceed to download the GPLPV drivers for Windows by James Harper.

Installing PV drivers for HVM guests

Signed drivers can be obtained from Univention's website.

Many thanks for Univention for making signed drivers available to the Xen Project community and of course a massive thanks to James for all his work on making Windows in guest VMs such a smooth experience.

On finalizing the installation and rebooting you should notice much improved disk and network performance and the hypervisor will now be able to gracefully shutdown your Windows domains.

Another slightly different version of James Harper's drivers can be found here.

Starting a GUI guest (with VNC server)

Here is the command to start the domain and connect to it via VNC from your graphical machine.

   xl create windows.cfg

The VNC display should be available on port 5900 of your dom0 IP, for instance using gvncviewer:

   gvncviewer <dom0-ip-address>:5900

If this does not work try it without the port number and if you are trying from a GUI on dom0, try specifying localhost instead of the dom0 ip:

   gvncviewer localhost

Conclusion

That concludes our introduction to the Xen Project software, by now you can setup both PV and HVM domains on a bare dom0 hypervisor!

You can now move onto building your own guest images or try out some prebuilt Guest VM Images.