Xen Common Problems

Needs Refactor

This page has some issues with syntax and formatting: [[[XenConfigurationFileOptions]]]should be XenConfigurationFileOptions, etc. The page will also benefit from being broken into smaller FAQs along topics and merged with XenFAQ and XenFAQ2.

Answers to some common problems and questions about Xen

I'd like to contribute to the Xen wiki pages, I have something to add/edit/fix, how can I do it?

Contributions are very welcome! You should create yourself a wiki account, and then send an email to Todd Deshane asking for edit permissions. Please include your Xen wiki username in the email. Email address for Todd Deshane is todd dot deshane at xen dot org.

I'd like to read some kind of overview about Xen

See the [[[XenOverview]]] wiki page. Also check the following PDF documents:

Where can I read about the history of Xen?

Please see: http://www.xen.org/community/xenhistory.html

I'm interesting in being a Xen developer! Are there projects to work on?

Yes! Please check the XenDevelopmentProjects wiki page for more information!

How does Xen compare to KVM? Which one is better?

Please see this blog post for more information: http://blog.xen.org/index.php/2010/05/07/xen-%E2%80%93-kvm-linux-%E2%80%93-and-the-community/

Also check these documents:

Is there a "Best Practices" document available?

See the [[[XenBestPractices]]] wiki page.

Where can I find a list of available Xen dom0 kernels?

See the [[[XenDom0Kernels]]] wiki page.

Where can I find a list of available features in different Xen kernels?

See the XenKernelFeatures wiki page.

I have problems getting Xen or dom0 kernel to boot, how can I set up a serial console to log and troubleshoot the boot process?

See the [[[XenSerialConsole]]] wiki page.

Where can I find a list of all the available domU configuration options in /etc/xen/<guest> cfgfile?

See the [[[XenConfigurationFileOptions]]] wiki page.

Is there a list of all available Xen hypervisor (xen.gz) commandline boot options for grub.conf?

Yes, please see the XenHypervisorBootOptions wiki page.

Can I passthrough an USB device connected to dom0 to a Xen guest?

Yes, please see the XenUSBPassthrough wiki page.

Can I passthrough a PCI device to Xen guest?

Yes, please see the XenPCIpassthrough wiki page.

Can I passthru a VGA graphics adapter to Xen guest?

Yes, please see the XenVGAPassthrough wiki page.

I have problems using my graphics card in Xen dom0, with the pvops dom0 kernel.. any tips?

Yes, please see the XenPVOPSDRM wiki page.

Is there more information about Xen PVSCSI passthrough functionality?

Yes, please see the XenPVSCSI wiki page.

Is there more information about Xen "blktap" disk backend?

Yes, Xen actually has two blktap backends:

blktap1: please see the blktap wiki page for more information.
blktap2: please see the blktap2 wiki page for more information.

blktap2 supports VHD images, and is considered much more robust than the old blktap1 qemu/qcow image file support. Xen 4.0 includes blktap2 support. Note that also the dom0 kernel needs to have the blktap2 driver.

Is there a list of various 3rd party management tools and (web) interfaces for Xen or XCP?

Yes, please see the XenManagementTools wiki page.

Is there more information about Xen Fault Tolerance, aka Remus Transparent High Availability (HA) for VMs?

Yes, please see the Remus wiki page for more information.

Where can I find optimized Xen PV-on-HVM drivers for Linux HVM (fully virtualized) guests?

Please see the XenLinuxPVonHVMdrivers wiki page.

Do you have a list of Xen related research papers?

Yes, please see http://www.xen.org/community/xenpapers.html

Where can I find information about Xen on RISC CPUs?

Please see XenPPC wiki page for Xen on PowerPC CPUs. XenPPC project is not active anymore.
Please see XenARM wiki page for Xen on ARM CPUs. Xen ARM project is actively developed.

What's the difference between Xen hypervisor (from xen.org) and Citrix XenServer or XCP?

Xen hypervisor from xen.org is the core hypervisor used in many different products. It includes basic command line management tools. It is distributed as a tarball and from mercurial source code repositories. You need to compile and install Xen hypervisor from sources, and combine it with a kernel of your choice. It can be thought as the "core engine" you can use to build your own virtualization platform.

Many Linux distributions package Xen and distribute it as prebuilt binaries, combined with Xen capable kernels of their choice. There are many additional thirdparty management tools available to manage the Xen hypervisor.

There are multiple companies shipping products based on the Xen hypervisor, including Citrix, Oracle, Novell, Redhat, and others.

Citrix XenServer is a commercial product that includes the core Xen hypervisor from xen.org, combined with CentOS-based dom0 distro, management toolstack, and everything else required to build a ready made virtualization platform. Citrix XenServer is a dedicated virtualization platform (not a general purpose Linux distro), and it is shipped as a ready to install ISO-image that contains everything you need out-of-the-box. Citrix XenServer includes the XAPI management toolstack (which not included in the core Xen hypervisor from xen.org), allowing you to pool multiple Xen hosts together and manage them centrally using either the graphical Citrix XenCenter management tool, the 'xe' commandline tool or using the XenAPI directly from your scripts. Citrix XenCenter graphical management tool is only available for Windows.

Xen Cloud Platform (XCP) was announced in 2009, and it is the opensourced version of Citrix XenServer, including the XAPI management toolstack. XCP and XAPI are now developed on xen.org repositories. XCP is shipped as ready to install ISO-images, just like Citrix XenServer. Citrix is basing their new Citrix XenServer releases on XCP. XCP includes the XenAPI provided by the XAPI toolstack and the 'xe' commandline tool. XCP does not include Citrix XenCenter and Citrix XenCenter cannot be used to manage XCP. XCP users can use the opensource OpenXenManager instead, which runs on Linux. See http://www.xen.org/products/cloudxen.html for more information about XCP.

How can I check the version of Xen hypervisor?

Run "xm info" in dom0. You can find the hypervisor version in "xen_major", "xen_minor" and "xen_extra" fields. The version is major.minor.extra.

Determining the version from within a domU is dependent on the guest operating system. For Linux guests:

$ dmesg | grep Xen\ version
Xen version: 3.4.2 (preserve-AD)

How can I check if I'm able to run Xen HVM (fully virtualized) guests?

Xen requires CPU virtualization support/extensions (Intel VT, AMD-V) for running HVM guests (=Windows). Virtualization support needs to be enabled in the system BIOS. Intel calls this feature also as "VT-x". Also note that Intel "VT-x" is different from "VT-d", which is a totally different feature (see the next section below).

NOTE that Linux dom0 kernel doesn't see 'vmx' or 'svm' CPU flags in "/proc/cpuinfo" because Xen *hypervisor* (xen.gz) is using the hardware virtualization features and hiding the flags from dom0! Xen dom0 is actually a virtual machine, so it doesn't see all the cpu flags as Xen hypervisor is hiding some flags from dom0.

You can run "xm info" in dom0 and check from the 'xen_caps' line if Xen is able to run hvm guests. Also Xen hypervisor boot messages in "xm dmesg" show if hardware virtualization (HVM) is enabled or disabled.

Example "xm dmesg | grep -i hvm" output for an Intel system where HVM is supported by the CPU:

(XEN) HVM: VMX enabled

Example "xm dmesg" output for an Intel system where HVM is supported by the CPU but it's disabled in the system BIOS:

(XEN) VMX disabled by Feature Control MSR.

Example "xm dmesg | grep -i hvm" output for an AMD system where HVM is supported:

(XEN) HVM: SVM enabled

You can also see the HVM support status from the "xen_caps" line on Xen hypervisor "xm info | grep xen_caps" output:

xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64

If your CPU supports Hardware Assisted Paging (HAP, Intel EPT or AMD NPT) you'll see a line like this in "xm dmesg" output:

(XEN) HVM: Hardware Assisted Paging detected and enabled.

How can I check if my hardware has an IOMMU?

Hardware IOMMU (Intel VT-d) is required for hardware assisted PCI passthru (I/O virtualization) to Xen HVM (fully virtualized) guest VMs. You can check for IOMMU status from "xm dmesg" output.

Example "xm dmesg" output for a system where IOMMU is supported:

(XEN) I/O virtualisation enabled

Example "xm dmesg" output for a system where IOMMU is not supported:

(XEN) I/O virtualisation disabled

You might also see additional output like:

(XEN) Intel VT-d Snoop Control supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping not supported.

And if hardware IOMMU is used for PV guest PCI passthru or not:

(XEN) I/O virtualisation for PV guests enabled

Xen doesn't require hardware IOMMU for PCI passthru to PV guests, but having an IOMMU makes it more secure. PCI passthru to Xen HVM guests requires hardware IOMMU.

Xen complains about "hotplug scripts not working"

This problem is often related to udev. Do you have udev installed? Is your udev the correct/supported version? This error usually has more information in the end revealing the real reason.. for example:

Error: Device 0 (vif) could not be connected. Could not find bridge device br0.

Which means exactly what it says.. the guest is configured to to use a bridge called "br0", but there's no such bridge in dom0. Run "brctl show" to verify what bridges you have. Create the missing bridge, or edit the VM cfgfile and make it use another (correct) bridge.

Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

This problem is often caused by not having "xen-netback" driver loaded in dom0 kernel.

The hotplug scripts are located in /etc/xen/scripts by default, and are labeled with the prefix vif-*. Those scripts log to /var/log/xen/xen-hotplug.log, and more detailed information can be found there.

Error: Device 5632 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/2/5632 state: 1"

This problem can be caused by not having "xen-blkback" driver loaded in dom0 kernel.

So when dealing with these problems always check you have all the required xen backend driver modules loaded in dom0 kernel! netbk or xen-netback for networking, and blkbk or xen-blkback for block devices (virtual disks).

Also read the full "xm log", it most probably includes additional information about the problem and the error messages.

I upgraded PV domU from old Xenlinux kernel to new pvops kernel, and now the domU doesn't work anymore!

Many people seem to upgrade their Xen guest PV kernels at the same time they upgrade from older Xen version to Xen 3.4.x or Xen 4.0.x. This is actually not a required step! You can still use your old domU kernels with the new Xen version, and with a new dom0 kernel version. Xen hypervisor is the compatibility layer, so dom0 and domUs can have totally different kernels.

Old Xenlinux PV kernels (linux-2.6.18-xen, ubuntu 2.6.24, debian 2.6.26, sles11 2.6.27, sles11 sp1 2.6.32, opensuse 2.6.31 etc) have different device names for virtual disks and virtual console than the new pvops (upstream kernel.org) Linux kernels. The old device names in Xenlinux kernels usually are:

/dev/sdX for virtual disks, for example /dev/sda
/dev/xvc0 for the virtual console

The new device names in pvops kernels are:

/dev/xvdX for virtual disks, for example /dev/xvda. XVD = Xen Virtual Disk.
/dev/hvc0 for the virtual console

So you need to do changes in the guest for it to boot up and work properly with the new kernel:

Fix "/etc/xen/<guest>" configuration file and change the virtual disks from /dev/sdX to /dev/xvdX
Fix domU kernel root filesystem parameter, ie. change root=/dev/sda1 to root=/dev/xvda1
Fix domU kernel console parameter, ie. have "earlyprintk=xen console=hvc0" in the extra="" section.
If you use pygrub to load the kernel/initrd from the guest filesystem then you need to make the changes above in the domU /boot/grub/grub.conf (or menu.lst).
Make sure the domU kernel has xen-blkfront driver built-in, or if it's built as a module (most common option), then you need to have a proper initrd (ramdisk) image for the domU kernel that actually loads the xen-blkfront driver so that the guest kernel can access the root filesystem! Failing to do this will cause the domU kernel to crash on boot because it can't find the root filesystem!
Note that you usually can't use the dom0 initrd image for a domU! dom0 initrd is generated for *dom0*, so it tries to load all the drivers for the *physical* hardware, while you have only virtual hardware in the domU! initrd ramdisk images are system specific, where dom0 and domU are different systems. Usually it's easiest to build an initrd/initramfs image for the domU in the *domU*. So boot the domU up with the old kernel, and generate an initrd image for the new pvops kernel. Make sure the new generated initrd-image loads xen-blkfront driver!
You need set up (or modify) a getty for the new console device in the domU init/inittab settings, so you can get a login prompt on the "xm console" session.
You need to add the console device name to "/etc/securetty" in the domU so root is allowed to login from the console.
If you're having networking problems with the new pvops domU kernels make sure you have "xen-netfront" driver loaded in the new kernel. It's usually built as a module in the recent pvops/upstream/distro kernels.

Also see Migrate_from_Linux_2.6.18_to_2.6.31_and_higher wiki page for more information.

I can't connect to the console of my guest using "xm console <guest>"

Do you have "xenconsoled" process running in dom0? If you do, did you try killing it and restarting it? Also, only one console session per domU can exist at a time. Currently, attempting to use more than one console session per domU will not raise an error, but will result in strange behavior.

Console of my PV guest shows kernel boot messages and then it stops and doesn't work anymore

Do you have a getty configured for the console device in the guest? You usually need to add an entry to /etc/inittab in the guest to make the guest present login prompt on the "xm console" session. Is the getty configured for a correct console device? See the chapter below for correct guest console devices.

Example "/etc/inittab" entry for Debian Lenny (linux-image-2.6.26-2-xen) guest, write this on a new (last) line:

vc:2345:respawn:/sbin/getty 38400 hvc0

Example "/etc/inittab" entry for CentOS 5 (kernel-xen-2.6.18) guest:

co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav

After modifying /etc/inittab in the guest, run "kill -1 1" to make init (process id 1) reload the configuration file on-the-fly, or reboot the guest. This should make the console work and login prompt appear on the "xm console" session.

NOTE! Some distros (like newer Ubuntu) don't use "/etc/inittab" anymore, but instead you have to configure the gettys in other places!

Ubuntu 6.06 LTS is the last ubuntu version to use "/etc/inittab" configuration file. Ubuntu 6.10 to 9.04 use "/etc/event.d/*" configuration files to configure init. Ubuntu 9.10 and newer use "/etc/init/*.conf".

Example "/etc/init/hvc0.conf" entry for Ubuntu 10.04 Xen PV (linux-image-2.6.32) guest:

# hvc0 - getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn exec /sbin/getty -L hvc0 9600 linux

Console of my PV guest is totally empty, it doesn't show anything, not even kernel boot messages!

This usually means your PV (paravirtual) guest is using a newer pv_ops (upstream Linux kernel.org) kernel that has different console device name. You need to add "console=hvc0" to the guest kernel command line options.

If you're using Xen pygrub or pv-grub to load the kernel from guests filesystem, then you need to configure the guest kernel settings in the guest /boot/grub/grub.conf (or menu.lst), not in dom0! So edit the guest /boot/grub/grub.conf (or menu.lst) and add the "console=hvc0" option there for the default kernel.

If you're not using pygrub or pv-grub, aka you're configuring everything in /etc/xen/<guest> configuration file in dom0, and you have the kernel/initrd in dom0 filesystem, then you can specify the additional "console=hvc0" parameter in the /etc/xen/<guest> configuration file in dom0, on the extra="" line.

What's the correct console device name for my Xen PV guest

Xenlinux ("xenified") kernels use "xvc0" as the console device.
Upstream kernel.org pv_ops kernels use "hvc0" as the console device

Examples of Xenlinux kernels using "xvc0" console:

xen.org linux-2.6.18-xen
RHEL5 / CentOS5 kernel-xen 2.6.18
Debian Etch 2.6.18-6-xen
Debian Lenny 2.6.26-2-xen
Ubuntu 8.04 LTS "hardy heron" 2.6.24 xen kernel
Gentoo Xenlinux kernels
Novell SLES 10 and SLES 11 xen kernels
OpenSUSE xen kernels
Various "xenified" forward-ports of the Novell SLES/OpenSUSE Xenlinux patches (2.6.27, 2.6.29, 2.6.30, 2.6.31, 2.6.32, 2.6.33, 2.6.34)

Examples of pv_ops domU kernels using "hvc0" console:

All upstream/mainline kernel.org Linux kernels since 2.6.26.
Fedora 10, Fedora 11, Fedora 12, Fedora 13 kernels
Ubuntu 8.10 2.6.27 server-kernel
Ubuntu 9.04 2.6.28 linux-image-server
Ubuntu 9.10 2.6.31 ec2-kernel
Ubuntu 10.04 ("Lucid Lynx") 2.6.32 kernel
Debian 6.0 ("Squeeze") 2.6.32 kernels

How do I exit domU "xm console" session

Press ctrl+] or if you're using Putty press ctrl+5.

Everything seems OK but I still can't access the domU "xm console"!

A couple of things to check:

Do you have some old already running "xm console" session running on the host? possibly stuck in some dead ssh connection?
Do you have virt-manager running on the host 'reserving' the guest console?
Did it work earlier?
Do you have problems with only one guest, or with all guests?
Does shutting down and then re-starting the guest help?
Are you really sure you are running a getty in the guest for the correct console device? :)
Is the guest really up and running? Can you access it over the network with ssh? Can you ping it?

Xen network-bridge script not working in Debian Lenny, errors about eth0

You edited "/etc/xen/xend-config.sxp" in your Debian Lenny dom0/host, and changed the default "network-dummy" to "network-bridge". When you reboot dom0, and xend is started, it gives error like this:

ifdown: interface eth0 not configured

RTNETLINK answers: Device or resource busy

This error can be caused by non-common settings for eth0 in "/etc/network/interfaces" configuration file. Try cleaning up the configuration, and having only a very basic ipv4-only configuration with a static IP, to begin with. Remove IPV6 configuration. Remove all the IP aliases. Reboot and try again. First get it working, then possibly later modify the configuration to be more advanced.

Xend does not start when using pv_ops dom0 kernel?

You need to have Xen event channel (evtchn) driver included in your dom0 kernel for xend to be able to start. If Xen event channel support is compiled as a module, use "lsmod" to check that you have xen-evtchn (or xen_evtchn) driver module loaded. It not, use "modprobe" to load it. If Xen event channel support is statically built-in to the dom0 kernel then you don't need to load any modules.

You also need xen-gntdev driver loaded.

In December 2009 pv_ops dom0 kernel modules were renamed to have a "xen-" prefix in them, ie. "evtchn.ko" became "xen-evtchn.ko", and "gntdev.ko" became "xen-gntdev.ko".

This makes Xen 3.4.x xend fail to start, because it tried to load "evtchn.ko", but that doesn't exist. You need to load "xen-evtchn.ko" and then start xend. Fedora 12 xen-3.4.2-2 rpms have this problem fixed. If your xend does not automatically load evtchn or xen-evtchn driver module, please load it manually with modprobe.

Also make sure you have xenfs mounted to "/proc/xen", that's needed aswell. Do you have files under "/proc/xen/" directory? You should have for example:

# ls /proc/xen/
capabilities  privcmd  xenbus  xsd_kva  xsd_port
# cat /proc/xen/capabilities
control_d

"control_d" value in "/proc/xen/capabilities" means you're in Xen dom0 ("control domain").

Do you have files under "/dev/xen/" ? You should have there at least the following device nodes:

# ls -la /dev/xen
total 0
drwxr-xr-x  2 root root     80 Jun 19 19:17 .
drwxr-xr-x 20 root root   4580 Jun 19 19:33 ..
crw-rw----  1 root root 10, 58 Jun 19 19:17 evtchn
crw-rw----  1 root root 10, 57 Jun 19 19:17 gntdev

If you're missing those device nodes you can use "mknod" to create them manually. Creating those device nodes should happen automatically by udev when you have xen-evtchn and xen-gntdev driver modules loaded.

You can see the correct major/minor for the device nodes from "/proc/misc":

# cat /proc/misc
 57 xen/gntdev
 58 xen/evtchn

Note how they match the values from the previous "ls -la /dev/xen". If the major/minor numbers don't match then xenstored and xend won't be able to start/function. Note the major/minor numbers for device nodes are dynamic nowadays, so you need to verify they're properly set up for your system.

How can I limit the number of vcpus my dom0 has?

The recommended way is to use "dom0_max_vcpus=X" boot time option for Xen hypervisor (xen.gz) in grub.conf. You need to reboot after modifying grub.conf. Using this method makes sure you allocate resources only for the actual number of vcpus you need in dom0.

Another option is to configure "(dom0-cpus X)" option in /etc/xen/xend-config.sxp, but it works in a different way: dom0 boots up with all the cpus visible to it, and when xend starts, it'll hot-remove the extra vcpus from dom0, and only leaves the number of vcpus specified in (dom0-cpus) to dom0. This option has the benefit that vcpus can be later added to dom0, if needed.

Can I dedicate a cpu core (or cores) only for dom0?

Yes, you can. It might a good idea especially for systems running IO intensive guests. Dedicating a CPU core only for dom0 makes sure dom0 always has free CPU time to process the IO requests for the domUs. Also when dom0 has a dedicated core there are less CPU context switches to do, giving better performance.

Specify "dom0_max_vcpus=X dom0_vcpus_pin" options for Xen hypervisor (xen.gz) in grub.conf and reboot.

After rebooting you can verify with "xm vcpu-list" that dom0 vcpus are pinned to only use the matching physical cpus.

The next step is to configure all the guests (domUs) to NOT use those same physical cpus. This can be done by specifying for example cpus="2-7" to all /etc/xen/<guest> cfgfiles. This example would leave physical cpus/cores 0 and 1 only for dom0, and make the guests use cpus/cores 2 to 7.

Booting Xen with GRUB2 fails?

It seems GRUB2 destroys the first parameter from Xen hypervisor "multiboot" and dom0 kernel "module" lines. For dom0 kernel this usually means root= parameter gets lost and thus dom0 kernel fails to set up the root filesystem and the boot fails. For xen.gz this usually means parameters like "dom0_mem=" or "dom0_vcpus" will get lost, whatever is the first parameter.

You can work around this problem by specifying "dummy=dummy" as the first parameter in grub2 config file for "multiboot" and "module" lines. This will fix the problem with Xen 4.0.0 and older hypervisor versions.

See this link for a working GRUB2 + Xen example:http://old.nabble.com/Strange-interaction-from-grub2-and-XEN-td26464067.html

The dummy workaround is not required anymore with Xen 4.0.1 and newer versions. See this patch for the Xen GRUB2 fix: http://xenbits.xen.org/xen-unstable.hg?rev/4207549948a4 and http://xenbits.xen.org/xen-4.0-testing.hg?rev/1da5224a2875 .

Other things that could go wrong. Did you modify the config file manually? If yes, you might need to add "insmod multiboot" (or "insmod multiboot2")

32bit Xen hypervisor crashes when using GRUB2

You get the following messages on the (serial) console when GRUB2 boots 32bit PAE Xen hypervisor:

(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Cannot access memory beyond end of bootstrap direct-map area
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Unknown interrupt (cr2=00000000)

This is because your version of GRUB2 dumps the dom0 kernel and initrd image to a memory region that Xen cannot access. Changing to different GRUB2 version fixes the problem. Another option is to switch to 64 bit Xen hypervisor. Note that you can still run 32bit dom0 on 64bit Xen hypervisor.

This problem has been fixed in Xen 4.0.0-rc8 and newer versions. You can use new enough 32bit PAE Xen hypervisor with GRUB2. See these changesets for patches: http://xenbits.xen.org/xen-unstable.hg?rev/bcc09eb7379f and http://xenbits.xen.org/xen-unstable.hg?rev/377433a77d70

In dom0 how can I access and mount partitions inside a Xen disk image?

The easiest way is to use "kpartx" in dom0. It allows you to easily access all the partitions inside an image file or inside LVM volume.

You can use "kpartx -l /path/guestdisk" to list the partitions the image has, and "kpartx -a /path/guestdisk" to add the partition mappings. After the device mapper (dm) mappings have been added, you can access the partitions using "/dev/mapper/guestdiskpX" block devices. There's a separate block device for each partition. You can mount, copy, format, or whatever you need to do and it works just like for "real" partitions.

When you're done using the partitions in dom0, and you have unmounted them, you have to remove the kpartx mappings by running "kpartx -d /path/guestdisk". It's important to do this before using the image again for other purposes or starting the guest again! If you forget to remove the mappings the image might get corrupted when you access the image with other tools or start the guest.

Another neat way which works for me, is to simply attach the block device to dom0 as you would to a domU. e.g.:

xm block-attach Domain-0 file:/disk/image/for/domU xvda w

I have found this to work for OSs which use slices (such as Solaris and *BSD), and are not picked up using kpartx. You should check dmesg and see which device nodes were created for you. Once you are done working with the disk image you can:

xm block-detach Domain-0 xvda

If your domU is LVM-based, see Access_Disk_from_DomUs_when_Xen_was_installed_with_LVM

Xen dom0 complains about not enough free loop devices when trying to start a new domU or when trying to "mount -o loop" from the cmdline

Linux "loop" module has max 8 loop devices (/dev/loop*) as a default. Every Xen "file:" backed domU disk uses one loop device in dom0, and if you do "mount -o loop disk.img /mnt" in dom0 that uses a loop device aswell.

You can check all the loop devices in use by running "losetup -a". You can detach a loop device (free it) by running "losetup -d /dev/loopX" when it's not mounted or otherwise in use.

You can increase the amount of available loop devices by loading the linux "loop" module with an option "max_loop=X", for example "modprobe loop max_loop=128". You can add "max_loop=X" option to /etc/modprobe.conf, /etc/modules (or whatever cfgfile your dom0 distribution uses for module options) to make it boot time default.

You can also use Xen "phy:" backed LVM volumes instead of disk images. "phy:" doesn't require loop devices.

As an example for Debian Lenny check this: http://www.howtoforge.com/virtualization-with-xen-on-debian-lenny-amd64

Such an error message may also be indicative of other issues (such as the guest crashing and restarting so quickly that xend does not have time to free the older loopback device).

Debian Lenny Xen PV domU doesn't start, in the console it says "Waiting for root file system" and then drops to a busybox shell

Is your root device set correctly for the guest (domU) kernel? Most probably you're using a wrong root= parameter, and thus the system cannot mount the root device and start. Check the early boot messages from the kernel, there should be a line starting with "Command line:". There you can see the parameters passed to the domU kernel. Do you see the correct root device there?

Remember the root= passed to a PV domU must be a path accessible from within the guest

You can set and edit the root device in dom0 from /etc/xen/<guest> configuration file. root device is specified either in "root=" line, or on "extra=" line. The configured root device should match the disk configuration for the guest.

Another problem can be that you're using wrong initrd image for the domU kernel. You should use an initrd generated for the guest, not the same initrd as you use to boot the dom0.

My Xen PV guest kernel crashes, how can I debug it and get a stack/call trace?

Edit "/etc/xen/<guest>" cfgfile, and set:

on_crash="preserve"

Then when your domU guest crashes, run this in dom0:

/usr/lib/xen/bin/xenctx -s System.map-domUkernelversion <domUid> <vcpu-number>

If you're running 64bit dom0, then xenctx might be under "/usr/lib64/". This command will give you a stack trace of the crashed domU kernel on the specified vcpu, allowing you to see where it crashes. You should get the stack trace for every vcpu your guest has! Vcpu numbers start from 0.

Note that you need to use the "System.map" file from the exact kernel version the domU is running, otherwise the stack trace results are not correct!

Can I run graphical X applications in Xen dom0 without installing X server and display drivers?

Yes, you can. Many people prefer to not install any graphical drivers or X on dom0 for maximum stability. You can still run graphical applications using ssh X11 forwarding. You can run for example "virt-manager" in dom0 and display the GUI on your desktop/laptop over ssh.

1) Install "xorg-x11-xauth" package in dom0 (centos/rhel/fedora, "xauth" in debian/ubuntu, other distros might have different name for it).

2) Log in to dom0 from your Linux workstation like this: ssh -X root@dom0.

3) After the password is accepted you should notice ssh setting up the ssh X11 forwarding like this:

/usr/bin/xauth:  creating new authority file /root/.Xauthority

4) You can now run graphical X applications, for example vnc viewer, virt-viewer etc, and the GUI will be tunneled securely over ssh to your local X desktop.

If you're using Windows you can install "xming" and "putty". Start Xming, and after that enable X11 forwarding in putty settings (Connection -> ssh -> X11 -> Enable X11 forwarding) and then connect to your dom0.

What emulated NIC types/models are available in Xen HVM fully virtualized guests?

The following emulated network interface cards are available for Xen HVM guests in Xen 3.4:

e100
e1000
rtl8139
ne2k_pci
pcnet

Emulation is done by Qemu-dm running for each Xen HVM guest. Intel e1000 is known to be the best performing emulated NIC. Even faster is to use PV-on-HVM drivers, which totally bypasses emulation.

Older Xen versions might not have all the above NIC options available.

Can I set up Xen HVM Linux guest to display the kernel boot messages on "xm console" ?

Yes, you can. You need to add this line to the "/etc/xen/<hvmguest>" configuration file in dom0:

serial='pty'

And then in the HVM guest grub.conf (inside the HVM VM) configure kernel logging to the (virtual) serial port like this:

#============================================================
# display the grub kernel selector menu on the serial console
serial --unit=0 --speed=9600
terminal --timeout=5 serial console
# kernel entries title openSUSE 11.1 - 2.6.27.29-0.1
root (hd0,1)
kernel /boot/vmlinuz-2.6.27.29-0.1-default root=LABEL=ROOT resume=LABEL=SWAP splash=silent showopts console=tty1 console=ttyS0
initrd /boot/initrd-2.6.27.29-0.1-default
#================================================

So basicly you need to add "console=ttyS0" to the kernel line, and also configure grub to display the kernel selector on the (virtual) serial console using "serial" and "terminal" settings.

After these settings "xm console <hvmguest>" works just like it does for PV (paravirtual) guests and allows you to see grub and the Linux kernel boot messages of the HVM guest. You can also use other tools (minicom, screen, etc) in dom0 to access the VM console pty device directly.

Also remember to set up a getty in the guest for the serial console device, ttyS0, so that you can also login from the serial console! See above for tips about that.

How can I boot Xen HVM guest from a virtual emulated floppy, using an image file as the floppy?

This is the required configuration in "/etc/xen/<hvmguest>" cfgfile:

fda = '/path/to/floppy.img' boot=a

You can also use this cmdline method while creating the guest:

xm create /etc/xen/hvmguest.cfg fda=/path/to/floppy.img boot=a

Important note: Make sure SElinux access restrictions are not blocking access to /path/to/floppy.img!

How do I change virtual/emulated CD .iso image for a Xen HVM guest?

First check "/etc/xen/<guest>" cfgfile and check what is your cdrom device in the guest. It's usually "/dev/hdc".

An example to swap the emulated CD using an iso image:

xm block-configure <guest> file:/path/to/new/cd.iso hdc:cdrom r

Are there FreeBSD Xen PV kernels/images available?

Please check the following links: http://wiki.freebsd.org/AdrianChadd/XenImages and http://wiki.freebsd.org/AdrianChadd/XenHackery .

FreeBSD Xen wiki: http://wiki.freebsd.org/FreeBSD/Xen .

Tutorial how to install 32bit Xen PV FreeBSD domU on a Debian Lenny Xen dom0, using LVM volumes: http://silverwraith.com/blog/?p=83 . Another Freebsd Xen PV domU tutorial: http://forums.freebsd.org/showthread.php?t=10268 .

Where can I find information about NetBSD Xen support?

NetBSD supports both Xen dom0 and PV domUs. For more information see: http://www.netbsd.org/ports/xen/

What's Xen pygrub?

Pygrub allows you to simply specify in Xen PV guest "/etc/xen/<guest>" cfgfile:

bootloader = "/usr/bin/pygrub"

Which replaces all these options:

kernel = "vmlinuz-2.6.32.12"
ramdisk = "initrd-2.6.32.12.img"
root = "/dev/xvda1"
extra = "earlyprintk=xen console=hvc0"

When you start a Xen PV guest pygrub will be executed in dom0 and it'll access the guest disk configured in the "/etc/xen/<guest>" cfgfile, find "/boot/grub/grub.conf" (or menu.lst) from the guest filesystem, parse it, fetch the kernel, initrd image and kernel parameters configured in the guest to dom0, and then boot up the guest using those kernel + initrd + parameters. All the domU specific settings are configured inside the domU, in grub.conf.

Requirements for using pygrub for Xen PV domUs:

Pygrub binary available in dom0, installed together with Xen.
Pygrub configured for the guest in "/etc/xen/<guest>" cfgfile
Guest has /boot/grub/grub.conf (or menu.lst) with proper entries in it. There's actually no need to install the real grub to the MBR of the guest disk, it's enough to just have the grub.conf/menu.lst in the guest.
If the guest has GRUB2 configuration file (for example Ubuntu 10.04 or Debian Squeeze 6.0) then you need to have new enough Xen pygrub in dom0. Pygrub versions included in Xen 4.0.2 and 4.1.1 and newer properly support parsing GRUB2 guest config files.

So pygrub makes it much easier to:

Manage the guest kernels inside every guest, keeping the guest distribution package managers (dpkg,rpm) and package dependencies happy.
Upgrade guest kernels without touching dom0 or guest configuration files in "/etc/xen/" in dom0.
Each guest distribution can easily use the default kernel provided by the distribution.
Pygrub allows you to select which kernel to boot, in the "xm console <guest>" session. It's easiest to start the guest with "xm create -f /etc/xen/guest -c" to attach to the console immediate for selecting the kernel from pygrub menu. If nothing is chosen then the default entry will be booted automatically.

Management tools like "virt-install" and "virt-manager" will use Xen pygrub as a default when you install new guests using them.

What's Xen pvgrub?

Xen pvgrub is like pygrub but more secure.

Pygrub is executed in dom0, so it could have security concerns/issues, if bugs were found from it. Pvgrub is executed in the Xen PV guest, so there are no such security issues with it.

Pvgrub is included in Xen 3.3 and newer versions. Pvgrub is a separate build and separate tool, so usually it's easier to begin with pygrub, and then later switch to pvgrub.

See PvGrub wiki page for more information and usage of pvgrub.

Does Xen pygrub support booting PV guests using GRUB2 config files?

Yes, pygrub in Xen 4.0.1 supports guests using GRUB2 config files (Ubuntu 10.04 LTS "Lucid Lynx"). Later the format of GRUB2 config files was changed, and Xen 4.0.2 and 4.1.1 (or newer versions) properly support this newest version of GRUB2 config files (Debian 6.0 "Squeeze").

Also there's a patch for RHEL/CentOS 5.5 version of pygrub to support GRUB2 (Ubuntu 10.04) guests: https://bugzilla.redhat.com/show_bug.cgi?id=577511 and latest version of GRUB2 config files (Debian 6.0 "Squeeze"): https://bugzilla.redhat.com/show_bug.cgi?id=675733 . These RHEL5 Xen pygrub patches are included in RHEL5.6 (first patch, Ubuntu 10.04) and RHEL5.7 (second patch, Debian 6.0).

Can I browse the Xen source trees online?

Yes, you can browse the source tree and see the changelogs/summaries and track changes online from the XenRepositories wiki page.

Can I browse the Xen Qemu-dm (HVM guest ioemu) source repositories online?

Yes, you can see the changelogs/summaries and track changes online from the XenRepositories wiki page.

I'm using Xen 4.0.0 with the latest pvops 2.6.32.x dom0 kernel (2.6.32.15+) and xend/xenstored doesn't start anymore?

Yes, this is a known issue. pvops dom0 kernel in xen/stable-2.6.32.x branch changed to using proper evtchn and gntdev device node names ("/proc/xen/*") in June 2010, which triggers a bug in libxc in Xen 4.0.0. This bug has been fixed in Xen 4.0.1-rc2 and newer versions. So please upgrade your Xen to 4.0.1-rc2 or newer version.

Alternatively you can apply this patch if you want to keep using Xen 4.0.0: http://xenbits.xen.org/xen-4.0-testing.hg?rev/0e1521f654f2 .

Another option is to manually fix the device node major/minor numbers by re-creating the device nodes manually and then re-starting xenstored and xend. You can see the proper major/minor for "/dev/xen/evtchn" and "/dev/xen/gntdev" from "/proc/misc":

# cat /proc/misc | grep xen
57 xen/gntdev
58 xen/evtchn

# ls -la /dev/xen
total 0
drwxr-xr-x  2 root root     80 Jun 19 19:17 .
drwxr-xr-x 20 root root   4580 Jun 19 19:33 ..
crw-rw----  1 root root 10, 58 Jun 19 19:17 evtchn
crw-rw----  1 root root 10, 57 Jun 19 19:17 gntdev

Check the proper major/minor values from "/proc/misc" and then use mknod to create the device nodes manually using the proper values. After that xenstored and xend should start normally. For more information see this email: http://lists.xensource.com/archives/html/xen-devel/2010-06/msg01129.html and XenParavirtOps wiki page.

Arp change notification problems after live migration when using pvops 2.6.32.x dom0 kernel

You need to have new enough version of pvops dom0 kernel (xen/stable-2.6.32.x branch, Aug 2010 or newer) and then you can enable "net.ipv4.conf.<dev>.arp_notify" sysctl setting for the devices where you want to send the arp notifications (gratuitous ARP). See this email for more information: http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01595.html .

Starting xend fails?

"ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)".

or "xm" commands give the following error message: "Error: Unable to connect to xend: No such file or directory. Is xend running?".

Please check these things:

First of all - are you actually running Xen? Did you modify grub.conf to boot Xen hypervisor (xen.gz)? You can verify with "dmesg | grep -i xen" or "grep -i xen /var/log/dmesg" - you should see at least a line like "Booting paravirtualized kernel on Xen" when you've actually booted into Xen dom0.
Do you have xenfs mounted? Check "/proc/xen" directory, you absolutely NEED to have files there.

Value of "control_d" in file "/proc/xen/capabilities" means you're in Xen dom0:

# ls /proc/xen/
capabilities  privcmd  xenbus  xsd_kva  xsd_port

# cat /proc/xen/capabilities
control_d

Maybe you need a line like this in "/etc/fstab":

none /proc/xen xenfs defaults 0 0

After adding that line to "/etc/fstab" reboot or run "mount /proc/xen".

Do you have "xen-evtchn" driver loaded? Check with "lsmod | grep -i xen" and load with "modprobe xen-evtchn".
Do you have "xen-gntdev" driver loaded? Check with "lsmod | grep -i xen" and load with "modprobe xen-gntdev"
Do you have the required device nodes under "/dev/xen/" directory? An example:

# ls -la /dev/xen
total 0
drwxr-xr-x  2 root root     80 Jun 19 19:17 .
drwxr-xr-x 20 root root   4580 Jun 19 19:33 ..
crw-rw----  1 root root 10, 58 Jun 19 19:17 evtchn
crw-rw----  1 root root 10, 57 Jun 19 19:17 gntdev

Correct major/minor numbers for those device nodes can be checked from file "/proc/misc", the values are dynamic and might change from system to system:

# cat /proc/misc | grep xen
57 xen/gntdev
58 xen/evtchn

Do you have SElinux running? Did you try turning it off? Run "setenforce 0" or disable it in "/etc/selinux/config" and reboot dom0.
Do you have xenstored or oxenstored running? xend required xen store daemon to be running before starting xend.
If you're using certain Debian or Ubuntu (10.04+) versions, and you're compiling Xen from source (.tar.gz), you might need to run "make install-tools PYTHON_PREFIX_ARG=" to install Xen.. notice the empty argument to PYTHON_PREFIX_ARG to get the Xen python modules installed to correct directory location.
If you upgraded from older Xen version, did you first make sure all the files from earlier Xen versions were completely removed before installing the new version? Xend might fail to start when there's a bad installation mixing files from multiple Xen versions.

For other troubleshooting tips related to pvops dom0 kernels try reading XenParavirtOps wiki page.

How to specify custom (non-default) vif-script for some domU network interface?

Here's an example how to have 2 nics (vifs) for the domU, each using different vif-script:

vif = [ 'mac=00:16:5e:72:04:01,bridge=xenbr0,script=your_custom_script1', 'mac=00:16:5e:72:04:02,bridge=xenbr1,script=your_custom_script2' ]

Problems with Xen on Redhat Enterprise Linux 5 (RHEL5) or CentOS5

Try these links:

I'd like to run Xen hypervisor/dom0 on Redhat Enterprise Linux 6 (RHEL6)

See this wiki page for a Xen 4.0 with RHEL6 tutorial and more info: RHEL6Xen4Tutorial .

Starting Xen HVM guests fails when using Xen 4.x and gcc 4.6

When Xen 4.x is compiled with gcc 4.6 compiler it seems Xen hvmloader binary gets miscompiled and makes it impossible to run HVM VMs. This issue does not happen when using gcc 4.5 or older gcc versions. The error message might be something like this:

(XEN) io.c:194:d18358 MMIO emulation failed @ 0018:9ffff: 00 e0 de be 00 83
(XEN) hvm.c:1099:d18358 Triple fault on VCPU0 - invoking HVM system reset.

or then just a line with words "HVM Loader" and nothing more, followed by a VM crash a couple of seconds after it started. This bug is fixed in xen.org repositories, but the fix is not yet in Xen 4.0.2 or 4.1.1, because the bug was found and fixed after the release of Xen 4.0.2 and Xen 4.1.1. See this email for info about the bugfix changesets: http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00922.html . Especially users of Debian Wheezy, Ubuntu 11.10+ and Fedora 15/16 have seen this issue.

xm: ImportError: No module named xen.xm

After installing Xen from source (either mercurial/hg repository or from .tar.gz) you get an error when trying to start xend or run "xm" commands. The error looks like:

ImportError: No module named xen.xm

This means Xen python modules are not installed to correct directory. Debian/Ubuntu changed the location of python modules include directory, so you need to use the following when installing Xen:

make install-tools PYTHON_PREFIX_ARG=

Notice the empty parameter to PYTHON_PREFIX_ARG. Re-install the modules/tools and the problem should get fixed.

With Linux 3.0 as a dom0 kernel using dom0_mem=<xyz> option for Xen in grub.conf doesn't work

This is a known bug in Linux 3.0 kernel. It is fixed in Linux 3.1 and Linux 3.0.5 stable update. If using 3.0 older than 3.0.5 the workaround is to use, in addition to "dom0_mem=2048M" for Xen hypervisor (xen.gz), also "mem=2048M" option for dom0 Linux kernel (vmlinuz) in grub.conf.

Also see Linux_30_bugs: wiki page for other known issues in upstream Linux 3.0 kernel.

How do I change the resolution of Xen PV domU vfb graphical VNC console?

You can specify the pvfb (paravirtual framebuffer) resolution and bpp (amount of colors) for the VM while loading the xen-fbfront driver in the domU kernel.

If xen-fbfront is built as a module, use the following options:

modprobe xen-fbfront video="32,1024,768"

Or add the options to "/etc/modprobe.conf", or whatever file your domU distro uses for driver module options. Note that you might need to regenerate the initrd/initramfs image for the domU kernel if the xen-fbfront driver is auto-loaded at boot time.

Or if xen-fbfront driver is built-in to the domU kernel, use the following cmdline options for the domU kernel:

xen-fbfront.video=32,1024,768

If you're using Xen pygrub you can place that option to grub configfile inside the domU "/boot/grub/" directory, or if you're using kernel/ramdisk from dom0, then add those options to "extra" line on /etc/xen/<domU> on dom0.

How can I get resolutions larger than 800x600 for Xen HVM guest graphical VNC console?

Edit "/etc/xen/<hvmvm>" cfgfile and add the following options:

stdvga=1
videoram=16

This will increase the amount of virtual video memory in the HVM VM, allowing it to use bigger resolutions, up to 2048x1536 @ 32bpp. If you don't specify "stdvga=1", ie. you keep using the Cirrus emulated graphics adapter, resolutions up to 1280x1024 are possible with "videoram=16".

After increasing the size of vram configure the resolution from inside the HVM guest in the usual way, just like you'd do on baremetal/native, using the resolution selector or configfile provided by the operating system in the VM.

I'm trying to create a new Xen VM but I get error there's not enough free memory

Tools like "free" or "top" in Xen dom0 show that you have a lot of free memory, but Xen still complains that there's not enough free memory to create/start a new VM, giving errors such as "Out of memory". This usually happens because dom0 is using all the memory, so there's no free memory in the Xen hypervisor (xen.gz) for other VMs. Note that dom0 is actually a Xen virtual machine!

Xen hypervisor dedicates physical memory for each VM, so you need to have actual free unallocated memory in the hypervisor to start a VM. You can check the Xen hypervisor memory usage with the following commands:

Check the amount of total, used and free memory in Xen hypervisor:

xm info

and check how much memory each VM is using:

xm list

It's possible that dom0 is using most of your memory and Xen hypervisor doesn't have any free memory left. You should use "dom0_mem=" setting for Xen in grub to configure dom0 a fixed/dedicated amount of memory, so the rest of the memory is free for other VMs to use. See XenBestPractices: wiki page for more info about configuring "dom0_mem=" option for Xen.

Error: (4, 'Out of memory', 'panic: xc_dom_core.c:442: xc_dom_alloc_segment: segment ramdisk too large (0xee93 > 0x8000 - 0x1755 pages)')

Getting this error when trying to start a new VM (running "xm create") means you're trying to use too big ramdisk image (initrd/initramfs file) for the domU. The ramdisk image configured is bigger than the amount of available memory for the VM. Remember the ramdisk images are compressed, and they need to fit as uncompressed to the domU memory. The solution is to edit "/etc/xen/<domU>" configuration file and increase the amount of memory for the VM, or switch to using a smaller ramdisk image for the domU kernel.

What's the difference between vifX.U and tapX.Y network interfaces in dom0?

vifX.Y network interfaces are used with domUs (VMs) using paravirtualized network drivers, which means for pure PV domUs or for Xen HVM guests with paravirtualized PVHVM drivers in use. tapX.Y interfaces are used for HVM guests which are using emulated virtual NICs (Intel E1000, Realtek RTL8139, NE2k).

There's exactly one vif/tap interface per virtual NIC in the VM. "X" means the domain ID, and "Y" is the number of the virtual NIC. So if you have a domU with ID 5 with 3 virtual NICs (eth0, eth1, eth2), the corresponding VIF interfaces in dom0 would be vif5.0, vif5.1, vif5.2.