XenFAQ2

Needs Refactor This document would benefit from being broken up into smaller documents alongside topics and merged with split up documents from XenFAQ

Needs Review Not quite clear how up-to-date these questions and answers are.

This document has been created from http://www.xen.org/files/Support/XenUsersCommonlyAskedQuestions.pdf as a life version. Also see XenFAQ: this page needs correcting and merging into XenFAQ

Guest Related Questions

Guest Conversion

How do I convert a Centos HVM Guest to a PV Guest?

Creating a Centos HVM domU with working PV drivers : http://pastebin.com/fb6fe631 Converting HVM guest to PV guest : http://pastebin.com/f6a5022bf

If you follow both parts correctly you should have a working PV domU. If anything goes wrong during conversion process, you should still be able to boot the previous HVM domU config if you select the non-xen kernel (second entry) from grub menu.list.

I have an Xen image that was built for a graphical console (VNC). Is there any way to change it to the non-graphical console (xen console)?

For HVM guest, you need to enable serial port on domU config file (example here: http://pastebin.com/fb6fe631), and setup domU to use serial port (ttyS0 on Linux) by modifying (for Linux domU) /boot/grub/menu.lst, /etc/inittab, and /etc/securetty.

If it's PV guest, you need to set up domU to use xen console (which is xvc0 on current xen version, hvc0 on pv_ops kernel). It's similar to setting up domU for serial console, you just need to change ttyS0 to hvc0. An example of domU setup that can use both xvc0 and vnc console is here : http://pastebin.com/f6a5022bf

How do I remove an active virtual machine?

xm shutdown or xm delete

How do I run xm console to a WindowsXP DomU?

You can't xm console to that (I'm not sure you can xm console to any hvm, but I know you can't to one that doesn't have a console).

I start a new DomainU (Guest) and some text scrolls by for launching the guest but then it just sits there with Continue and no actions takes place?

The console for this new DomainU is not properly available for you; fix this by adding xtra="xencons=tty" in the configuration file. This will bring up a login screen directly for your new DomainU.

One of our CentOS 5.3 randomly reboots, at different times of the day, and I can't see why it's doing it. I have looked through the logs, but don't see any thing in there that shows me why it has rebooted. How can I debug this?

The problem is that when the box panics, it stops syslogd, so you don't get the panic output in /var/log. The best way to fix this is to setup a logging serial console.

Drivers

What are the GPLPV Drivers and where can I get them?

A collection of open source Window PV drivers that allow Windows to be para-virtualized. They are currently being implemented under the leadership of James Harper. More information on these drivers at:

• http://wiki.xensource.com/xenwiki/XenWindowsGplPv/Installing
• http://lists.xensource.com/archives/html/xen-users/2009-04/msg00058.html
• http://meadowcourt.org/downloads/

How can I tell if the GPLPV Drivers are loaded correctly?

If the drivers are installed correctly there should be a Xen device under 'System Devices' in device manager.

Domain0

Why cannot I see all my RAM on my Dom0?

Domain 0 is a paravirt VM in reality, so the amount of ram you allocate to it is what you will see when using local tools like free, /proc/meminfo, top, etc.

To see the full system ram, you need to use the xm tools... and in this case, 'xm info' which will show you all the system resources, as opposed to the resources available to dom0.

Also, you have 16GB ram on the system... you probably already know this, but be aware that without a PAE enabled kernel (if you're using 32bit Xen) you'll only see 4GB of this. PAE will allow you to use up to 16, or maybe 32 (I don't remember what the upper limit for PAE enabled Xen is off the top of my head).

Is there any way of checking DomU´s I/O from Dom0?

iostat (Debian: sysstat-package)

Can I allocate one CPU to the Dom0 exclusively?

Add this to the kernel boot line - dom0_max_vcpus=1 & dom0_vcpus_pin

Edit /etc/xen/xend-config.sxp - set “(dom0-cpus 1)” Reboot Dom0 To try on an active system without a Reboot -

xm vcpu-set 0 1 xm vcpu-pin 0 0 0

Running xm info I see the following memory available; what does the free memory mean?

total_memory : 2046 free_memory : 5

Free_memory from "xm info" shows memory not allocated to any domain (inlcuding dom0). "free", "top" (or whatever) shows free memory on that particular domain (in your case, dom0). You can adjust memory allocation per domain using "xm mem-set".

DomainU

My DomU does not fully start; it shows the following output stopping at Continue...

$ sudo xm console test
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize

Xen virtual console successfully installed as xvc0 Event-channel device installed. netfront: Initialising virtual ethernet driver. i8042.c: No controller found. mice: PS/2 mouse device common for all mice TCP bic registered NET: Registered protocol family 1

NET: Registered protocol family 17

Using IPI No-Shortcut mode xen-vbd: registered block device major 8 blkfront: sda2: barriers enabled XENBUS: Device with no driver: device/console/0 Freeing unused kernel memory: 140k freed kjournald starting. Commit interval 5 seconds

EXT3-fs: mounted filesystem with ordered data mode.

***************************************************************

***************************************************************
** WARNING: Currently emulating unsupported memory accesses **
**	in /lib/tls glibc libraries. The emulation is	**
**	slow. To ensure full performance you should	**
**	install a 'xen-friendly' (nosegneg) version of   **
**	the library, or disable tls support by executing **
**	the following as root:	**
**	mv /lib/tls /lib/tls.disabled	**
** Offending process: modprobe (pid=663)	**
***************************************************************
*************************************************************** 
Continuing...

It might only be that you don't have a VPS physical console, and that your VPS is fully booted, but you can't see it. There are few things to check.

First, check that your VPS has a "console" device in /dev. Mount your domU filesystem in the dom0, go in /dev and do:/dev/MAKEDEV console

If you are using a modern Xen kernel and hypervisor, you should check the parameters of the startup file. Check that it has the following option:

extra = "4 TERM=xterm xencons=tty console=tty1"

Then start your VPS and watch it booting. Note that once it's booted up, you should check that it has a xen friendly libc6 installed (in Debian, you would do "apt-get install libc6-xen").

Is there a way to set the credit-scheduler's limits and weights per domU in the domU configuration file?

weight= in the xm config file works, unless you are using RHEL or CentOS.

How do I start an application from within a DomU?

Well, you could always just log in to that VM, open a terminal and run the program. Or you could SSH or telnet in to the VM, start a screen session and run the program.

A VM acts just like any other server, so the proceedure for starting programs and executing commands locally and remotely are exactly the same as doing so on any computer.

My domUs are in a permanent 'b' (blocked) status as shown by 'xm list', even though they are functioning just fine. That's not normal, is it?

It's normal for them to show as blocked when they aren't actively running something - in the same way that any process on a 'normal' machine will show as blocked when it's waiting for input, each guest will show as blocked when it's got nothing to do. Give something a processor intensive task to do and you'll find it changes state to running (at least some of the time).

Is there any way, to get the name of a domU from the network-common script?

hostname=$(xenstore_read "$XENBUS_PATH/domain" | tr -- '_.:/+' '-----')
}}

=== Is it possible to increase the screen resolution of my xen guest Windows Vista? ===

On current Xen, with stdvga=1 & videoram=16, resolutions up to 2048x1536x32 are possible.  All that said, the RDP suggestion is probably a better way to access the guest in any case.

=== How to install Solaris via HTTP as a para guest? ===

Solaris 10 can only be used as HVM guest. OpenSolaris can be used as PV guest, installed from iso. You can't install it from http. Once you have it installed, you also need zfs support for pygrub (either that, or manually copying kernel and boot archive to dom0)

Boris provides some nice examples on his site : http://bderzhavets.wordpress.com/ 

=== Is it possible to find out the specific vnc Display Number of a domU? ===

{{{
virsh vncdisplay domU_name_or_id xenstore-ls /local/domain/domU_id/console

I am trying to create a guest domain. I specified the configurations in /etc/xen-tools/xen- tools.conf and I ran $sudo xen-create-image --hostname=virtualrouter1 --role=udev the output is: sudo: xen-create-image: command not found

Make sure you installed the Xen tools, for example: apt-get install xen-tools

I'm trying to assign a dynamic hostname to a xen instance as follows: Cfg file

kernel = "/root/vmlinuz-2.6.18-128.1.14.el5xen" ramdisk = "/root/initrd-2.6.18-128.1.14.el5xen.img" memory = 512
hostname = "uniquehostname"

#hostname = " uniquehostname.xen.org" name = "my-vm-name"
. . . . .

In both the cases, the instance is unable to get the correct hostname..

From "xm create --help_config" : hostname=NAME Set the kernel IP hostname. interface=INTF Set the kernel IP interface name. dhcp=off|dhcp Set the kernel dhcp option.

On most LInux distros, kernel hostname and IP address is ignored, making it somewhat useless. You need to use your normal distro method to set hostname on domU (/etc/sysconfig/network on RHEL)

As far as I can see, there is something different between using 'xm create' and 'xm new' followed by 'xm start'. It's something to do with data being stored in XenStore. I couldn't suspend the one started with 'xm create'. Could someone please explain the effective difference between the two and when 'create' should be used instead of 'new' and vice-versa.

xm create -> domU configuration is NOT managed by xend. Usually using config files on /etc/xen. This is the easiest method to use for beginners, as you have a config file that you can edit manually. The default on RHEL5 (which uses Xen 3.1+).

"xm new" and "xm start" -> domU configuration is managed by xend. You change values using commands like "xm block-attach", which can modify settings online. No config file to edit manually. The default on current versions of Xen.

I have problem with domU clock. It lose 30 minutes each day. How can i synchronize it with dom0 clock?

Is this PV domU? If yes, setting /proc/sys/xen/independent_wallclock to 0 (the default) should make it sync with dom0. You only need ntp on dom0, and domUs will follow.

The alternative, set /proc/sys/xen/independent_wallclock to 1 and run ntp on domU. If this is a HVM dom0, running ntp on domU is your friend.

Also, check http://tuttodebian.blogspot.com/2008/05/xen-clocksource0-time-went-backwards.html to see if your system experience similar symptoms.

I would like to set sched-cred parameters on my domU configuration file. How can i do that?

cpu_cap & cpu_weight

Run "xm create --help_config" for details, and read http://wiki.xensource.com/xenwiki/CreditScheduler

Is it possible to increase guest memory without reboot?

You can do a "xm mem-set <Domain> <memory>" for a PV domU, but you had to set maxmem higher than current assignment beforehand.

Is it possible to take an already created domU sparse file and make it a non sparse file?

cp --sparse=never orig.img new.img

I have tried to change CD ISO images during a HVM install using the following commands but it doesn't work. After changing the CD ISO image, it doesn't detect the new ISO image.

(qemu) eject -f hdc (qemu) change hdc /media/hitachi/cd-rom-image.iso

se xm block-list <domid> to find the cdrom be-path for the domain, for example:

xm block-list 5 Vdev BE handle state evt-ch ring-ref BE-path 768 0 0 4 9 16383
/local/domain/0/backend/vbd/5/768 5632 0 0 1 -1 -1 /local/domain/0/backend/vbd/5/5632

Having identified the cdrom device (5632) you can check what iso image it is connected to:

xenstore-read /local/domain/0/backend/vbd/5/5632/params

(nothing returned)

To connect a new iso image:

xenstore-write /local/domain/0/backend/vbd/5/5632/params /mnt/gl3- tb1_store/MWWin2003R2SvrStdx86_BX2SVOL_EN.iso

And you can now see that it is connected:

xenstore-read /local/domain/0/backend/vbd/5/5632/params /mnt/gl3- tb1_store/MWWin2003R2SvrStdx86_BX2SVOL_EN.iso

This method works with both emulated devices and with gplpv drivers.

Is it possible to set the xen to boot the domU one by one when server starts, as currently we have 20 domU, and if boot them together, the the hard disk will be very very slow.

cd /etc/xen/config/........ && for i in * do ...... (start VM, .....)...... sleep 60 (or whatever time you think is right to start a VM) done

I use FluidVM on some of our VPS host nodes, and the management server has crashed, so now I need to recover the running VM's, somehow. FluidVM deploys the domU's on the hostnode dynamically from a database, i.e. there's no /etc/xen/vps1 (for example) config files. The domU's are still running on the servers, and I now want to create config files for them, while they're running. How would I be able todo this?

For example, here's a list of running VM's from one of the servers:

root@usaxen02:[~]$ xm list
Name	ID Mem(MiB) VCPUs State   Time(s) AndriesBurger_39_cronos		90	255	1 -b----	42.4
Bruce_18_carmen	60	255	1 r----- 3528327.5
Domain-0	0	3433	4 r----- 1116681.7
Rudi_14_mars	40	3007	2 -b---- 953036.3
Rudi_44_vps2	93	255	1 -b----	22.9

Is there any way to create a config file, /etc/xen/AndriesBurger_39_cronos, from the running domU AndriesBurger_39_cronos ?

You can use "xm list -l" to dump the configuration in SXP format; then you should be able to use "xm new" or "xm create" with the "-F" option to load an SXP-based config file. See the "xm" man page for more info - that's where I dug up this.

How to set up Xen DomU as Windows 2008 Server on a CentOS Dom0 machine?

Start using the normal way that you usually do when you install a HVM domU, whether it's virt-manager/virt-install or using manually-created config file. One additional thing to note is that for 64bit HVM domUs you need to make sure that acpi, apic, and pae is set to 1 on domU config file.

Once you get that Win2008 fully installed, you can install GPLPV driver later to improve performance.

Devices

For my xen domUs I'm using a mixture of locah physical partitions (with LVM) and iSCSI disks. For local partitions, I don't have any problem, because LVM volumes are always the same. But for iSCSI disks, devices are assigned in the order they are connected, so I can't be sured that device that now is /dev/sdb (for example) will always be /dev/sdb.

So, is there any way to identify the physical device in the domU configuration not as phy:/dev/sdb, but something like phy:label=fslabel? Or is there any other solution to this problem?

I go with phy:/dev/disk/by-path/ip-*-iscsi-iqn.* If you assign iscsi luns directly as domU's fs without additional partitioning, you could probably also use /dev/disk/by-label/* or /dev/disk/by-uuid/*

Installation Questions

File Systems

Is there a way to have a shared root file system amongst a set of Xen servers?

You can install the OS in an LVM partition and use it shared across all the xen domUs when you use the parition as 'r' instead of 'w' when defining the disk.

But you have to do all tasks needed for read-only root filesystem.

Like 1. mount ramfs in /tmp and in /var ... http://en.opensuse.org/How-To_Make_the_root_filesystem_read-only I use 2 disks in Xen with one as read-only mounted as / and the other is the data partition. I have a need to have scratch partition with pre-populated data and for this I create a LV and put data into it (eg:- software etc.,) and then create a snapshot of this volume and send it as rw to the xen machine. This way my original software partitions are intact and also the changes (may be damaging) done in the xen volumes are lost once the snapshot grows to 100%.

=== I installed two Debian web server which run a phpbb3 forum. One stays on a Xen paravirtualized domU (512 MB of ram, 1 vcpu, disc on a raw file file:/home/vale/debian.img,hda,w) on OpenSuse 11.0 and one run on a hyper-v virtual machine (512 MB 1 cpu) build on Windows Server 2008 R2. The performances on PV are very poor than hyper-v. ab -n 3000 -k -c50 http://site.lan/phpBB3/ returns 13,22 req/sec on PV domU and 38,37 req/sec on hyper-v. Why? ==

I installed O.S. guest as a HVM domain, then I installed linux-xen-image files and I use them for vmlinuz and initrd. I also installed libc6-xen.

Xen PV config file:

name='pv'
ramdisk='/home/vale/initrd.img-2.6.18-6-xen-686' kernel='/home/vale/vmlinuz-2.6.18-6-xen-686' bootloader=''
vif=['mac=00:16:3e:33:37:4f, bridge=xenbr0']
vcpus=1 memory=512 disk=['file:/home/vale/pv.img,hda,w'] on_reboot='restart'
on_crash='restart' extra='' root='/dev/hda1' platform='xen'

I'm assuming that phpBB3 is relatively I/O intensive (since it uses db, which I assume you also installed on the same host). In that case, your bad numbers are probably because of this

disk=['file:/home/vale/pv.img,hda,w']

On Xen, file:/ is not recommended, and you should use tap:aio:/ instead for file-backed storage. Then again, another user reported that even tap:aio isn't good enough

http://lists.xensource.com/archives/html/xen-users/2009-01/msg00820.html

So in short, if you use Xen PV, you might want to consider using LVM/partition-backed storage.

Is it possible to start a VM that contains just gpxe (which when started, will get an image from a provisioning server and will load that image)

In this article, we'll show you the prcesses to setup PXE boot environment for Xen host (hypervisor + dom0) and Xen guest, both PV (Para-Virtualized) guest and HVM (Hardware-assisted Virtual Machine). Details at http://os-drive.com/files/docbook/xen-pxeboot.html.

=== I tried to resize a disk of my data guest from 100 to 400 GB. I did an lvresize /dev/xendata/data-disk -L 400G an it works. I started the Guest and did an df -h to check the size but there are still 100 G ===

The container is bigger but the filesystem isn't. Resizing an LV doesn't make the FS any bigger.

Log into the DomU and do a resize2fs <device>. You can do this while it's mounted as long as the filesystem is getting bigger.

Oh, and if you've partitioned the LV inside the guest, you'll also need to resize the partition (BEFORE you do a resize2fs, etc.). There are two ways to do this - the safest is to use parted, which works if you're using ext2/ext3 (and a couple other of the most popular filesystems - reiser, I think). The other method is to delete the partition and recreate it with the extended end points. This isn't quite as safe and requires that 1) you're start point for the partition is exactly the same as it was before, and 2) the partition is the last (or only) one on the LV.

I want use xen with dynamic slices. For example, I have 20 domU based on FreeBSD, xen hypervisor 3.3.1, Debian Lenny dom0 system. All domUs have 80Gb LVM partitions, but realy they use 20 of this 80Gb and I want to create more domU's. How can I do it? I know that some virtualisation have possibility to do dynamic slices(4 example Virtul box)

Do you mean storage overcommit? That is, assign more storage to domU than what you actually have?

If yes, it's not a matter of Xen vs VirtualBox. It's a matter of what storage backend you use. If you use one of these: - sparse raw file (with file: or tap:aio:) - qcow - vmdk/vdisk (I think full support is only in newer Xen or Opensolaris) - zvol (on Opensolaris) then you can overcommit storage. But if you use disk/partition/LVM for domU storage, you won't be able to.

32bit vs 64 bit

Is there anyway to install 64Bit Linux DomU on 32Bit Linux Dom0?

Types of domU that can be run depends mostly on hypervisor, and not dom0. So if you have 64bit hypervisor, you should be able to run 32 and 64bit PV and HVM domUs, regardless whether dom0 is 32 or 64bit.

If you have 32bit dom0 and 32bit hypervisor, you should be able to run 64bit HVM domU, but not 64bit PV domU.

High Availability Questions

Services

What software exists for Xen to handle high availability? A (HA1.0): Here are several tools that currently exist:

• CentOS Cluster
• Project Kemari - http://www.osrg.net/kemari/
• Project Remus - http://blog.xen.org/index.php/2009/04/02/project-remus-released/
• Linux Cluster Resource Manager - http://clusterlabs.org/mediawiki/images/f/fb/Configuration_Explained.pdf

=== I'm not sure about how snapshots works on Xen. For example, if I snapshot a DomU with 10GB HD will take, for example, 30 seconds. But, if I snapshot a DomU with a 100GB HD will take longer (I guess). ===

So, I wanna know how the snapshot works on xen. What if I want to move a snapshotted with a 100GB HD from a Dom0 to another Dom0? I've to move a 100GB file?

You could try this approach:

http://www.infohit.net/blog/post/using-lvm2-snapshots-to-provide-rollback-functionality-for-xen.html

Makes snapshots quite quick.

Security Questions

If I install minimal linux for XEN in dom0 and a periphery firewall in domU and other applications in other instances of domU, is it possible to restrict/bind the network card to domU having periphery firewall and from there forward packets for dom0 or for other domUs?

Is this possible? If so, is it secure? Or does dom0 always have direct access to Network Card and needs a separate firewall? And packets will always route from dom0 to all domUs ?

What are the issues involved?

The approach I've used at home is to hide a network card from Dom0 (see pic-back.hide) and pass it through to a DomU which then sees it as a native interface. I then run a firewall in the DomU and the outside traffic does NOT go through Dom0. The route for packets is then :

real i/f -> DomU (firewall) -> VIF -> int bridge [ Dom0 | VIF -> DomU ]

From security perspective, this is the same as having an L2 switch (when dom0's bridges have no IP address) or L3 switch (when dom0's bridges have an IP address)

=== I want to use a Disk Encryption and the conplete physikal Disk in a DomU. I prefer Truecrypt or Loop-aes. i will going to test loop-aes cause it should have the better performance. But, did anybody here using truecrypt or loop-aes ? What is the better one, in the fact of speed ?===

dm-crypt/luks is one option, and performs about the same or better than loop-aes. Also it's less problematic because it doesn't use loop devic