Xen via libvirt for OpenStack juno

This document describes steps I take to setup a compute node based on Ubuntu14.04 for OpenStack juno, using the Xen via libvirt approach. Openstack does not support this approach well as it is in the group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!

Prerequisite

Follow "OpenStack Installation Guide for Ubuntu 14.04" to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.

Steps

1. Add OpenStack juno to the repository.

apt-get update
apt-get install software-properties-common
add-apt-repository cloud-archive:juno
apt-get update

2. Install nova-compute-xen, sysfsutils and python nova client

 apt-get install nova-compute-xen sysfsutils python-novaclient

3. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in the installation guide to configure nova in a compute node.

• In the [default] section, add the following.
  

 [default]
 ...
 rpc_backend = rabbit
 rabbit_host = controller
 rabbit_password = RABBIT_PASS
 auth_strategy = keystone
 my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
 vnc_enabled = True
 vncserver_listen = 0.0.0.0
 vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
 novncproxy_base_url = http://controller:6080/vnc_auto.html
 verbose = True

MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface for this compute node, typically 10.0.0.31 for the first compute node.

• Add keystone_authtoken section
  

 [keystone_authtoken]
 auth_uri = http://controller:5000/v2.0
 identity_uri = http://controller:35357
 admin_tenant_name = service
 admin_user = nova
 admin_password = NOVA_PASS

• Add glance section

 [glance]
 host=controller

4. Verify the conent of /etc/nova/nova-compute.conf is as following.

 [DEFAULT]
 compute_driver=libvirt.LibvirtDriver
 [libvirt]
 virt_type=xen

5. Installing nova-compute-xen doesn't install xencommons script for us. However, we need xencommons to start the qemu process to provide block device backend service for qemu disks. I compile and install xen-4.4 from source, to get the xencommons script. Note this is likely not the preferred approach, and please update it if you know a better approach.

• Download xen-4.4 source code, compile and install.

 apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 \
   module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended \
   texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial make gcc libc6-dev \
   zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev \
   iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils \
   libyajl-dev gettext libpixman-1-dev
 apt-get build-dep xen
 cd xen
 ./configure
 make dist -j16
 make install 

• Enable startup services

 update-rc.d xencommons defaults 
 update-rc.d xen-watchdog defaults

• Install virt-manager

 apt-get install virt-manager

If we do not install virt-manager, when you run "xl", it may say "xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory"

6. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in dom0 kernel crashes when creating a Xen domU from OpenStack GUI (horizon). I have applied this patch and make the modified qemu available in github.

 wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
 unzip master.zip
 cd qemu-2.0.2-master/
 ./configure
 make -j16
 make install

• Update /etc/init.c/xencommons to use the new version of qemu, rather than the default one.

 --- /etc/init.d/xencommons.bk	2014-11-14 10:57:32.183529318 -0700
 +++ /etc/init.d/xencommons	2014-11-14 10:56:42.474535348 -0700
 @@ -118,6 +118,7 @@ do_start () {
 ${SBINDIR}/xenconsoled --pid-file=$XENCONSOLED_PIDFILE $XENCONSOLED_ARGS
 echo Starting QEMU as disk backend for dom0
 test -z "$QEMU_XEN" && QEMU_XEN="${LIBEXEC}/qemu-system-i386"
+QEMU_XEN="/usr/local/bin/qemu-system-i386"
 $QEMU_XEN -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
	-monitor /dev/null -serial /dev/null -parallel /dev/null \
	-pidfile $QEMU_PIDFILE

7. Create a link at /usr/bin for pygrub

 ln -s /usr/local/bin/pygrub /usr/bin/pygrub

8. Install and configure network component in compute node. Follow steps in this guide. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if Neutron and the nova security groups are used together.

9. Reboot the machine.

Reference

Compile and Install QEMU
Compile and Install Xen 4.4 in Ubuntu14.04