Xen via libvirt for OpenStack juno

This document describes steps I take to setup a compute node based on Ubuntu14.04 for OpenStack juno, using the Xen via libvirt approach. Openstack does not support this approach well as it is in the group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!

Prerequisite

Follow "OpenStack Installation Guide for Ubuntu 14.04" to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.

Steps

1. Add OpenStack juno to the repository.

apt-get update
apt-get install software-properties-common
add-apt-repository cloud-archive:juno
apt-get update

2. Install nova-compute-xen, sysfsutils and python nova client

apt-get install nova-compute-xen sysfsutils python-novaclient

3. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in the installation guide to configure nova in a compute node.

• In the [default] section, add the following.
  

[default]
...
rpc_backend = rabbit
rabbit_host = controller
rabbit_password = RABBIT_PASS
auth_strategy = keystone
my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
vnc_enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
novncproxy_base_url = http://controller:6080/vnc_auto.html
verbose = True

MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface for this compute node, typically 10.0.0.31 for the first compute node.

• Add keystone_authtoken section
  

[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = NOVA_PASS

• Add glance section

[glance]
host=controller

4. Verify the conent of /etc/nova/nova-compute.conf is as following.

[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=xen

5. Installing nova-compute-xen doesn't install xencommons script for us. However, we need xencommons to start the qemu process to provide block device backend service for qemu disks. I compile and install xen-4.4 from source, to get the xencommons script. Note this is likely not the preferred approach, and please update it if you know a better approach.

• Download xen-4.4 source code, compile and install.

apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 \
   module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended \
   texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial make gcc libc6-dev \
   zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev \
   iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils \
   libyajl-dev gettext libpixman-1-dev
apt-get build-dep xen
cd xen
./configure
make dist -j16
make install 

• Enable startup services

update-rc.d xencommons defaults 
update-rc.d xen-watchdog defaults

• Install virt-manager

apt-get install virt-manager

If we do not install virt-manager, when you run "xl", it may say "xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory"

6. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in dom0 kernel crashes when creating a Xen domU from OpenStack GUI (horizon). I have applied this patch and make the modified qemu available in github.

wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
unzip master.zip
cd qemu-2.0.2-master/
./configure
make -j16
make install

• Update /etc/init.c/xencommons to use the new version of qemu, rather than the default one.

--- /etc/init.d/xencommons.bk	2014-11-14 10:57:32.183529318 -0700
+++ /etc/init.d/xencommons	2014-11-14 10:56:42.474535348 -0700
@@ -118,6 +118,7 @@ do_start () {
     ${SBINDIR}/xenconsoled --pid-file=$XENCONSOLED_PIDFILE $XENCONSOLED_ARGS
     echo Starting QEMU as disk backend for dom0
     test -z "$QEMU_XEN" && QEMU_XEN="${LIBEXEC}/qemu-system-i386"
+    QEMU_XEN="/usr/local/bin/qemu-system-i386"
     $QEMU_XEN -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \

-monitor /dev/null -serial /dev/null -parallel /dev/null \ -pidfile $QEMU_PIDFILE

7. Create a link at /usr/bin for pygrub

ln -s /usr/local/bin/pygrub /usr/bin/pygrub

8. Install and configure network component in compute node. Follow steps in this guide. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if Neutron and the nova security groups are used together.

9. Reboot the machine.

Reference

Compile and Install QEMU
Compile and Install Xen 4.4 in Ubuntu14.04