XCP Building Instructions

Building xapi on CentOS 6.3

In this guide we look at an experimental way to get xapi running on CentOS 6.3

This is not Zeus, there are no packages yet. This is a developer install using: http://opam.ocamlpro.com/

Setting up Xen

While not strictly required for the build, to help us test it, lets install Xen.

Minimal install of CentOS 6.3 x86_64. Use network install iso on VMware fusion, or full iso, minimal confuses things.

This is based on http://wiki.centos.org/QaWiki/Xen4

yum install wget

cd /etc/yum.repos.d/
wget http://dev.centos.org/centos/6/xen-c6/xen-c6.repo
yum repolist

yum install kernel kernel-firmware
yum install xen

Now need to edit /etc/grub.conf Note: check the paths and the kernel version, also check dom0 memory. Also: The above 3.4.x kernel should the second in the list, xen should be first.

title xen
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all
        module /vmlinuz-3.4.32-6.el6.centos.alt.x86_64 ro root=/dev/mapper/vg_jgarbuttcentos-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_LVM_LV=vg_jgarbuttcentos/lv_root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_DM rd_LVM_LV=vg_jgarbuttcentos/lv_swap
        module /initramfs-3.4.32-6.el6.centos.alt.x86_64.img

Reboot, then check all is well:

uname -a
xl info

Install OCAML 3.12.1

Currently this is not part of CentOS 6.3

One way is to build the following Fedora 17 srpm: http://dl.fedoraproject.org/pub/fedora/linux/releases/17/Everything/source/SRPMS/o/ocaml-3.12.1-1.fc17.src.rpm

Instructions can be found here, including the required setup: http://wiki.centos.org/HowTos/RebuildSRPM

You can download some very rough packages, and install them like this:

mkdir ocaml
cd ocaml

wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-3.12.1-12.el6.x86_64.rpm
wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-camlp4-3.12.1-12.el6.x86_64.rpm
wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-camlp4-devel-3.12.1-12.el6.x86_64.rpm
wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-runtime-3.12.1-12.el6.x86_64.rpm
wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-docs-3.12.1-12.el6.x86_64.rpm
wget http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-ocamldoc-3.12.1-12.el6.x86_64.rpm

yum install gdbm-devel ncurses-devel rpm-build
rpm -i ocaml-*

The were created using this source rpm:

• http://8ac390279a4a48a6f8b1-aaca1c7c7029a65ba2e38949339e1d8b.r35.cf3.rackcdn.com/ocaml-3.12.1.ocamlspotter-unknown.src.rpm

Install OPAM (from source)

Before you build, you will need:

• ocaml-3.12.1 ocaml-runtime ocaml-camlp4 ocaml-camlp4-dev (see above)

And a few other tools:

yum install make gcc

Build instructions are here: http://opam.ocamlpro.com/doc/Advanced_Install.html

Download the latest stable, and build it:

wget https://github.com/OCamlPro/opam/archive/latest.tar.gz -O opam-latest.tgz
tar -xf opam-latest.tgz

cd opam-latest
./configure
make
make install

Now you have OPAM you can initialise it:

opam init
eval `opam config env`

To make sure you don't need the above line all the time, add the following into .bash_profile before the PATH is set:

eval `opam config env`

Install OCAML 4.00.1

It is now possible to install the latest OCAML

opam switch 4.00.1

Now setup the environment again:

eval `opam config env`

Install External Dependencies

Yum

There are a few dependencies we need:

yum install git xen-devel 
yum install libuuid-devel time pam-devel tk-devel

Unpackaged

XCP has a custom version on vncterm:

wget http://a94cd2de16980073c274-9e5915cce229bfd373f03bf01a9a7c85.r57.cf3.rackcdn.com/vncterm-1.6.10-251.x86_64.rpm
rpm -i vncterm-1.6.10-251.x86_64.rpm

This was built for CentOS 6.3 from the XCP srpm:

• http://a94cd2de16980073c274-9e5915cce229bfd373f03bf01a9a7c85.r57.cf3.rackcdn.com/vncterm-1.6.10-251.src.rpm

There is also an important python script:

wget http://a94cd2de16980073c274-9e5915cce229bfd373f03bf01a9a7c85.r57.cf3.rackcdn.com/eliloader -O /usr/bin/eliloader
chmod +x /usr/bin/eliloader

This was taken from the debian version:

• https://github.com/xen-org/xcp-eliloader

OPAM

There are quite a few XCP libraries we can install using OPAM:

opam install ocamlfind omake

opam remote add xen-dev git://github.com/xen-org/opam-repo-dev
opam install xen-api-libs-transitional stdext nbd tapctl libvhd oclock cdrom netdev xenopsd

Build XAPI

There is work going on to improve the build and moving it towards OPAM. But XAPI is still not packaged using OPAM. You can find this work here: https://github.com/jonludlam/xen-api/tree/ea-1254

A tree with additional fixes for building on CentOS 6.3 can be found here: https://github.com/JohnGarbutt/xen-api/tree/centos63

Now lets try and build this:

git clone https://github.com/JohnGarbutt/xen-api.git
cd xen-api
git checkout centos63

make

set DEST_DIR=/tmp/xen-api
mkdir $DEST_DIR
make install

if you need to start again, try:

find -name "*.omc" | xargs rm -f
make clean

Configuration

Now we need to configure things correctly. Append the following settings at the end of the appropriate file.

./.opam/system/etc/xenopsd.conf

sockets-path=/var/lib/xcp # use the legacy path for old xapi
sockets-group=root
hvmloader="/usr/lib/xen/boot/hvmloader"
pygrub=/usr/lib64/xen/bin/pygrub
vncterm=/usr/lib64/xen/bin/vncterm

/etc/xensource/network.conf

openvswitch

Building and Running OVS

While not needed, this is handy, and assumed below.

The info below is a modified version of these docs:

• http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD

Some build prerequs:

yum install openssl-devel

autoconf 2.64 is required by 2.63 is packaged, so we try and get a more recent version from here:

• http://ftp.gnu.org/gnu/autoconf/autoconf-latest.tar.gz
• http://ftp.gnu.org/gnu/automake/automake-1.13.tar.xz

Due to issues with pkgconfig rpm, you then need to (see http://openvswitch.org/pipermail/discuss/2012-August/008064.html):

cp /usr/share/aclocal/pkg.m4 /usr/local/share/aclocal-1.13/

Now you can pull the latest ovs code and start to build it:

wget http://openvswitch.org/releases/openvswitch-1.4.5.tar.gz
tar -xf openvswitch-1.4.5.tar.gz
cd openvswitch-1.4.5
./boot.sh
./configure
make

Edit the Makefile so that you have the following setting, to match those expected by XenAPI:

prefix=/usr

Now we can install the binaries:

make install

Running OVS

First we need to turn off the bridge module, and check it is not loaded:

rmmod bridge
lsmod | grep bridge

We can make it stop loading in future by:

echo >> /etc/modprobe.d/blacklist.conf <<EOF
# Stop using bridge, using openvswitch instead
blacklist bridge
EOF

Now we need to load the kernel module for ovs:

insmod /lib/modules/3.4.32-6.el6.centos.alt.x86_64/kernel/net/openvswitch/openvswitch.ko

First time, we create the ovs db:

mkdir -p /usr/etc/openvswitch
ovsdb-tool create /usr/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema

Now we can start the ovs config server (at the moment, this is without ssl):

ovsdb-server --remote=punix:/usr/var/run/openvswitch/db.sock \
                     --remote=db:Open_vSwitch,manager_options \
                     --pidfile --detach

First time, you need to init the db:

ovs-vsctl --no-wait init

Now start main ovs deamon listening to same port:

ovs-vswitchd --pidfile --detach

Running xapi

There are a few more runtime dependencies:

yum install dmidecode

We can now run all the services we need:

./.opam/system/bin/xcp-fe # used to fork process service
./.opam/system/bin/xenopsd -config ./.opam/system/etc/xenopsd.conf # the component that talks to xen
/opt/xensource/libexec/v6d #the license server
/opt/xensource/libexec/xcp-networkd

Now we can run xapi:

/opt/xensource/bin/xapi # the api server, and database, and others things too

Running XAPI and OVS after restart

After you restart, by default, non of the services will restart. To help development, we can run theses services in a screen by doing:

git clone https://github.com/JohnGarbutt/xcp-devinstall.git
cd xcp-devinstall
./run.sh
screen -x xcp

Configure XenAPI Networking

No we have xapi running, we can ask it to detect all the physical interfaces:

host_uuid=`xe host-list --minimal`
xe pif-scan host-uuid=$host_uuid

# assuming a single pif
pif_uuid=`xe pif-list --minimal`
xe pif-reconfigure-ip uuid=$pif_uuid mode=dhcp
xe host-management-reconfigure pif-uuid=$pif_uuid

To update OVS config we can do the following (TODO - find plugin, configure inventory):

host_uuid=`xe host-list --minimal`
xe host-call-plugin host-uuid="$host_uuid" plugin="openvswitch-cfg-update" fn="update"

Configure XenAPI Storage

We first need to instal the storage SRs:

cd ~
git clone https://github.com/JohnGarbutt/xcp-storage-managers.git
cd xcp-storage-managers

yum install swig python-devel
export DESTDIR=/
export PYTHON_INCLUDE=/usr/include/python2.6/
make

We need an SR to store the VM disks. It is best to add a new disk or partion, then just use that:

sr_uuid=`xe sr-create type=file device-config:location=/root/vhds name-label=localstorage sm-config:type=vhd`
pool_uuid=`xe pool-list --minimal`
xe pool-param-set uuid=$pool_uuid default-SR=$sr_uuid

Xen Cloud Platform Building Instructions (CentOS 5.6)

Building the xapi toolstack

The recommended way to build the xapi toolstack is in an environment similar to that in which it will be installed - therefore you'll need a CentOS 5.5 32-bit OS installed - we tend to use VMs for this.

For convenience, there are YUM repositories hosted on downloads.xen.org that contain the build dependencies of xapi. To enable these, create a a file /etc/yum.repos.d/xcp.repo with the following contents:

[xcp]
name = xcp
baseurl = http://downloads.xen.org/XCP/xcp-1.0-repo
gpgcheck = 0

If you're building XCP unstable, replace '1.0' in the URL with 'unstable'.

Then execute the following commands:

rpm -e libaio
yum install gcc autoconf automake tetex ghostscript java-1.6.0-openjdk java-1.6.0-openjdk-devel ant pam-devel python-devel zlib-devel openssl-devel dev86 rpm-build texinfo flex bison sharutils elfutils-devel ncurses-devel e4fsprogs-devel ocaml ocaml-findlib ocaml-getopt ocaml-type-conv ocaml-xmlm omake ocaml-xmlm-devel xen-devel xapi-libs-devel xapi-libs-utils

You'll also need mercurial or git depending upon whether you're building the unstable version of XCP or 1.0. These are both available in the EPEL repository (http://download.fedora.redhat.com/pub/epel/5/i386/repoview/)

Check out the xapi source:

hg clone http://xenbits.xen.org/XCP/xen-api.hg

or

git clone git@github.com:xen-org/xen-api.git

and then build it:

cd xen-api.hg
make

or

cd xen-api
make

It is also possible to build xapi in a non-CentOS 5.5 32-bit environment with the help of mock. Instructions for this are coming.