RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial
From Xen
Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen
Requirements:
- You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
- You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.
Components used in this tutorial:
- Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
- Intel Xeon CPU L5640.
- Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
- IOMMU/VT-d enabled in BIOS.
- CentOS 5.8 x86_64 DVD1.
- Stock Xen rpms from CentOS 5.8.
- Stock kernel-xen from CentOS 5.8 as dom0 kernel.
EL5 Host installation and configuration
- Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
- Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
- After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
- Do all the usual network/IP, hostname, DNS etc configuration.
- Update the system with "yum update".
- Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off chkconfig mcstrans off chkconfig haldaemon off chkconfig hidd off chkconfig autofs off chkconfig avahi-daemon off chkconfig xfs off chkconfig bluetooth off chkconfig pcscd off chkconfig iptables off chkconfig ip6tables off
- Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
- Edit "/etc/xen/xend-config.sxp" and disable Xen network-script line:
#(network-script network-bridge)
- We want to configure dom0 networking settings and bridges ourselves.
- Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz #hiddenmenu title CentOS (2.6.18-308.11.1.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00 module /initrd-2.6.18-308.11.1.el5xen.img
- Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
- Make sure the Xen entry is the default in Grub
- Reboot the system to Xen.
- When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 2048 12 r----- 788.0
- Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info host : dom0.localdomain release : 2.6.18-308.11.1.el5xen version : #1 SMP Tue Jul 10 09:29:47 EDT 2012 machine : x86_64 nr_cpus : 12 nr_nodes : 1 sockets_per_node : 1 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 2266 hw_caps : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001 total_memory : 49139 free_memory : 46145 node_to_cpu : node0:0-11 xen_major : 3 xen_minor : 1 xen_extra : .2-308.11.1.el5 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Tue Jul 10 08:40:13 EDT 2012 xend_config_format : 2