RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial
From Xen
Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen
Requirements:
- You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
- You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.
Components used in this tutorial:
- Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
- Intel Xeon CPU L5640.
- Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
- IOMMU/VT-d enabled in BIOS.
- CentOS 5.8 x86_64 DVD1.
- Stock Xen rpms from CentOS 5.8.
- Stock kernel-xen from CentOS 5.8 as dom0 kernel.
EL5 Host installation and configuration
- Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
- Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
- After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
- Do all the usual network/IP, hostname, DNS etc configuration.
- Make sure there is free unallocated space in the LVM volume group:
[root@dom0 ~]# vgdisplay --- Volume group --- VG Name VolGroup00 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 38 VG Access read/write VG Status resizable MAX LV 0 Cur LV 27 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 1.82 TB PE Size 32.00 MB Total PE 59551 Alloc PE / Size 55670 / 1.70 TB Free PE / Size 3881 / 121.28 GB VG UUID QuouSc-MXj2-Hhdo-Ut9k-RLLO-jk9N-RjQD6E
- Note the "Free PE / Size" line on the vgdisplay command output above.
- Update the system with "yum update".
- Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off chkconfig mcstrans off chkconfig haldaemon off chkconfig hidd off chkconfig autofs off chkconfig avahi-daemon off chkconfig xfs off chkconfig bluetooth off chkconfig pcscd off chkconfig iptables off chkconfig ip6tables off
- Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
- Edit "/etc/xen/xend-config.sxp" and disable (comment out) Xen network-script line:
#(network-script network-bridge)
- We want to configure dom0 networking settings and bridges ourselves.
- Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz #hiddenmenu title CentOS (2.6.18-308.11.1.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00 module /initrd-2.6.18-308.11.1.el5xen.img
- Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
- Make sure the Xen entry is the default in Grub
- Reboot the system to Xen.
- When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 2048 12 r----- 788.0
- Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info host : dom0.localdomain release : 2.6.18-308.11.1.el5xen version : #1 SMP Tue Jul 10 09:29:47 EDT 2012 machine : x86_64 nr_cpus : 12 nr_nodes : 1 sockets_per_node : 1 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 2266 hw_caps : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001 total_memory : 49139 free_memory : 46145 node_to_cpu : node0:0-11 xen_major : 3 xen_minor : 1 xen_extra : .2-308.11.1.el5 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Tue Jul 10 08:40:13 EDT 2012 xend_config_format : 2
- Versions of xen related rpms:
[root@dom0 ~]# rpm -qa | grep xen xen-libs-3.0.3-135.el5_8.4 xen-3.0.3-135.el5_8.4 xen-libs-3.0.3-135.el5_8.4 kernel-xen-2.6.18-308.11.1.el5
- Note that the actual Xen hypervisor is version 3.1.2 (+ a lot of patches from later Xen versions), only the userland tools are 3.0.3 based. Xen hypervisor is included in kernel-xen rpm in RHEL5/CentOS5.
Intel SR-IOV NIC
- In this tutorial we're using the following Intel SR-IOV capable 10 Gbit/sec 82599 NIC:
[root@dom0 ~]# lspci | grep 10-Gig 03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
- which uses the Intel ixgbe driver, as can be seen from the "ethtool -i" output:
[root@dom0 ~]# ethtool -i eth2 driver: ixgbe version: 3.4.8-k firmware-version: 0.9-3 bus-info: 0000:03:00.0 [root@dom0 ~]# ethtool -i eth3 driver: ixgbe version: 3.4.8-k firmware-version: 0.9-3 bus-info: 0000:03:00.1
Enabling IOMMU support in Xen command line options
- Edit "/boot/grub/grub.conf" and add "iommu=1" option for Xen, and "pci_pt_e820_access=on" option for dom0 Linux kernel (vmlinuz). Both options are required:
default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz #hiddenmenu title CentOS (2.6.18-308.11.1.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all iommu=1 module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00 pci_pt_e820_access=on module /initrd-2.6.18-308.11.1.el5xen.img
Enable Intel SR-IOV NIC Virtual Functions (VFs) in the ixgbe driver options
- Add the following line to end of "/etc/modprobe.conf":
options ixgbe max_vfs=8
- 8 is just an example here, 82599 NIC supports up to 64 VFs.
- Blacklist the Intel VF driver (ixgbevf) in dom0 so that dom0 kernel doesn't try to use the Virtual Functions (we want to PCI passthru them to Xen VMs!). Create "/etc/modprobe.d/blacklist-ixgbevf.conf" file with the following contents:
# intel ixgbe sr-iov vf (virtual function) driver blacklist ixgbevf
- Now reboot the system to activate the changes.
- After reboot check that the IOMMU support gets enabled in Xen hypervisor from the Xen dmesg log:
[root@dom0 ~]# xm dmesg | grep -i vt-d | grep -i enable (XEN) Intel VT-d has been enabled (XEN) Intel VT-d snoop control enabled (XEN) [VT-D]iommu.c:619: iommu_enable_translation: iommu->reg = ffff828bfff58000
- If VT-d IOMMU doesn't get enabled, read the full "xm dmesg | less" log for more information.
- Also verify you can see the VFs in "lspci" output:
[root@dom0 ~]# lspci | grep "Virtual Function" 03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 03:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
- Why 16 VFs when we configured max_vfs=8? There's 8 per NIC/port, so 16 total for 2 ports.
- Also check the dom0 Linux kernel dmesg for the ixgbe driver logs:
[root@dom0 ~]# dmesg | grep ixgbe ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k ixgbe: Copyright (c) 1999-2011 Intel Corporation. ixgbe 0000:03:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 ixgbe 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1c ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: E81283-002 ixgbe 0000:03:00.0: eth2: IOV is enabled with 8 VFs ixgbe 0000:03:00.0: eth2: IOV: VF 0 is enabled MAC fe:71:c2:a8:4d:28 ixgbe 0000:03:00.0: eth2: IOV: VF 1 is enabled MAC a2:cd:8b:d7:a0:3e ixgbe 0000:03:00.0: eth2: IOV: VF 2 is enabled MAC 2a:e4:d6:69:a2:9a ixgbe 0000:03:00.0: eth2: IOV: VF 3 is enabled MAC 5a:56:e0:a4:7d:36 ixgbe 0000:03:00.0: eth2: IOV: VF 4 is enabled MAC ae:f5:bb:ff:f4:bc ixgbe 0000:03:00.0: eth2: IOV: VF 5 is enabled MAC 2a:be:89:3e:53:3b ixgbe 0000:03:00.0: eth2: IOV: VF 6 is enabled MAC 1a:61:ce:c8:a7:c0 ixgbe 0000:03:00.0: eth2: IOV: VF 7 is enabled MAC 96:d7:3f:c4:f2:aa ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection ixgbe 0000:03:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 ixgbe 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1d ixgbe 0000:03:00.1: MAC: 2, PHY: 8, SFP+: 4, PBA No: E81283-002 ixgbe 0000:03:00.1: eth3: IOV is enabled with 8 VFs ixgbe 0000:03:00.1: eth3: IOV: VF 0 is enabled MAC ee:b0:2b:65:c4:6c ixgbe 0000:03:00.1: eth3: IOV: VF 1 is enabled MAC 0a:f0:de:c0:b1:4d ixgbe 0000:03:00.1: eth3: IOV: VF 2 is enabled MAC 9a:ab:7f:4a:d6:ce ixgbe 0000:03:00.1: eth3: IOV: VF 3 is enabled MAC ee:f3:59:15:1b:02 ixgbe 0000:03:00.1: eth3: IOV: VF 4 is enabled MAC be:cd:68:3c:bc:f2 ixgbe 0000:03:00.1: eth3: IOV: VF 5 is enabled MAC a2:66:eb:00:25:80 ixgbe 0000:03:00.1: eth3: IOV: VF 6 is enabled MAC 6e:3e:da:74:84:63 ixgbe 0000:03:00.1: eth3: IOV: VF 7 is enabled MAC de:06:3d:af:8c:ce ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
Configuring Xen pciback for VF PCI passthru
- First check available PCI devices for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices Error: pciback not loaded?
- The error message makes sense, we haven't yet configured any PCI devices for passthru, and we haven't loaded the Xen pciback module in dom0 kernel.
- Edit "/etc/modprobe.conf" and add options for the Xen pciback dom0 kernel module:
options pciback hide=(03:10.0)(03:10.1)(03:10.2)(03:10.3)(03:10.4)(03:10.5)(03:10.6)(03:10.7)(03:11.0)(03:11.1)(03:11.2)(03:11.3)(03:11.4)(03:11.5)(03:11.6)(03:11.7)
- Here we are "hiding" all the Virtual Function PCI device IDs, so they can be used for Xen PCI passthru.
- Now load the "pciback" module in dom0 kernel:
modprobe pciback
- After loading "pciback" module check the dom0 Linux kernel dmesg:
pciback 0000:03:10.0: seizing device pciback 0000:03:10.2: seizing device pciback 0000:03:10.4: seizing device pciback 0000:03:10.6: seizing device pciback 0000:03:11.0: seizing device pciback 0000:03:11.2: seizing device pciback 0000:03:11.4: seizing device pciback 0000:03:11.6: seizing device pciback 0000:03:10.1: seizing device pciback 0000:03:10.3: seizing device pciback 0000:03:10.5: seizing device pciback 0000:03:10.7: seizing device pciback 0000:03:11.1: seizing device pciback 0000:03:11.3: seizing device pciback 0000:03:11.5: seizing device pciback 0000:03:11.7: seizing device PCI: Enabling device 0000:03:11.7 (0000 -> 0002) PCI: Enabling device 0000:03:11.5 (0000 -> 0002) PCI: Enabling device 0000:03:11.3 (0000 -> 0002) PCI: Enabling device 0000:03:11.1 (0000 -> 0002) PCI: Enabling device 0000:03:10.7 (0000 -> 0002) PCI: Enabling device 0000:03:10.5 (0000 -> 0002) PCI: Enabling device 0000:03:10.3 (0000 -> 0002) PCI: Enabling device 0000:03:10.1 (0000 -> 0002) PCI: Enabling device 0000:03:11.6 (0000 -> 0002) PCI: Enabling device 0000:03:11.4 (0000 -> 0002) PCI: Enabling device 0000:03:11.2 (0000 -> 0002) PCI: Enabling device 0000:03:11.0 (0000 -> 0002) PCI: Enabling device 0000:03:10.6 (0000 -> 0002) PCI: Enabling device 0000:03:10.4 (0000 -> 0002) PCI: Enabling device 0000:03:10.2 (0000 -> 0002) PCI: Enabling device 0000:03:10.0 (0000 -> 0002)
- Now check again what PCI devices are ready for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices 0000:03:10.1 0000:03:10.0 0000:03:10.2 0000:03:10.3 0000:03:10.4 0000:03:10.5 0000:03:10.6 0000:03:10.7 0000:03:11.0 0000:03:11.1 0000:03:11.2 0000:03:11.3 0000:03:11.4 0000:03:11.5 0000:03:11.6 0000:03:11.7
Install new CentOS 5.8 PV domU
- Create a new LVM volume for the new VM:
[root@dom0 ~]# lvcreate -L18G -nc58x64pv /dev/VolGroup00 Logical volume "c58x64pv" created
- Install a new CentOS 5.8 x64 PV domU, for example with virt-install. ssh with X11 forwarding enabled to dom0, so you can view the GUI installer (VNC client virt-viewer session) on your local desktop/X-server:
[root@dom0 ~]# virt-install --debug -b virbr0 -n c58x64pv -r 1024 --vcpus=2 -f /dev/VolGroup00/c58x64pv --vnc -p -l "http://ftp.funet.fi/pub/mirrors/centos.org/5.8/os/x86_64"
- Replace "ftp.funet.fi" with your local CentOS mirror site.
- Note that virt-install will attach the VM to "virbr0" ethernet bridge, which provides DHCP+NAT services, so the VM can connect to the Internet thru dom0.
- Install the VM with minimal package selection profile, no need to choose any extra packages, GUI, or anything.