Xen Networking
|
Needs Review
This page has been marked as out-of-date and needs review and its content needs to be updated for Xen 4.x. Note that section 1-3 have been recently updated. |
Virtual Network Interfaces
When a domain is started with a network interface then the Xen toolstack will create a pair of network devices. The first of these (the frontend) will reside in the guest domain while the second (the backend) will reside in the backend domain (typically Dom0). A similar pair of devices is created for each virtual network interface
The frontend devices appear much like any other physical Ethernet NIC in the guest domain. Typically under Linux it is bound to the xen-netfront driver and creates a device ethN. Under NetBSD and FreeBSD the frontend devices are named xennetN and xnN respectively.
The backend device is typically named such that it contains both the guest domain ID and the index of the device. Under Linux such devices are by default named vifDOMID.DEVID while under NetBSD xvifDOMID.DEVID is used.
In both cases the device naming is subject to the usual guest or backend domain facilities for renaming network devices. For the remainder of this document the default Linux naming, that is ethN for frontend and vifDOMID.DEVID for backend devices, will be used.
The front and backend devices are linked by a virtual communication channel, guest networking is achieved by arranging for traffic to pass from the backend device onto the wider network, e.g. using bridging, routing or Network Address Translation (NAT).
MAC addresses
Virtualised network interfaces in domains are given Ethernet MAC addresses. By default most Xen toolstacks will select a random address, depending on the toolstack this will either be static for the entire life time of the guest (e.g. Libvirt, XAPI or xend managed domains) or will change each time the guest is started (e.g. XL or xend unmanaged domains).
In the latter case if a fixed MAC address is required e.g. for using with DHCP then this can be be configured using the mac= option to the vif configuration directive (e.g. vif = ['mac=aa:00:00:00:00:11']). See XL Network Configuration for more details of the syntax.
When choosing MAC addresses there are in general three strategies which can be used. In decreasing order of preference these are:
- Assign an address from the range associated with an Organizationally Unique Identifier (OUI) which you control. If you do not know what this means then you likely do not control an OUI and this option does not apply to you.
- Generate a random sequence of 6 bytes, set the locally administered bit (bit 2 of the first byte) and clear the multicast bit (bit 1 of the first byte). In other words the first byte should have the bit pattern xxxxxx10 (where x is a randomly generated bit) and the remaining 5 bytes are randomly generated. See wikipedia for more details the structure of a MAC address.
- Assign a random address from within the space 00:16:3e:xx:xx:xx. 00:16:3e is an OUI assigned to the Xen project and which has been made available for Xen users for the purposes of assigning local addresses within that space.
A MAC address must be unique among all network devices (both physical and virtual) on the same local network segment (e.g. on the LAN containing the Xen host). For this reason if you do not have your own OUI to use it is in general recommended to generate a random locally administered address (the second option above) rather than using the Xen OUI (the third option) since it gives 46 bits of randomness rather than 12 which significantly reduces the chances of a clash.
Bridging
The default (and most common) Xen configuration uses bridging within the backend domain (typically domain 0) to allow all domains to appear on the network as individual hosts.
In this configuration a software bridge is created in the backend domain. The backend virtual network devices (vifDOMID.DEVID)) are added to this bridge along with an (optional) physical Ethernet device to provide connectivity off the host. By omitting the physical Ethernet device an isolated network containing only guest domains can be created.
There are two common naming schemes when using bridged networking. In one scheme the physical device eth0 is renamed to peth0 and a bridge named eth0 is created. In the other the physical device remains eth0 while the bridge is named xenbr0 (or br0 etc). We shall use the eth0+xenbr0 naming scheme here.
Of course you are free to use whatever names you like, including descriptive names (e.g. "dmz", "internal", "external" etc).
Setting up bridged networking
The recommended method for configuring bridged networking is to use your distro supplied network configuration tools as described in Host Configuration/Networking.
Prior to Xen 4.1 when xend started up it would run the network-bridge script which would reconfigure any existing physical network configuration into a bridged network configuration i.e. it would create a bridge, move the IP address from the physical device to the bridge, add the physical device to the bridge etc. However this was fragile and prone to breaking and therefore is no longer recommended.
After Xen 4.1 xend will only do this if no bridges currently exist, so as to avoid overwriting any locally configured network configuration.
The XL toolstack will never modify the network configuration and expects that the administrator will have configured the host networking appropriately.
Attaching virtual devices to the appropriate bridge
When a domU starts up the vif-bridge script is run which:
- attaches vifDOMID.DEVID to the appropriate bridge
- brings vifDOMID.DEVID up.
With XL and xend the bridge to us for each VIF can be configured using the bridge configuration key. e..g
vif=[ 'bridge=mybridge' ]
or
vif=[ 'mac=00:16:3e:01:01:01,bridge=mybridge' ]
or to create multiple interfaces attached to different bridges:
vif=[ 'mac=00:16:3e:70:01:01,bridge=br0', 'mac=00:16:3e:70:02:01,bridge=br1' ]
Links
Some relevant topics from the mailing list:
- eth0 IP in dom0 2005/01/14
- Bridging vs. Routing 2005/01/13
- Bridging vs. Routing 2004/07/18
- An attempt to explain Xen networking 2006-02-01
- Firewall in domU with bridging
- Xen and Shorewall (with bridging)
- Xen and the Art of Consolidation (with bridging)
- Another way for making multiple Xen bridges
- Advanced bridging (2007/05) You can also have a look to: http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1310165,00.html#
Routing
This section applies only if you choose to use network-route and vif-route instead of network-bridge and vif-bridge.
Illustration on network-route and vif-route:
Routing creates a point-to-point link between dom0 and each domU. Routes to each domU are added to dom0's routing table, so domU must have a known (static) IP. DHCP doesn't work, because the route won't be created, and the DHCP offer won't arrive.
network-route
When xend starts up, it runs network-route which:
- enables ip forwarding within dom0
vif-route
When domU starts up, xend runs (within dom0) vif-route which:
- copies the ip address from
eth0tovif<id#>.0 - brings up
vif<id#>.0 - adds host static route for domU's ip address specified in domU config file, pointing at interface
vif<id#>.
More information on vif-route can be found here: http://wiki.xensource.com/xenwiki/vif-route
Reference
Virtual Network
The virtual network is currently a non-standard configuration.
Illustration of virtual network setups:
The virtual network configuration places all domU on a shared virtual network with dom0. This allows domU to use a DHCP server provided by dom0, without allowing DHCP requests from domU to escape onto the physical network. (As far as I can tell, vif0.0 and dummy0 are not strictly required.)
Links
- Setup with DHCP server on dom0, with domU VMs receiving their IP addresses from DHCP and dynamically setting their DNS hostnames: Getting Xen up and running: part II (archive.org copy from 2008, article from 2007)
- brouter (bridge plus router) setup for placing a virtual network in a DMZ from the opensuse wiki Xen3 and a Virtual Network (archive.org copy from 2010, article from 2007/2009), version with working images on tr.opensuse.org
Interface names
The default configuration for Xen systems is to use bridging. When xend starts it creates a bridge called xen-br0. xend takes the IP address etc. from eth0 and assigns to xen-br0 (as dom0's interface onto the bridge). So dom0's own external-facing interface is now xen-br0 - make sure any firewall config reflects this.
VLANs
1st method of having VMs using vlan interfaces with XEN.
Multiple tagged VLANs can be supported by configuring 802.1Q VLAN support into domain 0. A local interface in dom0 is needed for each desired VLAN although it need not have an IP address in dom0. A bridge can be set up for each VLAN, and guests can then connect the the appropriate bridge.
My (JamesBulpin) prefered method is to define the bridge as an interface which is not brought up automatically (e.g. for Debian /etc/network/interfaces, with no "auto" entry):
iface xen-br293 inet manual
up vconfig add eth0 293
up /etc/xen/scripts/network start netdev=eth0.293 bridge=xen-br293 antispoof=no
up /sbin/ifconfig eth0.293 up
down /etc/xen/scripts/network stop netdev=eth0.293 bridge=xen-br293 antispoof=no
down vconfig rem eth0.293
I then add an init.d script to bring the interface up between xend and xendomains starting.
2nd method for XEN with vlans: modify network-bridge script
I (OrianaPalivan) used this method because I needed to have a particular use (and quite general on my oppinion) of my VMs. Hope this may help you with your future VMs.
It was tested with xen 3.3, and Debian Etch 4.r05. Please notice that with xen 3.3, the name of the bridge is the name of the interface in the domain 0 it is attached to (so no xenbr anymore by default).
Domain 0 has 2 interfaces: eth0 and eth1, each of them has 2 vlan interfaces attached, for example eth0.20, eth0.21 and eth1.3916 and eth1.3999.
Here is the network configuration of the VMs:
- VM1: eth0 -> domain 0, eth0 / eth1 -> domain 0, eth1
- VM2: eth0 -> domain 0, eth0.20 / eth1 -> domain 0, eth1.3999
- VM3: eth0 -> domain 0, eth0.21 / eth1 -> domain 0, eth1.3916
- VM4: eth0 -> domain 0, eth0 / eth1 -> domain 0, eth1
I needed to be able to reach domain 0 thru IP.
Here are the steps for having all the interfaces from domain 0 (vlan or not vlan interface) ready to be used by the VMs:
- Configure vlan interfaces (needs the vlan package)
- Change the network-bridge script (located in /etc/xen/scripts)
- Copy for example the network-bridge script to network-bridge-withvlan script
- Edit the network-bridge-withvlan script. Comment each time ifup or ifdown commands are executed.The reason is that ifdown ends with error for vlan interfaces.
Change do_ifup() function:
do_ifup() {
# if ! ifup $1 ; then
if [ -n "$addr_pfx" ] ; then
# use the info from get_ip_info()
ip addr flush $1
ip addr add ${addr_pfx} dev $1
ip link set dev $1 up
[ -n "$gateway" ] && ip route add default via ${gateway}
fi
#fi
}
Change op_start() and op_stop() functions:
op_start () {
[...]
# if ! ifdown ${netdev}; then
# If ifdown fails, remember the IP details.
get_ip_info ${netdev}
ip link set ${netdev} down
ip addr flush ${netdev}
# fi
op_stop () {
[...]
# if ! ifdown ${bridge}; then
get_ip_info ${bridge}
<!-- # fi
-->
- Create your network-bridge wrapper which will call the network-bridge-withvlan:
<16:39>[root:/etc/xen/scripts]# cat network-bridge-wrapper #!/bin/sh /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0.20 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0.21 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1.3916 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1.3999#
- Call your new network-bridge-wrapper in your xend-config.xsp
(network-script network-bridge-wrapper)
- Before you restart your domain 0, do not forget to disable transmit checksum offloading (see below). You can make a script at startup:
<16:40>[root:/etc/xen]# cat /etc/init.d/xen-vlan #!/bin/sh ethtool -K eth0 tx off ethtool -K eth1 tx off
- At boot time, and before starting your VMs, make a :
/etc/init.d/networking restart
In some cases, the static routes you have may not come up after xend starts.
- In the configuration file of your VMs, you can now use the vlan interfaces:
VM1:
vif = ['type=ioemu, mac=00:16:3E:00:00:13, bridge=eth0' , 'type=ioemu, mac=00:16:3E:00:00:14, bridge=eth1']
VM2:
vif = ['type=ioemu, mac=00:16:3E:00:00:10, bridge=eth0.20' , 'type=ioemu, mac=00:16:3E:00:00:12, bridge=eth1.3999']
VM3:
vif = ['type=ioemu, mac=00:16:3E:00:00:01, bridge=eth0.21' , 'type=ioemu, mac=00:16:3E:00:00:02, bridge=eth1.3916']
VM4:
vif = ['type=ioemu, mac=00:16:3E:00:00:03, bridge=eth0' , 'type=ioemu, mac=00:16:3E:00:00:04, bridge=eth1']
Yet Another ASCII Graphics Description of Xen Networking
Xen 3.1- Networking
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | peth0 xenbr0 | | xenbr1 peth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- vif0.0 is absolute different from vif1.0. vif0.0 is created by netloop, while vif1.0 is created by netback.
Xen 3.2+ Networking
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | peth0 | | peth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- eth0 and eth1 are the bridge names, which is a bit confusing.
Alternative Xen Networking Architecture
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | eth0 | | eth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- based on xen 3.3, change bridge name to xenbr0, xenbr1, ...
- xenbrX has an active address, which is used by dom0 to communicate with outside.
Xen Networking with vlan
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| eth0 eth1 |
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | eth0.100 | | eth1.200 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- With this configuration, DomUs are completely unaware of the fact that they are utilizing a VLAN, all the work is done within the bridges in Dom0.
- Dom0 is aware of the traffic within the VLAN, because it has an active address on the xenbrX interfaces. To prevent it, don't give the xenbrX an active address, but configure a extra interface for management.
- There are two things may need to be configured:
- If your ethernet card does not natively support VLAN tags, you will have to set the maximum MTU to 1496 to make room for the tag. With command:
# ifconfig eth0 mtu 1496
- With the DomUs bridged to VLAN interfaces, some optimizations need to be disabled or tcp and udp connections will fail. This is done by disabling transmit checksum offloading:
# ethtool -K eth0 tx off
- Need further test in the production environment.
Xen Networking with bonding
PRT0 PRT1 PRT2 PRT3
| | | |
+--------------+---+---------------------------+---+--------------+
| | | | | |
| eth0 eth1 eth2 eth3 |
| | | | | |
| +-+-+ +-+-+ |
| | | |
| +--------------+--------------+ +--------------+--------------+ |
| | | | | | | |
| | bond0 | | bond1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- Need some more destructive testing.
Xen Networking with vlan on bonding
PRT0 PRT1 PRT2 PRT3
| | | |
+--------------+---+---------------------------+---+--------------+
| | | | | |
| eth0 eth1 eth2 eth3 |
| | | | | |
| +-+-+ +-+-+ |
| | | |
| bond0 bond1 |
| | | |
| +--------------+--------------+ +--------------+--------------+ |
| | | | | | | |
| | bond0.100 | | bond1.200 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ftp | | www ftp ssh dns | | www ftp ssh dns | | ssh dns |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- The connections at the top are switch ports - probably on 2 switches with an ISL
- bond0 has eth0 and eth1 ; bond1 has eth2 and eth3
- In the VMs eth0 maps to bond0.100 and eth1 maps to bond1.200
- Protocols suggest a service VLAN (100) and a mgmt VLAN (200)
Collection of Examples
this is a collection of configurations that can help you to understand how it is working in detail and rapidly deploy one of them.
if you want to add a configuration to this list, please look at http://wiki.xensource.com/xenwiki/ under "Join Wiki as a Contributor"
