Xen Networking
|
Needs Review
This page has been marked as out-of-date and needs review and its content needs to be updated for Xen 4.x. |
Virtual Ethernet interfaces
Xen creates, by default, seven pair of "connected virtual ethernet interfaces" for use by dom0. Think of them as two ethernet interfaces connected by an internal crossover ethernet cable. veth0 is connected to vif0.0, veth1 is connected to vif0.1, etc, up to veth7 -> vif0.7. You can use them by configuring IP and MAC addresses on the veth# end, then attaching the vif0.# end to a bridge.
Diagram of Physical and Logical network cards:
Every time you create a running domU instance, it is assigned a new domain id number. You don't get to pick the number, sorry. The first domU will be id #1. The second one started will be #2, even if #1 isn't running anymore.
For each new domU, Xen creates a new pair of "connected virtual ethernet interfaces", with one end in domU and the other in dom0. For linux domU's, the device name it sees is named eth0. The other end of that virtual ethernet interface pair exists within dom0 as interface vif<id#>.0. For example, domU #5's eth0 is attached to vif5.0. If you create multiple network interfaces for a domU, it's ends will be eth0, eth1, etc, whereas the dom0 end will be vif<id#>.0, vif<id#>.1, etc.
Logical network cards connected between dom0 and dom1:
When a domU is shutdown, the virtual ethernet interfaces for it are deleted.
MAC addresses
Virtualised network interfaces in domains are given Ethernet MAC addresses. By default xend will select a random address, this will differ between instantiations of the domain. If it is required to have a fixed MAC address for a domain (e.g. for using with DHCP) then this can be configured using the mac= option to the vif configuration directive (e.g. vif = ['mac=aa:00:00:00:00:11']).
When choosing MAC addresses to use, ensure you choose a unicast address. That is, one with the low bit of the first octet set to zero. For example, an address starting aa: is OK but ab: is not. It is best to keep to the range of addresses declared to be "locally assigned" (rather than allocated globally to hardware vendors). These have the second lowest bit set to one in the first octet. For example, aa: is OK, a8: isn't.
In summary, an address of the following form should be OK:
XY:XX:XX:XX:XX:XX
where X is any hexadecimal digit, and Y is one of 2, 6, A or E.
It's recommended to use a MAC address inside the range 00:16:3e:xx:xx:xx. This address range is reserved for use by Xen.
Bridging
Illustration on network-bridge and vif-bridge:
The default Xen configuration uses bridging within domain 0 to allow all domains to appear on the network as individual hosts. If extensive use of iptables is made in domain 0 (e.g. a firewall) then this can affect bridging because bridged packets pass through the PREROUTING, FORWARD and POSTROUTING iptables chains. This means that packets being bridged between guest domains and the external network will need to be permitted to pass those chains. The most likely problem is the FORWARD chain being configured to DROP or REJECT packets (this is different from IP forwarding in the kernel).
iptable FORWARDing can be disabled for all packets; to prevent the dom0 from acting as an IP router: echo 0 > /proc/sys/net/ipv4/ip_forward.
A slightly more secure method is to allowing packet forwarding (at the iptables level) between the external physical interface and the vifs for the guests. For a machine with a single ethernet card this would be:
iptables -A FORWARD -m physdev --physdev-in eth0 --physdev-out '!' eth0 -j ACCEPT iptables -A FORWARD -m physdev --physdev-out eth0 --physdev-in '!' eth0 -j ACCEPT
(needs the ipt_physdev [aka xt_physdev] module to be available).
The ebtables project has an interesting document on the interaction of bridging and iptables.
Packet flow in bridging
Packet arrives at hardware, is handled by dom0 Ethernet driver and appears on peth0. peth0 is bound to to the bridge, so its passed to the bridge from there. This step is run on Ethernet level, no IP addresses are set on peth0 or bridge.
Now the bridge distributes the packet, just like a switch would. Filtering at this stage would be possible with ebtables.
Now there's a number of vifX.Y connected to the bridge, it decides where to put the packet based on the receiver's MAC.
The vif interface puts the packet into Xen, which then puts the packet back to the domain the vif leads to (its also done that way for dom0, hence the vif0.0->(v)eth0 pair).
The target device in the dom0/domU finally has an IP address, you can apply iptables filtering here.
network-bridge
When xend starts up, it runs the network-bridge script, which:
- creates a new bridge named
xenbr0 - "real" ethernet interface
eth0is brought down - the IP and MAC addresses of
eth0are copied to virtual network interfaceveth0 - real interface
eth0is renamedpeth0 - virtual interface
veth0is renamedeth0 peth0andvif0.0are attached to bridgexenbr0.Please notice that in xen 3.3, the default bridge name is the same than the interface it is attached to. Eg: bridge name eth0, eth1 or ethX.VlanID- the bridge,
peth0,eth0andvif0.0are brought up
It is good to have the physical interface and the dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does not affect the traffic to the domUs (just for protecting dom0 alone).
vif-bridge
When a domU starts up, xend (running in dom0) runs the vif-bridge script, which:
- attaches
vif<id#>.0toxenbr0 vif<id#>.0is brought up
Additional Notes
- you can change the bridge name from
xenbr0using:
(network-script 'network-bridge bridge=mybridge')
inxend-config.sxpand rebooting or restartingxend
- remember to configure the bridge to attach to in the domU's config file using:
vif=[ 'bridge=mybridge' ]
or perhaps something like:
vif=[ 'mac=00:16:3e:01:01:01,bridge=mybridge' ]
- you can create multiple network interfaces, and attach them to different bridges using:
vif=[ 'mac=00:16:3e:70:01:01,bridge=br0', 'mac=00:16:3e:70:02:01,bridge=br1' ]
- if you want to use multiple bridges, you must create them yourself, either manually, or via your own startup script, or via a custom script to replace
network-bridge. For example:
$ cd /etc/xen/scripts
$ cp network-bridge network-custom
$ cp vif-bridge vif-custom
$ vi /etc/xen/xend-config.sxp
(network-script network-custom)
(vif-script vif-custom)
$ vi network-custom
# whatever you want
- before you connect a physical interface to a bridge, remember to reset it's mac and turn arp off. For example:
# ip link set eth1 down # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off # brctl addif br1 eth1 # ip link set eth1 up
- With Xen 3.0 the best method for additional bridges is to use the default Xen scripts with a slight modification. Following the XenBug #332. For example in a two bridge network with eth0 and eth1. Create /etc/xen/scripts/my-network-script with
#!/bin/sh dir=$(dirname "$0") "$dir/network-bridge" "$@" vifnum=0 "$dir/network-bridge" "$@" vifnum=1
- With Xen 3.2.1 (tested on Debian Etch 4.0r3), here is a script example that creates two virtual interfaces corresponding to the 2 physical network interfaces
# xemacs /etc/xen/scripts/network-bridge-wrapper #!/bin/sh /etc/xen/scripts/network-bridge "$@" netdev=eth0 /etc/xen/scripts/network-bridge "$@" netdev=eth1
The $1 will use the argument of xend (in the /etc/xen/xend-config.sxp configuration file). If there is a default physical network interface, the standard network-bridge script of Xen will create a vif for this interface, and not the other ones also.
- (Additional note by steve_from_moreover - May be stating the obvious but remember to do - chmod 755 /etc/xen/scripts/my-network-script or when you reboot it will silently not be able to run this script).
- On SuSE Linux (at least), each interface requires an ifcfg script in /etc/sysconfig/network, e.g. /etc/sysconfig/network/ifcfg-eth1. Otherwise, network-bridge will create the bridge with no interfaces attached.
- Then change /etc/xen/xend-config.sxp with the following (network-script my-network-script).
- The same principle can apply to networks without a physical ethernet device. Use a dummy interface with
"$dir/network-bridge" "$@" vifnum=2 netdev=dummy0
Links
Some relevant topics from the mailing list:
- eth0 IP in dom0 2005/01/14
- Bridging vs. Routing 2005/01/13
- Bridging vs. Routing 2004/07/18
- An attempt to explain Xen networking 2006-02-01
- Firewall in domU with bridging
- Xen and Shorewall (with bridging)
- Xen and the Art of Consolidation (with bridging)
- Another way for making multiple Xen bridges
- Advanced bridging (2007/05) You can also have a look to: http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1310165,00.html#
Routing
This section applies only if you choose to use network-route and vif-route instead of network-bridge and vif-bridge.
Illustration on network-route and vif-route:
Routing creates a point-to-point link between dom0 and each domU. Routes to each domU are added to dom0's routing table, so domU must have a known (static) IP. DHCP doesn't work, because the route won't be created, and the DHCP offer won't arrive.
network-route
When xend starts up, it runs network-route which:
- enables ip forwarding within dom0
vif-route
When domU starts up, xend runs (within dom0) vif-route which:
- copies the ip address from
eth0tovif<id#>.0 - brings up
vif<id#>.0 - adds host static route for domU's ip address specified in domU config file, pointing at interface
vif<id#>.
More information on vif-route can be found here: http://wiki.xensource.com/xenwiki/vif-route
Reference
Virtual Network
The virtual network is currently a non-standard configuration.
Illustration of virtual network setups:
The virtual network configuration places all domU on a shared virtual network with dom0. This allows domU to use a DHCP server provided by dom0, without allowing DHCP requests from domU to escape onto the physical network. (As far as I can tell, vif0.0 and dummy0 are not strictly required.)
Links
- My (sapphirecat) virtual network and DHCP setup: Getting Xen up and running: part II (archive.org copy from 2008)
- Another virtual network setup: http://en.opensuse.org/Xen3_and_a_Virtual_Network
Interface names
The default configuration for Xen systems is to use bridging. When xend starts it creates a bridge called xen-br0. xend takes the IP address etc. from eth0 and assigns to xen-br0 (as dom0's interface onto the bridge). So dom0's own external-facing interface is now xen-br0 - make sure any firewall config reflects this.
VLANs
1st method of having VMs using vlan interfaces with XEN.
Multiple tagged VLANs can be supported by configuring 802.1Q VLAN support into domain 0. A local interface in dom0 is needed for each desired VLAN although it need not have an IP address in dom0. A bridge can be set up for each VLAN, and guests can then connect the the appropriate bridge.
My (JamesBulpin) prefered method is to define the bridge as an interface which is not brought up automatically (e.g. for Debian /etc/network/interfaces, with no "auto" entry):
iface xen-br293 inet manual
up vconfig add eth0 293
up /etc/xen/scripts/network start netdev=eth0.293 bridge=xen-br293 antispoof=no
up /sbin/ifconfig eth0.293 up
down /etc/xen/scripts/network stop netdev=eth0.293 bridge=xen-br293 antispoof=no
down vconfig rem eth0.293
I then add an init.d script to bring the interface up between xend and xendomains starting.
2nd method for XEN with vlans: modify network-bridge script
I (OrianaPalivan) used this method because I needed to have a particular use (and quite general on my oppinion) of my VMs. Hope this may help you with your future VMs.
It was tested with xen 3.3, and Debian Etch 4.r05. Please notice that with xen 3.3, the name of the bridge is the name of the interface in the domain 0 it is attached to (so no xenbr anymore by default).
Domain 0 has 2 interfaces: eth0 and eth1, each of them has 2 vlan interfaces attached, for example eth0.20, eth0.21 and eth1.3916 and eth1.3999.
Here is the network configuration of the VMs:
- VM1: eth0 -> domain 0, eth0 / eth1 -> domain 0, eth1
- VM2: eth0 -> domain 0, eth0.20 / eth1 -> domain 0, eth1.3999
- VM3: eth0 -> domain 0, eth0.21 / eth1 -> domain 0, eth1.3916
- VM4: eth0 -> domain 0, eth0 / eth1 -> domain 0, eth1
I needed to be able to reach domain 0 thru IP.
Here are the steps for having all the interfaces from domain 0 (vlan or not vlan interface) ready to be used by the VMs:
- Configure vlan interfaces (needs the vlan package)
- Change the network-bridge script (located in /etc/xen/scripts)
- Copy for example the network-bridge script to network-bridge-withvlan script
- Edit the network-bridge-withvlan script. Comment each time ifup or ifdown commands are executed.The reason is that ifdown ends with error for vlan interfaces.
Change do_ifup() function:
do_ifup() {
# if ! ifup $1 ; then
if [ -n "$addr_pfx" ] ; then
# use the info from get_ip_info()
ip addr flush $1
ip addr add ${addr_pfx} dev $1
ip link set dev $1 up
[ -n "$gateway" ] && ip route add default via ${gateway}
fi
#fi
}
Change op_start() and op_stop() functions:
op_start () {
[...]
# if ! ifdown ${netdev}; then
# If ifdown fails, remember the IP details.
get_ip_info ${netdev}
ip link set ${netdev} down
ip addr flush ${netdev}
# fi
op_stop () {
[...]
# if ! ifdown ${bridge}; then
get_ip_info ${bridge}
<!-- # fi
-->
- Create your network-bridge wrapper which will call the network-bridge-withvlan:
<16:39>[root:/etc/xen/scripts]# cat network-bridge-wrapper #!/bin/sh /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0.20 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth0.21 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1.3916 /etc/xen/scripts/network-bridge-withvlan "$@" netdev=eth1.3999#
- Call your new network-bridge-wrapper in your xend-config.xsp
(network-script network-bridge-wrapper)
- Before you restart your domain 0, do not forget to disable transmit checksum offloading (see below). You can make a script at startup:
<16:40>[root:/etc/xen]# cat /etc/init.d/xen-vlan #!/bin/sh ethtool -K eth0 tx off ethtool -K eth1 tx off
- At boot time, and before starting your VMs, make a :
/etc/init.d/networking restart
In some cases, the static routes you have may not come up after xend starts.
- In the configuration file of your VMs, you can now use the vlan interfaces:
VM1:
vif = ['type=ioemu, mac=00:16:3E:00:00:13, bridge=eth0' , 'type=ioemu, mac=00:16:3E:00:00:14, bridge=eth1']
VM2:
vif = ['type=ioemu, mac=00:16:3E:00:00:10, bridge=eth0.20' , 'type=ioemu, mac=00:16:3E:00:00:12, bridge=eth1.3999']
VM3:
vif = ['type=ioemu, mac=00:16:3E:00:00:01, bridge=eth0.21' , 'type=ioemu, mac=00:16:3E:00:00:02, bridge=eth1.3916']
VM4:
vif = ['type=ioemu, mac=00:16:3E:00:00:03, bridge=eth0' , 'type=ioemu, mac=00:16:3E:00:00:04, bridge=eth1']
Yet Another ASCII Graphics Description of Xen Networking
Xen 3.1- Networking
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | peth0 xenbr0 | | xenbr1 peth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- vif0.0 is absolute different from vif1.0. vif0.0 is created by netloop, while vif1.0 is created by netback.
Xen 3.2+ Networking
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | peth0 | | peth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- eth0 and eth1 are the bridge names, which is a bit confusing.
Alternative Xen Networking Architecture
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | eth0 | | eth1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- based on xen 3.3, change bridge name to xenbr0, xenbr1, ...
- xenbrX has an active address, which is used by dom0 to communicate with outside.
Xen Networking with vlan
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| eth0 eth1 |
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | eth0.100 | | eth1.200 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- With this configuration, DomUs are completely unaware of the fact that they are utilizing a VLAN, all the work is done within the bridges in Dom0.
- Dom0 is aware of the traffic within the VLAN, because it has an active address on the xenbrX interfaces. To prevent it, don't give the xenbrX an active address, but configure a extra interface for management.
- There are two things may need to be configured:
- If your ethernet card does not natively support VLAN tags, you will have to set the maximum MTU to 1496 to make room for the tag. With command:
# ifconfig eth0 mtu 1496
- With the DomUs bridged to VLAN interfaces, some optimizations need to be disabled or tcp and udp connections will fail. This is done by disabling transmit checksum offloading:
# ethtool -K eth0 tx off
- Need further test in the production environment.
Xen Networking with bonding
PRT0 PRT1 PRT2 PRT3
| | | |
+--------------+---+---------------------------+---+--------------+
| | | | | |
| eth0 eth1 eth2 eth3 |
| | | | | |
| +-+-+ +-+-+ |
| | | |
| +--------------+--------------+ +--------------+--------------+ |
| | | | | | | |
| | bond0 | | bond1 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- Need some more destructive testing.
Xen Networking with vlan on bonding
PRT0 PRT1 PRT2 PRT3
| | | |
+--------------+---+---------------------------+---+--------------+
| | | | | |
| eth0 eth1 eth2 eth3 |
| | | | | |
| +-+-+ +-+-+ |
| | | |
| bond0 bond1 |
| | | |
| +--------------+--------------+ +--------------+--------------+ |
| | | | | | | |
| | bond0.100 | | bond1.200 | |
| | | | | |
| | xenbr0 vif1.0 vif1.1 | | vif2.0 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ftp | | www ftp ssh dns | | www ftp ssh dns | | ssh dns |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+
Notes:
- The connections at the top are switch ports - probably on 2 switches with an ISL
- bond0 has eth0 and eth1 ; bond1 has eth2 and eth3
- In the VMs eth0 maps to bond0.100 and eth1 maps to bond1.200
- Protocols suggest a service VLAN (100) and a mgmt VLAN (200)
Collection of Examples
this is a collection of configurations that can help you to understand how it is working in detail and rapidly deploy one of them.
if you want to add a configuration to this list, please look at http://wiki.xensource.com/xenwiki/ under "Join Wiki as a Contributor"
