Xen FAQ Networking

Needs Review Some of the issues described on this page have been resolved. This page could do with a check and clean-up!

General Networking Questions

Can I share vdi and vbds devices with several VMs?

I want to ask how to mount one storage device to 2 guests? When I try to create vm handle everything is fine I can create one vdi and vbds for every guest. When I start first machine everything is ok, but when I try to start second one it says that

 ERROR: 2 INTERNAL_ERROR Device 2051 (vbd) could not be connected.
 Device /dev/mapper/test_vg-test64_454 is mounted in a guest domain,
 and so cannot be mounted now.

Ideas how I can share one device to two or more virtual machines? I don't want network solution like nfs,iscsi ... etc but instead use ocfs2. How I can set mode to w! ?

Answer: First of all, you DO realize that sharing a block device without some kind of cluster file system could lead to data corruption? If you want to share it anyway, you can try changing the mode to "r"(for read only) or "w!" (to force read-write multiple mount). in your domU.cfg:

'phy:/dev/data/bla-disk,sda1,w' => 'phy:/dev/data/bla-disk,sda1,r'

How do I monitor Networking Activity?

I'm looking for a way to monitor network activities of processes in Guest OS. I want to get a list of Guest OS processes that open TCP connections to other machines (like "lsof" command).

Answer: If you're thinking about doing on from dom0, that's not possible. You need something that runs on domU for that, possibly by using snmpd and extending it to run "netstat -anp --tcp". Other host (including dom0) can then collect the information using snmp.

Also, have a look at Versiera, it provides what you are looking for including, user IDs, inbound/outbound communications, IPv4, IPV6, etc. There are many more capabilities. Versiera is not open-source, but the Internet self-manage service is free.

How do I gather CPU, IO, and network stats on the hardware?

I am attempting to gather stats on usage of the metal, by which I mean the physical host's hardware. I would like to know the CPU, IO, and network stats for the hardware.

Answer: All DomU's IO passes through Dom0. there you can measure all you want. for disk IO, if you use phy: devices, you can use iostat to see the usage of each device. if you use file-based backends, it would be easier to check the userspace daemons. Maybe iotop would help. In any case, if aggregate usage is all you need, just measure the disk usage seen at Dom0 for network, if you can measure at peth0, that would give you the aggregate usage. if you need stats for each DomU, check the respective tun devices.

General Networking Requirements

I have some trouble finding the best solution to my networking requirements.

I want to have the following things:

• dom0 : have 2 physical networks devices * 1 eth with public IP (static) * 1 eth with private IP (static)
• domU * 1 eth with private IP
• I also want openVPN solution to let people outside the private network have access to it.
• A DHCP server is required so that domU get their IP from it.

I have debian lenny and xen 3.2 installed and working. Actually openvpn and dhcp are on the dom0. All is fine *except* that domU don't have access to internet (this is my main problem). My current config use network-bridge netdev=eth1 (eth1 have a static private ip).

It is perhaps better to have dhcp and openvpn server in a domU, feel free to suggest what you think is the best choice (and the config that go with it) :)

Answer: When designing such setups, with bridge networking, I often find it easier to think of dom0 as a switch or router, and domU like any other physical server on your network. In your setup you're making dom0 act as router/firewall. Your problem is that probably you haven't setup ip forwarding and NAT on dom0 to allow domU internet access. Note that (if you want) you could also have dom0 act like a switch. In that scenario you'd need another domU, with two network interfaces connected to both dom0's eth0 and eth1, acting as router/firewall.

Networking Issues

What is vif or xenbr0?

You should read XenNetworking http://wiki.xen.org/wiki/XenNetworking

Why can't I ssh into or ping a newly created domain?

In the default configuration we rely on the Linux bridge-utils in domain 0 to set up virtual networking. After you've created a new domain (e.g., domain 1) you should be able to run ifconfig in domain 0 and see an interface with a name like vif1.0; you should also be able to check that bridging is working by typing brctl show xen-br0. Finally, you can check the IP confiuration in the new domain by logging into it via the console (xm console) and running standard tools such as ifconfig and route.

Xen and Shorewall

There is a document about configuring Shorewall in Dom0 at http://www.shorewall.net/Xen.html

http://www1.shorewall.net/XenMyWay.html can be useful also.

Bridging

Which Mechanism is used by Xen bridging to handle packets coming from various VMs to forward them to their destination

Nothing. Xen by itself does not handle bridge. dom0 OS does that. On Linux dom0 : http://www.linuxfoundation.org/en/Net:Bridge On opensolaris dom0: http://opensolaris.org/os/project/crossbow/

IP Determination

I want to know the IP of a running VM in XEN. Is there any way to have this without login to that VM?

Yes. First, you need to find the MAC address of the domU. This can be done by running:

xl network-list <domU name>

which should produce an output similar to:

Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path 0 0 00:16:3E:F7:D6:E7 0 4
6 16238/16237 /local/domain/0/backend/vif/163/0

The domU's MAC address is 00:16:3E:F7:D6:E7

You now need to snoop the bridge for domU's MAC. For example:

# tcpdump -n -i eth0 ether src 00:16:3e:f7:d6:e7 tcpdump: verbose output suppressed, use
-v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15:54:56.419482 IP 10.0.0.10 > 10.0.0.1: ICMP echo reply, id 5443, seq 1, length 64 15:54:57.422349
IP 10.0.0.10 > 10.0.0.1: ICMP echo reply, id 5443, seq 2, length 64

Then you know that domU has IP address 10.0.0.10.

NAT

I managed to configure NAT on dom0 but this does not work properly. Outgoing traffic from domU is seen with the original domU ip address instead of the dom0 ip address and the requests can't get back to the domU.

I figured out MASQUERADING was not set.

The following rule needs to be set:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

SSL/VPN

Why can't I use openvpn with a xen guest? I can't load the tun module

From openvpn perspective, the requirements for openvpn on xen domU is the same as openvpn on native Linux. If you can't load the tun module, then you need to get a kernel that supports it.

The easiest way is to use a distro that supports it. For example, I'm using RHEL/Centos 5.3 domU, loaded with pygrub, and they can run openvpn just fine.

Another alternative is compile your own kernel with tun/tap support.

Can I share one storage mount between multiple guests?

This is possible, however it can lead to corruption if the filesystem doesn't support a cluster file system. If you want to share it anyway, you can try changing the mode to "r" (for read only) or "w!" (to force read-write multiple mount). In your domU.cfg:

 
'phy:/dev/data/bla-disk,sda1,w' => 'phy:/dev/data/bla-disk,sda1,r'

I'm looking for a way to monitor network activities of processes in Guest OS. I want to get a list of Guest OS processes that open TCP connections to other machines (like "lsof" command).

If you're thinking about doing on from dom0, that's not possible. You need something that runs on domU for that, possibly by using snmpd and extending it to run "netstat -anp --tcp". Other host (including dom0) can then collect the information using snmp.

Also, have a look at Versiera, it provides what you are looking for including, user IDs, inbound/outbound communications, IPv4, IPV6, etc. There are many more capabilities. Versiera is not open-source, but the Internet self-manage service is free.

How do I view the resources used by guest VMs?

All of DomU's I/O passes through Dom0. Therefore, you can measure resources used by VMs here.

• For disk I/O:
  • phy: devices, can be measured by running iostat to see the usage of each device.
  • File-based backends, are most-easily checked using the userspace daemons. Maybe iotop would help. In any case, if aggregate usage is all you need, just measure the disk usage seen at Dom0.
• For network:
  • Aggregate usage of peth0 can be found by running `ifconfig' on dom0.
  • If you need stats for each DomU, check the respective tun devices by running `ifconfig' in dom0.

Which parts of my network infrastructure should I run in dom0, and which should I run in domU?

Network infrastructure, such as DHCP, DNS and OpenVPN servers, should all run in domUs. Installing such services in dom0 is dangerous, as if the services are compromised, an attacker could access other running VMs.

As a general rule, a domU should be used to perform a small, independent task. Therefore if the OS crashes, or is compromised, the effect is constrained to a small part of your setup. This makes it easier to find the problem, and then fix.

How do I configure Xen guests to work with my VLANs?

Have a trunk port in your dom0 and create bridges for each VLAN for Xen. If you have the VLAN trunk set up, you can create bridges as follows. For this example, my trunk interface is on eth0 and the vlan I am adding is 2.

# vconfig add eth0 2 # brctl addbr xenbr2 # brctl addif xenbr2 eth0.2 # ifconfig eth0.2 up # ifconfig xenbr2 up

Now add the following to your domU configuration file:

vif=['bridge=xenbr2']

And the domU will be on VLAN 2. To access the trunk port directly inside the guest, pass through the network card to get VLAN access. You can also script this and give it a space separated list of VLANs and loop it through.