Xen in WiFi networks

Xen in WiFi network

Successfully running Xen on a machine with a single WiFi card can be tricky. In my situation the network in dom0 worked fine but in domU I could only see incoming packets. Outgoing ones were sent out to xen-br0 but never transmitted over the wireless network.

Sample network configuration

There are two physical hosts on the network - Router and a Xen host running dom0 and domU:

Host	IP address
Router	192.168.157.20
Xen dom0	192.168.157.21
Xen domU	192.168.157.41

Xen dom0 network interface has the real MAC address of the WiFi card, Xen domU MAC address is set in /etc/xen/domU config:

~# cat /etc/xen/domU
[...]
nics = 1
vif = ['eth0=00:60:00:00:00:01']

Problem and solution

After some research it turned out that the culprit blocking outgoing traffic wasn't Xen but the wireless card which refused to send out packets with a MAC address different from its own. Since Xen deploys ethernet bridge to provide network for domU kernels the outgoing packets have a virtual source MAC address different from the real one of the WiFi card. You don't need Xen to observe this problem - simple ethernet bridge constructed with brctl(8) between wired and wireless segments will not work with many wifi cards either. See section It doesn't work with my Wireless card at Linux Bridging FAQ.

The solution obviously is to change the source MAC address to the real one on all outgoing packets. At first I tried to set it in vif=['mac=...'] config option for domU host, but it didn't work. Better solution turned out to be deployment of ebtables - netfilter infrastructure similar to iptables but working with MAC addresses instead of IP addresses. The simplest ebfilter rule to make it work is:

ebtables -t nat -A POSTROUTING -s 00:60:00:00:00:01 \
              -j snat --to-source 00:60:aa:bb:cc:dd

This rule rewrites the source MAC address of all packets from 00:60:00:00:00:01 to 00:60:aa:bb:cc:dd. You may want to add some more checks, e.g. for source interface name or alike. This is the bare minimum. I have executed this command in dom0 but in theory it could work in domU as well.

Details

Surprisingly enough there doesn't have to be a corresponding reversed dnat rule. Ping from outside to domU triggers this packet sequence:

01:56:03.972501 00:02:11:22:33:44 > ff:ff:ff:ff:ff:ff,
	ethertype ARP (0x0806), length 42:
		arp who-has 192.168.157.41 tell 192.168.157.20

01:56:03.975205 00:60:aa:bb:cc:dd > 00:02:11:22:33:44,
	ethertype ARP (0x0806), length 60:
		arp reply 192.168.157.41 is-at 00:60:00:00:00:01

Interesting point is the ARP reply. Although it comes from WiFi real MAC address, it carries the domU virtual MAC inside in the packet. Subsequently my workstation created an entry in its ARP table with the virtual(!) address of domU and started sending all further communication to that MAC:

01:56:03.975231 00:02:11:22:33:44 > 00:60:00:00:00:01,
	ethertype IPv4 (0x0800), length 98:
		IP 192.168.157.20 > 192.168.157.41:
			icmp 64: echo request seq 1

01:56:03.982348 00:60:aa:bb:cc:dd > 00:02:11:22:33:44,
	ethertype IPv4 (0x0800), length 98:
		IP 192.168.157.41 > 192.168.157.20:
			icmp 64: echo reply seq 1

Fortunately the Linux kernel on Router only updates its ARP table from the body of ARP replies and not from packet headers, which I believe is a correct behaviour. However I'm not sure that all operating systems do like this.

Notes

• To get the MAC address of your WiFi card use command ip link show:

ip link show
# ip link show
[...]
2: eth0: <BROADCAST,MULTICAST,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:60:aa:bb:cc:dd brd ff:ff:ff:ff:ff:ff

• The chipset on my WiFi card that caused these troubles is Intersil Corporation Prism 2.5 Wavelan chipset with firmware 1.3.6. Other chipsets may work just fine without these kludges.
• Xen version in use here is 2.0.6 and both dom0 and domU are running SuSE Linux Professional 9.2 with kernel 2.6.11.4, but it is fairly irrelevant as the problem is in hardware not in software.