Vif-route

The vif-route script is a bit complicated at first but it has important uses:

The problem:

In many situations, one has a block of public IPs, but those IPs are not routed to the customer. Assuming the IP block is 1.2.3.0/24:

[ ISP router:1.2.3.254]                           [ dom0:1.2.3.1 ]                        [ domU:1.2.3.2,3,4..]
[ route to 1.2.3.0/24 ][router_eth0]<->[dom0_eth0][0dea.dbee.ff00]                        [0caf3.beba.d000     ]
[ via eth0            ]                           [              ][vif_domU]<->[domU_eth0][                   ]

In the above diagram, you are given a network of 1.2.3.0/24, and this network is accessible via an ethernet interface of the dom0 (dom0_eth0). The problem is letting other domUs on the dom0 use IPs within the same IP network. One solution for this is bridging, but it will often be impractical for applications where control or security is desired. All bridged interfaces function as a single IP interface as far as the kernel and netfilter (iptables) is concerned, thus limiting the amount of control (say, rate limiting, logging, IPS/IDS etc.)

When more control is desired, routing becomes a necessity. However as seen in the diagram, 1.2.3.0/24 is not routed to the dom0, rather it is routed to a network of which dom0 is also present. This means that packets going to the 1.2.3.0/24 network will not use dom0 as a hop, since the router (1.2.3.254) is the final hop. It expects all IPs within that network to be part of the same physical (layer 2) network.

The solution:

proxy-arp can allow a (somewhat hackish, but nevertheless necessary) solution for this. When host A wants to contact host B, it sends out an ARP request asking which MAC address has the IP of host B, when it receives a reply, it adds the MAC to its ARP table and starts communicating with host B. proxy-arp allows a router to respond to ARP requests on behalf of interfaces which are not directly connected to the physical network. In the example network shown in the diagram, dom0 will respond that MAC 0ded.dbee.ff00 has the IP of 1.2.3.2 even though this in fact belongs to 0caf3.beba.d000, assuming that dom0 has a route to 1.2.3.2 This will only work if the first route for 1.2.3.2 in dom0's routing table is via vif_domU, which introduces another issue.

Netmasks:

A netmask is not a holy number. When your ISP gives a netmask, they are describing the limits of your network in terms of net/host bits.You do not have to use the netmask assigned by your IP. The netmask is there simply to assist the kernel routing table in establishing directly connected routes. If you do not use the netmask assigned by your ISP, you will likely need to add a few more manual routes, as shown below.If dom0's IP was added via a command such as:

dom0# ip addr add 1.2.3.1/24 dev eth0
#or
dom0# ifconfig eth0 1.2.3.1 netmask 255.255.255.0

then:

dom0# ip route ls table all
broadcast 1.2.3.255 dev eth0  table local  proto kernel  scope link  src 1.2.3.1
...
1.2.3.0/24 dev eth0  proto kernel  scope link  src 1.2.3.1

What this means is that the kernel will treat the network of 1.2.3.0/24 as being directly connected to eth0 (all routes in the local table are directly connected routes). A directly connected route is consulted first in the routing table. Therefore, adding a static route of say

dom0# ip route add 1.2.3.2/32 dev vif_domU

will not work, because the local table is consulted first. You cannot change the local routing table (Well, you can, but it won't do what you expect it to.. better to just leave it alone).

The solution is to give your dom0 an IP with a /32 netmask. This way the kernel will not have any assumptions about what network you are physically attached to. A preliminary configuration for the above diagram is as follows:

First, set up routing.

#Clear the existing eth0 IP:
dom0# ip addr del 1.2.3.1/24 dev eth0
#Add our rectified address
dom0# ip addr add 1.2.3.4/32 dev eth0
#Add a route to our network first. The kernel will look up more specific
#routes in the same table before falling back to more general ones, therefore:
dom0# ip route add 1.2.3.0/24 dev eth0 src 1.2.3.1
#default route
dom0# ip route add default via 1.2.3.254
# Enable routing:
dom0# sysctl -w net.ipv4.ip_forward=1
# Enable proxy_arp for eth0
dom0# sysctl -w net.ipv4.conf.eth0.proxy_arp=1

We are still missing static routes for domU's IP. The vif-route script will take care of this; to use the vif-route script, either have (vif-script vif-bridge) in your xend-config.sxp configuration, or use script=vif-route in your vif [ ] config tuple. I will be assuming the latter.

vif = [ 'script=vif-route,ip=1.2.3.2/32 1.2.3.3/32 1.2.3.4/32,gatewaydev=eth0' ]

Note that gatewaydev is new, the old, and now deprecated, label for this was netdev.

There are a few parameters which you may not have seen before:

script: this is the script (located in /etc/xen/scripts/) which will be executed to manage the vif-backend of the domU;

ip: this is a whitespace-delimited list of IPs and their netmasks that should be routed to the vif. The relevant code from the script:

if [ "${ip}" ] ; then
    # If we've been given a list of IP addresses, then add routes from dom0 to
    # the guest using those addresses.
    for addr in ${ip} ; do
      ${cmdprefix} ip route ${ipcmd} ${addr} dev ${vif} src ${main_ip}
    done
fi

netdev: this is the interface which has an IP and which is in the network you wish the domUs to communicate with:

#from vif-route:
main_ip=$(dom0_ip)
#from vif-common (sourced by all vif-* scriptS)
dom0_ip()
{
  local nd=${netdev:-eth0}
  local result=$(ip_of "$nd")
  if [ -z "$result" ]
  then
      fatal
"$netdev is not up.  Bring it up or specify another interface with " \
"netdev=<if> as a parameter to $0."
  fi
  echo "$result"
}

Note, the ip= value does not set the IPs for the domU, you must do this within the domU itself. The ip= value is used only to establish a route from dom0 to that IP.