my web 0.2 website

I have been fairly consistently telling people a lie for the last ten years - and that is that Round Robin DNS can not be used for high availability. Its a view I have held pretty strongly, but two people have shown me techniques in the last week that have made me change my mind.

First, a note on what Round Robin DNS is. Hopefully you know what dns is! When a resource record is answered in a round robin fashion, a single resource record has several answers. When high-availability is build into a protocol (e.g. DNS itself - retries) RRDNS works pretty well. dig . @a.root-servers.net if you want a reference service which uses this principal.

RRDNS is also used to share load between nodes in a cluster where the protocol is not as forgiving, with unpredictable results. If I serve my website via two nodes, so the dns looks like:

www IN A 10.1.1.2
www IN A 10.1.1.3

then my website is not at all highly available. Pretty far from it, in fact - if either the nodes 10.1.1.2 or 10.1.1.3 fail, then I have suboptimal results, when in a highly-available configuration, I only want to suffer a performance degredation or outage if all of my nodes fail.Two colleagues showed me two different systems earlier this week that solve this fundamental issue with traditional RRDNS implementations - and that is handling an outage. Both of them use the premise that you never use the box’s own, unique (e.g. management) IP address in the RRDNS Resource Record, instead you use ‘virtual’ or aliases IP addresses that are free to roam around your server cluster. Consider my zone with this change:

www1 IN A 10.1.1.2
www2 IN A 10.1.1.3
www IN A 10.1.1.201
www IN A 10.1.1.202

Now I can address both servers in this small cluster with their own address, and also point customers to an address which can be served by any of those machines. If both web servers are up, they each have one of the additional addresses aliased. If 10.1.1.2 goes down, the computer 10.1.1.3 can adopt both 10.1.1.201 and 10.1.1.202 as aliased interfaces.

I can’t take the credit for finding either of the technologies I mention below, that can manage the automated failover of aliased interfaces, but I have seen both work this week.

Firstly, Wackamole which is brilliant if you are already using ‘Spread’. Nodes on a Spread ring send health messages to each other. If a node goes silent, or sends a communication that it is failing (deliberately stopping), another server in the cluster adopts that server’s aliased address. You can also configure a mechanism that notifies other devices that their arp cache entry for the vip is staled.

The second is good-old VRRP, which I had only ever considered as a redundant first-hop protocol, but actually can be run on a Linux server in order to provide these Virtual-interface floating features. This system does not rely on the Spread messaging bus, but in order to float several VIPs in a cluster, one vrrpd instance, and therefore group, must be configured for each vip. For my simple cluster of two servers above, I would need to run vrrpd twice on each server, to balance both vips as such :

/usr/local/sbin/vrrpd -Ni eth0 -v 48 -p 120 -g 3 -t 10.1.1.201 -S /var/run/vrrpd_48.state
/usr/local/sbin/vrrpd -Ni eth0 -v 49 -p 100 -g 3 -t 10.1.1.202 -S /var/run/vrrpd_49.state

Careful planning needs to be done in order to use RRDNS to share load. I have deliberately refrained from describing this technique as Load Balancing, because at best it is load sharing - you can’t balance traffic, only politely suggest to visitors that that might like to share traffic across your cluster. Additionally, you need to do some capacity thinking - if you have two, or even three nodes in your cluster, if one of the servers fails, one of the servers which is left is dealing with a doubling of the traffic that it was doing before. There is really no elegant way to share three vips across two servers.

Its cheap though. And as I have learned, you can force it to work with a bit of ingenuity.