Since January 1st, a few key milestones have passed, indicating how urgent the IPv4 rundown problem has become. Firms that rely on internet connectivity must take urgent action in light of the events:

• The allocation last week of two further /8s (blocks of IPv4 addresses with the same number before the first dot) to APNIC mean that for the first time, less than just ten percent of the IPv4 unallocated pool is available to be assigned.  At current utilisation rates, this pool will be exhausted in only 600 days.  Of course, the internet could stop growing, but all signs point away from this…
• The allocation of 1.0.0.0/8 is the assignment of the first really ‘dirty’ block of addresses, signalling that we really are in the run-down period.  Bad network design decisions in the past have meant that networks have ‘borrowed’ the use of addresses starting 1. for ‘internal use only’ or special applications on their network.  This means that organisations assigned address space starting ’1′ may well have partial connectivity even though they are rightfully assigned the space.  Examples are the braindead hotspot operators who take addresses like 1.1.1.1 to trigger hotspot logout, but a handful of examples appear across this address range.
• RIPE NCC, the organisation who assign addresses to networks in and around Europe have this month implemented their ‘run down’ policy which will mean that organisations requesting space will only be able to cater for their growth requirements for a very short amount of time.  This is to evenly spread the inevitable misery across the ISP community.

RIPE members should thoroughly audit their address space so that they can ensure that their records are accurate, because RIPE are more likely to ensure that address space is assigned to your end users in line with the community’s policies.  ISPs and services providers who need help can contact me for further information or specific assistance.

Organisations who rely on internet connectivity for their products should ensure their providers have an IPv6 migration plan in place.  Otherwise end-to-end connectivity for your home or office is unlikely to be something you can enjoy looking beyond the runout period.  Companies hosting network services, for example a website, should enquire what their host’s IPv6 plans are, and start to enable their services via v6.

There is real traction to ensure v6 support appears in both the hardware and services you need to connect to the internet.  It is easier today than before to find help making your services available via v6.  The alternatives – patchy connectivity via nested stacks of ipv4 islands, or no more end-to-end connectivity (so that your internet service is a walled garden), have much worse consequencies than learning to roll v6.

Engineers know the facts by now and have no excuse.  For more information, see the RIPE NCC’s information site, ipv6actnow.

• Stop the inflow of new data
• mysqldump the database to a .sql file
• Bin the junk innodb data files
• Edit my.cnf to create files of a spec that I wanted, start mysql
• Import the data.

I decided to create 500 data files of 100MB.  When MySQL started, it ignored the config I had set in my.cnf and loaded the innodb defaults – 5MB log files and an ibdata file of 10MB which could autoextend.

It turns out that if the innodb_data_file_path line is too long then the innodb section is ignored and the defaults will prevail.  I changed the plan to create 200 files of 250MB, which made for much smaller config, and the innodb settings loaded just fine.

If you want a perl one liner to generate the innodb_data_file_path, I used:

perl -e ‘$i=1; do { $padded = sprintf(“%03d”, $i); print “ibdata$padded:250M;”; $i++; } until ($i eq 200)’;

trials:/usr/src/openssh-5.1p1# ssh hextreeme -l netadmin

Keyboard-interactive authentication

Enter password for netadmin:

channel 0: open failed: resource shortage: Channel open failed

The bug is described in Debian bug 495917, and it also prevents connection to some NetScreen firewalls.  I have this bug with Debian 4 (openssh-client 4.3p2-9etch3) and Debian 5 (openssh-client 1:5.1p1-5).

If anyone else is experiencing the same bug and needs a quick fix, then you can download my Debian packages which replace openssh-client.  You of course need to hold the packages if you don’t want them overwriting by a security fix.

• Debian 4 openssh-client_4.3p2-9etch3_i386.deb
• Debian 5 openssh-client_5.1p1-5_i386.deb

By using this software you agree to hassle both the debian-ssh team and extreme to sort their stuff out!

How can I put this..? This is quite a novel way of doing it.  I think there’s a reason that more virtualisation systems don’t work in this way.  VNC is not great, but I am sure there is a reason that I can’t use a dummy serial port instead.  I’d have preferred RDP, but perhaps there’s a reason I can’t use that too.

I normally use Chicken of the VNC as a mac osx client, because it has a funny name, and has always worked.  However, it crashes and burns (see screen shot) when trying to install Debian on a KVM guest.  Hopefully I can save someone else an evenings’s worth of trying every other mac vnc client, and offer the fix.  Just use VNCViewer.  I tried this after half a dozen others which all failed in a similar way to Chicken.

Any comments on why RDP or Serial might not have been better welcome.

First, a note on what Round Robin DNS is. Hopefully you know what dns is! When a resource record is answered in a round robin fashion, a single resource record has several answers. When high-availability is build into a protocol (e.g. DNS itself – retries) RRDNS works pretty well. dig . @a.root-servers.net if you want a reference service which uses this principal.

RRDNS is also used to share load between nodes in a cluster where the protocol is not as forgiving, with unpredictable results. If I serve my website via two nodes, so the dns looks like:

www IN A 10.1.1.2
www IN A 10.1.1.3

then my website is not at all highly available. Pretty far from it, in fact – if either the nodes 10.1.1.2 or 10.1.1.3 fail, then I have suboptimal results, when in a highly-available configuration, I only want to suffer a performance degredation or outage if all of my nodes fail.Two colleagues showed me two different systems earlier this week that solve this fundamental issue with traditional RRDNS implementations – and that is handling an outage. Both of them use the premise that you never use the box’s own, unique (e.g. management) IP address in the RRDNS Resource Record, instead you use ‘virtual’ or aliases IP addresses that are free to roam around your server cluster. Consider my zone with this change:

www1 IN A 10.1.1.2
www2 IN A 10.1.1.3
www IN A 10.1.1.201
www IN A 10.1.1.202

Now I can address both servers in this small cluster with their own address, and also point customers to an address which can be served by any of those machines. If both web servers are up, they each have one of the additional addresses aliased. If 10.1.1.2 goes down, the computer 10.1.1.3 can adopt both 10.1.1.201 and 10.1.1.202 as aliased interfaces.

I can’t take the credit for finding either of the technologies I mention below, that can manage the automated failover of aliased interfaces, but I have seen both work this week.

Firstly, Wackamole which is brilliant if you are already using ‘Spread’. Nodes on a Spread ring send health messages to each other. If a node goes silent, or sends a communication that it is failing (deliberately stopping), another server in the cluster adopts that server’s aliased address. You can also configure a mechanism that notifies other devices that their arp cache entry for the vip is staled.

The second is good-old VRRP, which I had only ever considered as a redundant first-hop protocol, but actually can be run on a Linux server in order to provide these Virtual-interface floating features. This system does not rely on the Spread messaging bus, but in order to float several VIPs in a cluster, one vrrpd instance, and therefore group, must be configured for each vip. For my simple cluster of two servers above, I would need to run vrrpd twice on each server, to balance both vips as such :

/usr/local/sbin/vrrpd -Ni eth0 -v 48 -p 120 -g 3 -t 10.1.1.201 -S /var/run/vrrpd_48.state
/usr/local/sbin/vrrpd -Ni eth0 -v 49 -p 100 -g 3 -t 10.1.1.202 -S /var/run/vrrpd_49.state

Careful planning needs to be done in order to use RRDNS to share load. I have deliberately refrained from describing this technique as Load Balancing, because at best it is load sharing – you can’t balance traffic, only politely suggest to visitors that that might like to share traffic across your cluster. Additionally, you need to do some capacity thinking – if you have two, or even three nodes in your cluster, if one of the servers fails, one of the servers which is left is dealing with a doubling of the traffic that it was doing before. There is really no elegant way to share three vips across two servers.

Its cheap though. And as I have learned, you can force it to work with a bit of ingenuity.

There are quite a lot of config examples in the notes which aren’t on the slides to support the talk.

Overview:

• Definition of Scaling & Problems of scaling
• You have to monitor things
• Scaling individual machines – Disk IO, Processors, Memory, Connectivity
• Multiple Servers – L7 proxying, L4 switching, CDNs
• Memcache
• Simple tuning

Its a 30 minute talk so its an overview of all of those things.

Download the slides here. 

Step in some newer projects. Ganglia was really interesting, and a colleague found it thanks to some presentation that Flickr demo’d. I really liked how easy it was to configured. Set the agent up on a bunch of PCs, run the web interface on one, and bang, graphs. Its that easy. We installed the agent on a couple of trial PCs and we had graphs. We then wrote some scripts to measure other metrics from custom applications. If we could write a script that produced a number, then we could graph that metric in ganglia, just by passing the number to the bundled ‘gmetric’ application. Brilliant. But what about if we can’t run an agent on the device that we want to monitor, such as a switch ? There has been talk on the ganglia developers list since 2004 about incorporating snmp support, but no real evidence of traction. So it wont work for me.

So let me offer a golden rule of performance monitoring. If you are going to write a performance monitoring suite, make sure it supports SNMP on day one. If you are writing a monitoring layer for your application, make sure it uses SNMP.

In steps Zabbix. The best of both worlds. Here, there’s an agent again, so if you want to monitor the health over time of a server, you configure the agent and send back figures to a monitoring box. Figures appear. There’s also an snmp interface, so you point it at a router, tell it the community, and more figures appear.

No graphs yet, but thats because you configure them yourself, but its really easy. Want to aggregate all of the exit ports on your router – make a graph using those metrics ! If you can imagine it, you can graph it with Zabbix. Some of it is quite clunky, i.e. configuring the snmp community for each device is a bit slow, but the back end if just MySQL, so you can change the community for a device with an “update items set snmp_community =’xx’ where hostid=’yy’;” instead of using the clunky interface. Also, to measure interface stats, you must change the ifInOctets and ifOutOctets delta to ‘speed over time’ not just accept the counter value, otherwise your graphs will show nothing more than the port counting more data as time goes by.

I strongly recommend Zabbix to anyone who finds cacti arduous to configure.

I now have a stable enough kernel package for Debian users who may want to try it out. The kernel package uses the same defaults as etch, but there’s a version bump to 2.6.21, and of course I have turned on the ARA features!

Download the Debian Adaptive Readahead package and install with dpkg -i

Please post a comment if you try it out and have anything to report.