my web 0.2 website

IP Drought begins today in Asia-Pacific

That’s it – the Asia Pacific region is the first to run out of IPv4 addresses.

This happened following an assignment of around half a million addresses to support the users at the Chinanet Fujian Province Network.

The pool of available addresses to the region including some of the world’s largest populations, such as China, India, Indonesia, and some of the world’s largest economies, such as Japan and Australia, has depleted to such low levels, that the registry responsible for distribution of these addresses will now ration them, such that any ISP requesting space will be given a single block of 1,024 addresses, on a single occasion only.

This is enough space to allow the ISP only to host NAT or ipv4 to ipv6 translation technologies.  It is not enough to address a large content infrastructure, hosting environment, or internet access customer-base.

The rules of the game have today changed for 50% of the world’s population, and they will change in Europe too in a few short months too.  If you do not have an IPv6 plan, then this is your new significant business risk – how will users with v6 only connections reach your content?  And if this is through a translation mechanism, how will you ensure quality, or that your end-to-end protocols (like voice, video, etc.) will work ?

Get in touch to continue the conversation!

Can you fill all of the Great Lakes with M&M sized /64s?

Posted to my blog at the request of RobL !

On Nanog, Owen DeLong and Larry Sheldon were discussing the relative size of the IPv6 address space:

>> 64 bits is enough networks that if each network was an almond M&M,
>> you would be able to fill all of the great lakes with M&Ms before you
>> ran out of /64s.
> Did somebody once say something like that about Class C addresses?

Well, this seemed like a challenge for Maths, and the answer is:

No. There are only 2,097,152 Class C networks.

Assuming all M&Ms are spheroids of uniform oblate nature, radius major axis=6mm, minor axis=3mm. Volume is (4/3)Pi (Major²) Minor

They will be poured into a great lake of your choice, and we will assume random close packing (agitation mechanisms are probably best discussed off-list) and a (generous, but this Wikipedia article insists) void fraction of 32%.

Volume of m&m = 0.452cm³, occupies 0.665cm³.

Lake Erie is 484km³ – See: http://www.epa.gov/glnpo/factsheet.html

1 km³ = 1,000,000,000,000,000 cm³

484,000,000,000,000,000 * 0.665 = 321,860,000,000,000,000 m&ms needed to
fill this lake.

There are 4,294,967,296 /64s in my own /32 allocation. If we only ever use 2000::/3 on the internet, I make that 2,305,843,009,213,693,952 /64s. This is enough to fill over seven Lake Eries. The total amount
of ipv6 address space is exponentially larger still – I have just looked at 2000::/3 in these maths.

THE IPv6 ADDRESS SPACE IS VERY, VERY, VERY BIG.

Can we please now just go ahead and roll out some ipv6 services?

2010 will be a bad year for ipv4

We are now at the end of January, but IPv4, the Internet’s core addressing protocol still has a nasty hangover, and all signs are pointing to 2010 being a bad year for the protocol.

Since January 1st, a few key milestones have passed, indicating how urgent the IPv4 rundown problem has become. Firms that rely on internet connectivity must take urgent action in light of the events:

• The allocation last week of two further /8s (blocks of IPv4 addresses with the same number before the first dot) to APNIC mean that for the first time, less than just ten percent of the IPv4 unallocated pool is available to be assigned.  At current utilisation rates, this pool will be exhausted in only 600 days.  Of course, the internet could stop growing, but all signs point away from this…
• The allocation of 1.0.0.0/8 is the assignment of the first really ‘dirty’ block of addresses, signalling that we really are in the run-down period.  Bad network design decisions in the past have meant that networks have ‘borrowed’ the use of addresses starting 1. for ‘internal use only’ or special applications on their network.  This means that organisations assigned address space starting ’1′ may well have partial connectivity even though they are rightfully assigned the space.  Examples are the braindead hotspot operators who take addresses like 1.1.1.1 to trigger hotspot logout, but a handful of examples appear across this address range.
• RIPE NCC, the organisation who assign addresses to networks in and around Europe have this month implemented their ‘run down’ policy which will mean that organisations requesting space will only be able to cater for their growth requirements for a very short amount of time.  This is to evenly spread the inevitable misery across the ISP community.

RIPE members should thoroughly audit their address space so that they can ensure that their records are accurate, because RIPE are more likely to ensure that address space is assigned to your end users in line with the community’s policies.  ISPs and services providers who need help can contact me for further information or specific assistance.

Organisations who rely on internet connectivity for their products should ensure their providers have an IPv6 migration plan in place.  Otherwise end-to-end connectivity for your home or office is unlikely to be something you can enjoy looking beyond the runout period.  Companies hosting network services, for example a website, should enquire what their host’s IPv6 plans are, and start to enable their services via v6.

There is real traction to ensure v6 support appears in both the hardware and services you need to connect to the internet.  It is easier today than before to find help making your services available via v6.  The alternatives – patchy connectivity via nested stacks of ipv4 islands, or no more end-to-end connectivity (so that your internet service is a walled garden), have much worse consequencies than learning to roll v6.

Engineers know the facts by now and have no excuse.  For more information, see the RIPE NCC’s information site, ipv6actnow.

DNSSEC and SSL certificates

Dr. Jörg Schweiger of the German domain name registry DENIC posed an interesting question at this morning’s first DENOG meeting, in Frankfurt.

Would domain name users who are concerned about the accuracy of data served pay extra for the ability to sign their DNS zone ?  A handful of people in the room raised their hand in agreement, but the overwhelming majority of operators did not.

His argument was that this compared well with SSL certification authorities who sell certificates that suggest that visitors to a website are interacting with a validated entity, and the technology guarantees privacy between the visitor and the website.  It’s this technology which makes buying and selling online safe.

However, I think that DNSSEC has different aims altogether – simply to guarantee that DNS data is not changed en-route between the authoratative server, through the caches, all the way to users.  Therefore there are significant attack mitigation reasons to deploy DNSSEC, so I hope that operators will begin trials (we are doing so), and that the pace of trials will quicken as the root zone will be signed this year.

If DNSSEC is deployed as designed, then temporary and brief mistakes will not be imported into DNS caches, users will not fall foul to tampered data in caches, and we all receive an authenticated/secure channel for distributing DNS data inside an organisation.

The argument that Dr. Schweiger used is that DNSSEC adds an operational and technical burden to registries (extra communication with registrars, more complex software, additional CPU and bandwidth requirements).

I hope that my colleagues in other organisations agree that there are significant infrastructure advantages to freely allowing DNSSEC to grow, and that Moore’s Law, automation, and the fact that DNS registries normally find it simple to peer widely with ISP networks will offset the needs to consider the commercial signing model.

innodb_data_file_path bug with long line limits

I have a MySQL server which was starting to scrunch its data more and more slowly.  Some analysis led me to blame an autoextending innodb file which had grown somewhat unkept to several GB.  I wish the autoextend behaviour could be configured to grow more files rather than grow one file, but that’s another rant.  I decided to clear it up which is a simple process :

• Stop the inflow of new data
• mysqldump the database to a .sql file
• Bin the junk innodb data files
• Edit my.cnf to create files of a spec that I wanted, start mysql
• Import the data.

I decided to create 500 data files of 100MB.  When MySQL started, it ignored the config I had set in my.cnf and loaded the innodb defaults – 5MB log files and an ibdata file of 10MB which could autoextend.

It turns out that if the innodb_data_file_path line is too long then the innodb section is ignored and the defaults will prevail.  I changed the plan to create 200 files of 250MB, which made for much smaller config, and the innodb settings loaded just fine.

If you want a perl one liner to generate the innodb_data_file_path, I used:

perl -e ‘$i=1; do { $padded = sprintf(“%03d”, $i); print “ibdata$padded:250M;”; $i++; } until ($i eq 200)’;

Preventing Mailman annoyances

Inspired by TheHodge’s “After you install WordPress” article, I made a note of the things I did to configure a Mailman mailing list, after creating it.  Much of this is to make the look-and-feel replicate how I used to run Majordomo lists.

Firstly, I like the Bounce handling and web-interface to Mailman, so this is why I don’t just run Majordomo for lists any more.  Its worth pointing this out, in case you wonder why I still use tool B, even though I have to do lots of work to make it work like tool A !

After running newlist, I recommend the following configuration changes (defaults which are changed assume you are running the Debian packaged Mailman) :

• General Options – make the administrators email address a role account that you will not subscribe to the mailing list. This is basically so that if you bounce an administration message from Mailman, due to your spam filters or an error, Mailman wont decide to unsubscribe you from the mailing list!  I have had this happen to me before when I used to hand-check spam directed at uknot.
• Decapitalise public name of the list – to make it look neater, and more like the output of the ‘lists’ command in majordomo.  Don’t forget to decapitalise the subject line tag if you do this.  I also tend to make the subject line tag very small so that on little displays, it’s still possible to scan a folder and read threads of interest on subject alone.
• Disable monthly reminders – they are really annoying to your subscribers, and Debian’s default position is disabled, but some implementations do not disable subscription reminders.
• My users get confused by the Filter out duplicate messages to list members option.  When a list subscriber is cc:d to a list post, users tend to expect to see a copy of the mail in their inbox, and their mailing list archive.  I turn the filter off so that this happens.
• I tend to enable the Should administrator get notices of subscribes and unsubscribes option, so that I can track whether promoting a list in a certain place has worked!
• In “Non Digest options”, if I am migrating for a Majordomo list, I empty the box for message footer, and also tend to remove it when its a geek mailing list, as to most high volume mail readers, its obvious when an email has been posted to a mailing list, because its filtered into the correct mailbox !  For low volume lists that are intended for low volume mail readers, the footer might be useful.
• Check the reply to option, so that mailing lists that are intended to promote on-list discussion have a header that directs conversation back to the list, and mailing lists that will yield a high proportion of off-list mail do not have this header.
• Be slightly annoyed with me that the only English option in ‘Language Options’ is English (USA) when there is no English (UK).

Happy list-administration!

And you may find the following lists interesting to your work :

• mailop, for those who work in the field of mail systems administration.
• experts, for those who work in expert e-commerce roles.
• uknof, for those who work in network engineering roles, or systems/ISP environments.

Asterisk 1.4.22 Agent call acknowledgement bug

The behaviour of Asterisk has been altered since 1.4.21, possibly in error, with regard to answering calls from call queues.

There is a feature that requires agents to press # when they are ready to speak to a caller.  Since we forward calls to agents via their mobiles, rather than auto-answer calls in a desk environment, we disabled that feature with ackcall=no in agent.conf.

After upgrading to 1.4.22 we see this configuration is nolonger honoured.  Diffing chan_agent.c between version 1.4.21 and 22 shows a new section of code saying (in English) that ‘if there is no per-channel override specified in the dialplan, default the configured variable’ (line 2048).  I looked at where the default was read from the config file and it looks like a lot of different chunks of chan_agent want to set the ackcall default!

The bug shows up in the asterisk console as :

– Agent/xxx is ringing
– SIP/voip-out-081e5a88 is making progress passing it to Local/447xxxxxxxxx@uk_all-a667,2
– SIP/voip-out-081e5a88 answered Local/447xxxxxxxxx@uk_all-a667,2
– Local/447xxxxxxxxx@uk_all-a667,1 answered, waiting for ‘#’ to acknowledge

The workaround is that the only safe place to set the default ackcall behaviour is for each channel in the dialplan.  If you want to disable the ‘waiting for ‘#’ to acknowledge’ behaviour, configure your dialplan as such :

exten => 1701,1,Answer()
exten => 1701,n,Set(AGENTACKCALL=no)
exten => 1701,n,Queue(noc|r|||40)
exten => 1701,n,Voicemail(xxxxx)
exten => 1701,n,Hangup

2011 – An addressing odyssey. Preparing enterprise for IPv6.

Yesterday I gave a talk to Sheffield GeekUp on preparing enterprises for IPv6 [download].  The premise of the talk was :

• IPv4 addresses are scarse, and at current consumption rates, the IANA pool of free v4 addresses will be gone at the start of 2011.
• This starts a “Post IPv4 world” where the IPv4 internet continues to function as before (certainly initially), but obtaining new addresses becomes harder and expensive.  This inhibits expansion of existing firms, and new entrants to the market.
• Address trading is likely to lead to a larger routing table, meaning that failure-recovery times increase, and the risk of blackholes on the internet increases.
• Large broadband providers may not have enough v4 addresses to give one address per customer.  This means protocol translation techniques need to be used, which break the end to end model.  We rely on the end to end model when innovating new services on the internet.
• If services and consumers gradually roll v4 and v6 (dual stack), the negative impact of markets for addresses, routing problems, and translation can be mitigated.
• Service providers are enabling v6 in the core.  Enterprises need to move next in order to get the world v6 ready.

The advice I gave was :

• Today’s market leaders are already learning v6 lessons in their labs, (e.g. ipv6.google.com).  They are doing this to help them retain market leadership.  If you want to retain your market position, start labbing your applications and service provision with v6.
• Write a policy stating all new purchases of infrastructure and services need to be from providers with v6 support, or a well defined v6 road map.  In other words, make v6 a “life cycle upgrade”.
• Share information, and learn information from your industry peers.
• I also listed some advice to developers with regard to v4 and v6 differences.
• I then delivered a very quick primer to those who have not seen v6 deployed before.

My hope is that this talk is improved upon and delivered internationally to enterprises.