my web 0.2 website

Youtube pushed off the air

In between browsing Facebook and Youtube, the UK economy generates $1,930,000,000 of output a year. Thats $550,000 every two and a half hours. Well if today had been a work day, there’d have been one two and a half hour period where that was much higher. That’s because in a pique of routing excitement, Pakistan Telecom managed to hide Youtube from most of the internet for that length of time.

Pakistan Telecom and Youtube are likely to have no commercial relationship in place to carry Youtube traffic – particularly as around two hours ago, according to Yahoo News, the story broke that the Pakistan Government required ISPs operating in the country to block Youtube. Despite this, Pakistan Telecom were able to cause ISPs all over the world to send traffic that should be destined for Youtube to Pakistan instead.

This is because the protocol that determines how to find my network on the internet, is shaped by how “specific” the announcement of my network is. If I make an announcement of a network of 1,024 addresses, and someone else makes a second announcement of 256 addresses within a subset of my 1,024, then the network which announces the smaller subset win the traffic destined to those hosts. This is a feature – fully by design – of the BGP routing protocol. Almost every time a more specific block of addresses is announced, this is because the administrators of those networks intend for the routing to be different for a subset of a large number of addresses.

Sadly, there are accidents from time to time – another network can announce a subset of my addresses without my knowledge or permission, and they win the traffic that should have gone to me. This happened today – it seems that Pakistan Telecom decided to inject a fake route to their network containing Youtube’s webservers, and accidently then leaked that route to the networks they connect to.

Small networks and end sites can limit the chances that they will leak bad routes by explicitly listing the network addresses that they intend to send to their upstream or peered networks. Larger networks may find it harder to stop themselves propagating someone else’s mistake, because they may have a contract to carry forward any announcement that their customers make. Furthermore, the complexities of their own networks mean that an engineer working under pressure after announcements made by government ministers are more likely to make a typo error and do the wrong thing.

Richard Clayton presented a very interesting set of commentaries at the last LINX meeting. He commented that right now its very obvious indeed when someone hijacks some of my network space in this way, because all of my traffic disappears. Youtube were probably aware that something was very wrong within moments of the announcement. What if someone builds an infrastructure to steal my traffic – or at least some of my traffic – but after doing something with it, they send it back to me, it is much harder for me to spot that anything is wrong.

This is a significant risk to ecommerce infrastructures that competitors or e-pirates could seize upon opportunities to steal customer behaviour data. What if a wizard stole the network containing your web server, proxied your shop, but set up a fake checkout? How quickly would you spot?

Because this problem is inherent to the routing protocol, this is the obvious place to fix it. There are attempts to blend PKI with routing information, so that peers can verify the validity of your announcements. S/BGP (secure BGP) requires me to sign my announcements, and gives my peers a method to check in an impartial internet community database that my announcement is valid. It is the sort of technology that would have prevented Youtube from disappearing off the air today.

Life after email

The death of email has already been predicted on approximately 1,258,926 blogs, so I barely need to recount the chant that IM is already replacing regular conversation, social networking manages infrequent messaging between your peers and introduces you to new business partners, and that web forums are how the population now find out information.  Email is struggling to be the ubiquitous gold-standard for internet communication because of the amount of spam and malware that is distributed through the medium.
This doesn’t explain how email marketing will be replaced.  Email is an inexpensive and relatively simple and successful way of driving repeat business.  Once your customers have found your company useful once, you take future opportunities to remind them to spend some money with you again.  If legitimate email marketing dies, this could cause a dent in online trading.  That’s bad for folks like me and the folks who read these articles.

GMTV are finding they’re already a victim of spam filtering, or customer reluctance to read marketing email.  They’re originating messages that either get binned by the audience, or their audience’s automatic spam filters.  Customers who traditionally would have received emails informing them of new online content are now being encouraged to install a desktop client that alerts them in real time about new content.

I’m pretty worried that in order to stay in touch with suppliers in the future, I’ll be expected to use one particular desktop client.  This means in some cases, I may have to use one particular desktop environment for a start.  Secondly, this makes it more likely that I’ll receive malware – how can I trust the originators of the client?  And as it’s a network service, any desktop alerting system is also potentially at risk of abuse or spam.  So I get realtime spam as well as spam waiting for me when I check my mail.

Instead, I hope that more email clients incorporate RSS systems in the future, as Apple Mail has done in the latest release.  Using Apple Mail, I can subscribe to marketing announcements from the companies that I want to hear from, and have those arrive in a specified area of my mail client, and they’re not inflicted by spam.  As I try to make clear everywhere possible – use open standards and open protocols, if you want to keep your doors open to new business.

Internationalisation of DNS continues

Like most original internet standards, the DNS was designed to initially suit the needs of any section of the world that could communicate using 7-bit ASCII and Latin character sets. Then the internet became really popular. Everywhere. The DNS had to evolve to cope with naming schemes that came from alphabets all over the world.

All successful internet protocols are elegant and simple by design. This makes it possible to retro-fit great ideas someone has one. Internationalisation was proposed in 1992, and it eventually became possible to register Internationalised Domain Names (IDNs) in the .com space in 2003. Standards move slowly on the internet!

IDN is up for discussion again at the 31st ICANN meeting on Monday. This time, the world’s registry community are meeting in New Delhi, one of the most significant IT regions of the non-Latin world, to discuss the remaining “glitch” in the IDN system. An IDN might look like this: .com. Therefore any user still needs to be able to type .com in order to reach the resource they request. There is a proposal at the ICANN meeting to add Internationalised top-level domains, actual complimentary TLDs to .com, that will mean that resources can be reached in any supported alphabet.

This is interesting stuff. One school of thought is that this could significantly assist the development of electronic enterprise in many more pockets of the world. The supremacy of Silicon Valley as the web’s main economy would then be broken. I think differently – I think that .com is now too established as the main ecommerce ‘brand’ TLD, and attempts to localise the meaning of .com will be fruitless. .com means “I trade online”. Despite .biz and similar TLDs being equal in technical terms, they are not equal in the eyes of shoppers or traders. .com now has specific global meaning, and can’t be diluted.

The Network Is The Computer. Again.

Ever since John Gage of Sun first offered the phrase “The Network Is The Computer” to the world, people have been using it as inspiration. Sun use it to explain that they mean Social Networking without actually using the phrase (they prefer the old fashioned “community”).

I think web 2.0 developers are offering a new perspective on the phrase. There was a time that if I wanted my computer to do a job, I would find or buy a piece of software that was engineered to cause that job to be performed. The Internet and the Web 2.0 culture is changing this model completely. Its becoming the case that the computer in front of me is stupid, and some cluster of servers in MarketPost Tower, San Jose actually does all the work.

Enter Mint. Mint is self described as refreshing money management. Sound like what Quicken does if I install it on my computer, but with the word ‘refreshing’ in front? It’s refreshing (well, different) because to use their software I simply need to visit their website and sign up. Mint will learn about your spending via your online banking accounts, and aggregate your personal finance situation into one simple application. It sounds so simple, and solves no problems that weren’t also solved with personal finance software on the ZX Spectrum. And yet on Launch day just a couple of months ago it won $50,000 in the innovation contest, ‘Techcrunch40‘.

There’s two lessons from this. Firstly, there’s a new blueprint for guaranteed success in the web 2.0 world, and that is if you can take all the features of a market leading piece of traditional software, and build that functionality into a website, then you will be heralded an innovator. But you have to be the first person to do that. So hurry.

Secondly, and this is a lesson that wont be learned by the general public for a while, is that putting your private data in a central place that you do not control, is a weak point of attack when someone wants to learn something about you. Maybe someone wants to target your assets. Maybe the government want a nosey in what you’ve been spending. Maybe a consumer profiling company want to run a survey. They only have to get hold of one set of keys to your mint account, and all three of these groups can do what they want with your data. If my computers were stolen, then an expert would be able to find out quite a lot about me. If I put my data online and it gets stolen, the same can happen. And if everyone puts their data online in the same place, then it becomes very attractive to break in and steal it for lots of groups.

Making the right ipv6 noises

I’ve been allowing the webcast of RIPE55 to mutter away in my ears all week and have let myself get distracted from time to time when the topics turned relevant to networks I operate or the chatter got interesting.  A bit like the end of today’s ipv6-wg session.
Six months ago I was quite sure that v6 was not likely to be viable as a way to guarantee the continuity of fresh (and useful) addressing resources when the IANA ipv4 pool is expired (around two years from today).  I thought that our best chances remained with working with our remaining v4 options; reclassifying 240/4 as normal unicast, pressure or perhaps even (shudder) regulation imposed on the holders of unused legacy class-A space, tighter policy control on remaining v4, and eventually a market model for new address distribution.

This is operator speak for burying one’s head in the sand.  240/4 is not going to useful for end-to-end internet use any time soon, thanks to the amount of equipment which is configured to not permit assignment of a class-e address to an interface, which unless you are using a very old or patched operating system will include the computer you are sitting in front of.  The best we can hope for is that it will be useful for private internetworking at some point in the future.  The class-A holders wont give their legacy allocation back because they think its worth lots of money, but the holders (and their future customers who ‘buy’ rights to a slice of addresses) are going to be disappointed when their announcements are ignored – anyone who thinks that HP can deaggregate 15.0.0.0/8 into 65,000 /24 networks and sell it off (HP not subject to the cost of every single other router in the world carrying the routes for free) is mistaken.

I have changed my mind and started testing v6 out.  I care about the end to end internet, and I care that addressing resources should be available to anyone who can justify need and that this is good for innovation. I am therefore pleased to hear the ipv6-wg discussing text for a new RIPE recommendation document that boils down to:

• New v4 addresses wont be available in two to four years, and this means your future network plans STOP HERE if they only involve v4.
• RIPE urges the deployment of v6

I am however concerned about one thing.  There are a number of networks which use provider independent addresses so that they can multihome (connect to more than one ISP for redundancy and performance.)  This is not catered for in general circumstances with RIPE v6 policies, except in very special circumstances.  This is due to a desire to keep the v6 routing table small so that we avoid the problems of an unwieldy routing table like today’s (by today’s equipment standards) in the future.

I don’t think the RIPE community can justify publishing such a document until 2006-01 reaches concensus, and in fact that this resolution is that IPv6 PI is available, it is a /48, and it’s possible to get some if you are multihoming.  You can’t tell operators that they must move to v6 without giving them a migration path.

The argument revolves around routing table size and control, but the community must allow networks that rely on connectivity to multihome, it’s an accepted technique for many operations now.  By all means require the organisation to continuously multihome or give the addresses back, but we must let the concept of v6 PI exist.

Otherwise needs based address resource allocation dies in around two years.

Making Round Robin DNS usable.

I have been fairly consistently telling people a lie for the last ten years – and that is that Round Robin DNS can not be used for high availability. Its a view I have held pretty strongly, but two people have shown me techniques in the last week that have made me change my mind.

First, a note on what Round Robin DNS is. Hopefully you know what dns is! When a resource record is answered in a round robin fashion, a single resource record has several answers. When high-availability is build into a protocol (e.g. DNS itself – retries) RRDNS works pretty well. dig . @a.root-servers.net if you want a reference service which uses this principal.

RRDNS is also used to share load between nodes in a cluster where the protocol is not as forgiving, with unpredictable results. If I serve my website via two nodes, so the dns looks like:

www IN A 10.1.1.2
www IN A 10.1.1.3

then my website is not at all highly available. Pretty far from it, in fact – if either the nodes 10.1.1.2 or 10.1.1.3 fail, then I have suboptimal results, when in a highly-available configuration, I only want to suffer a performance degredation or outage if all of my nodes fail.Two colleagues showed me two different systems earlier this week that solve this fundamental issue with traditional RRDNS implementations – and that is handling an outage. Both of them use the premise that you never use the box’s own, unique (e.g. management) IP address in the RRDNS Resource Record, instead you use ‘virtual’ or aliases IP addresses that are free to roam around your server cluster. Consider my zone with this change:

www1 IN A 10.1.1.2
www2 IN A 10.1.1.3
www IN A 10.1.1.201
www IN A 10.1.1.202

Now I can address both servers in this small cluster with their own address, and also point customers to an address which can be served by any of those machines. If both web servers are up, they each have one of the additional addresses aliased. If 10.1.1.2 goes down, the computer 10.1.1.3 can adopt both 10.1.1.201 and 10.1.1.202 as aliased interfaces.

I can’t take the credit for finding either of the technologies I mention below, that can manage the automated failover of aliased interfaces, but I have seen both work this week.

Firstly, Wackamole which is brilliant if you are already using ‘Spread’. Nodes on a Spread ring send health messages to each other. If a node goes silent, or sends a communication that it is failing (deliberately stopping), another server in the cluster adopts that server’s aliased address. You can also configure a mechanism that notifies other devices that their arp cache entry for the vip is staled.

The second is good-old VRRP, which I had only ever considered as a redundant first-hop protocol, but actually can be run on a Linux server in order to provide these Virtual-interface floating features. This system does not rely on the Spread messaging bus, but in order to float several VIPs in a cluster, one vrrpd instance, and therefore group, must be configured for each vip. For my simple cluster of two servers above, I would need to run vrrpd twice on each server, to balance both vips as such :

/usr/local/sbin/vrrpd -Ni eth0 -v 48 -p 120 -g 3 -t 10.1.1.201 -S /var/run/vrrpd_48.state
/usr/local/sbin/vrrpd -Ni eth0 -v 49 -p 100 -g 3 -t 10.1.1.202 -S /var/run/vrrpd_49.state

Careful planning needs to be done in order to use RRDNS to share load. I have deliberately refrained from describing this technique as Load Balancing, because at best it is load sharing – you can’t balance traffic, only politely suggest to visitors that that might like to share traffic across your cluster. Additionally, you need to do some capacity thinking – if you have two, or even three nodes in your cluster, if one of the servers fails, one of the servers which is left is dealing with a doubling of the traffic that it was doing before. There is really no elegant way to share three vips across two servers.

Its cheap though. And as I have learned, you can force it to work with a bit of ingenuity.

Net Neutrality debate gets traction

I cited a DoJ statement in a previous article that was destined to stagnate or kill all innovation on the web, by permitting ISPs to end the end-to-end nature of the internet.

I’ve been trying to draw the attention of some other technical people by talking about NN on mailing lists.  Sadly some people have got the wrong end of the stick – they think that the rule simply states that ISPs will be allowed to carry customer Quality of Service preferences to the edge of their ISP networks.  This is not the case – the Net Neutrality debate is about whether an ISP should be permitted to be un-neutral irrespective of, or even against their customers’ wishes.

One company that stands to loose most from an un-neutral net is Google who have been quick to earn column inches supporting the neutral net.

Peter Norvig, Google’s Director of Research speaks from the front cover of this weeks’ Computing arguing that the only reason “the net has grown far beyond the original perception bounds” was in fact ” because it is open, and because services can be launched without being fettered by higher level control.  At Google, we think it is good for competition to try to keep services this way, and that is what we are going to push for.”

I don’t want an internet dominated only by companies with deep pockets.  Google, the archetypal company with the deepest pockets don’t even want an internet dominated only by companies with deep pockets.  The only parties that stand to gain from the NN war as the ISPs that have been the most aggressive at driving down consumer internet charges, to un-sustainable points.  Don’t give them a life line.  Let’s have a fair internet.

A license to do something bad isn’t reason to.

For months, ISPs in Europe have been campaigning to preserve their ‘mere conduit’ status, or in English they have been fighting to prove that they should be able to treat all packets, between customers and the resources that they want to access equally.  This means, no content blocking, monitoring, and fundamentally no commercial favouritism – or as it is referred to Network Neutralism.

At the same time, some ISPs in America have been petitioning to abolish the concept of Net-neutrality on their networks.  These campaigns have not fallen on deaf ears, the US Department of Justice now consider that without legislation permitting ISPs to charge companies who host content for access to the ISP’s customers shifts the “entire burden of implementing costly network expansions and improvements onto consumers“.

This shows a wanton lack of understanding of the model with which internet connectivity is sold.  I, as a consumer, buy connectivity from a company who will try hard to ensure my packets will flow towards their intended recipient.  Someone who wants to host content pays another ISP some money to ensure that their packets flow back towards me when I visit their website, online shop, or use their service.  That’s the model, and its the only one that works.

If I want to set up a shop online, I should only have to pay the companies that I have a direct connectivity relationship with.  It should not be permitted for a company that I do not connect through, to hold their eyeball customers to ransom, demanding money from me, the shopkeeper, to allow me to reach our mutual customers.  Particularly if this customer is selling internet connectivity – there is a strong ‘end to end’ implication with that phrase.

The DoJ are wrong – what they think is a burden to consumers is just the fair market price that internet connectivity is sold for today.  An ISP should charge their customers enough to make sure that they can pay to support their network.  As a consumer, if I want to use MSN, Skype, Hotmail, Google, the BBC, iTunes, or any other online service, I should be able to, because I have paid for my ISP to try their best (without getting in the way) to get data between me and the services I use.

Net-neutrality has been the driving force behind innovation online, and we would not have the wide variety of services online today, if starting to trade online meant you had to pay lots of ransom notes to every ISP in America first.