my web 0.2 website

Internet broken for ASN32 speakers today.

Not trying to point fingers or name-and-shame, just to raise the profile of a nasty little bug handling breaches of RFC4893.  This post is basically shaped from a message I posted to nanog earlier.

AS196629 (3.21 in asdot) announce 91.207.218.0/23.  Experienced eyes will notice that this is quite a large as number.  It’s a ‘new’ 4-byte ASN.  When an OpenBGPd speaker with 4-Byte ASN support receives the update for this message, the session is torn down with the daemon logging a ‘fatal error’. Why?
OpenBGPd is checking AS4_PATH to ensure that it contains only AS_SET and AS_SEQUENCE types, as per RFC4893.  When processing the UPDATE for 91.207.218.0/23 it sees :

91.207.218.0/23
Path Attributes – Origin: Incomplete
Flags: 0×40 (Well-known, Transitive, Complete)
Origin: Incomplete (2)
AS_PATH: xx xx 35320 23456 (13 bytes)
AS4_PATH: (65044 65057) 196629 (7 bytes)

See the confederation ASNs in the AS4_PATH ?  Thats forbidden :

To prevent the possible propagation of confederation path segments outside of a confederation, the path segment types AS_CONFED_SEQUENCE and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH attribute. RFC 4893.

The RFC does not suggest how to handle AS4_PATH violations, but if the bad path is learned on every upstream, this will cause a network with obgpd edges to disconnect from the internet…. Modifying the OpenBGPd software to permit AS_CONFED_SEQUENCE, AS_CONFED_SET in an as4_path causes the path to be accepted and the session is not torn down.  This isn’t a great fix.
The impact today is fairly limited as there are relatively few bgp speakers honouring the 4-byte ASN protocol extension rules, but as code that support these features creeps around the internet, the next time this happens the impact could be much greater, so we need to understand which implementation of which BGP software caused this illegal origination.

From a software point of view, I want to see a configurable option to reject the route but keep the session, reject the route and drop the session, accept the route but log/send trap, etc.

In any case we need to publish the arrangement that has led to this mistake so that other networks using the same toolset to originate prefixes can avoid the same situation happening.  I have made contact with an engineer at the NOC who are investigating.

2011 – An addressing odyssey. Preparing enterprise for IPv6.

Yesterday I gave a talk to Sheffield GeekUp on preparing enterprises for IPv6 [download].  The premise of the talk was :

• IPv4 addresses are scarse, and at current consumption rates, the IANA pool of free v4 addresses will be gone at the start of 2011.
• This starts a “Post IPv4 world” where the IPv4 internet continues to function as before (certainly initially), but obtaining new addresses becomes harder and expensive.  This inhibits expansion of existing firms, and new entrants to the market.
• Address trading is likely to lead to a larger routing table, meaning that failure-recovery times increase, and the risk of blackholes on the internet increases.
• Large broadband providers may not have enough v4 addresses to give one address per customer.  This means protocol translation techniques need to be used, which break the end to end model.  We rely on the end to end model when innovating new services on the internet.
• If services and consumers gradually roll v4 and v6 (dual stack), the negative impact of markets for addresses, routing problems, and translation can be mitigated.
• Service providers are enabling v6 in the core.  Enterprises need to move next in order to get the world v6 ready.

The advice I gave was :

• Today’s market leaders are already learning v6 lessons in their labs, (e.g. ipv6.google.com).  They are doing this to help them retain market leadership.  If you want to retain your market position, start labbing your applications and service provision with v6.
• Write a policy stating all new purchases of infrastructure and services need to be from providers with v6 support, or a well defined v6 road map.  In other words, make v6 a “life cycle upgrade”.
• Share information, and learn information from your industry peers.
• I also listed some advice to developers with regard to v4 and v6 differences.
• I then delivered a very quick primer to those who have not seen v6 deployed before.

My hope is that this talk is improved upon and delivered internationally to enterprises.

VoIP For Network Operators Tutorial

These are the slides that I presented at NANOG44 in Los Angeles on Sunday, “VoIP For Network Operators“.

This talk was for network operators looking to build voice segments of their network, and the slides cover

• Voice Basics for SPs
• Why Operators should care
• Voice Peering
• Metrics
• VoIP Security

Mac VNC Client for Linux KVM

When you build a KVM guest, if you want to install the guest over the network, you should attach the video console of your guest to a VNC display.

How can I put this..? This is quite a novel way of doing it.  I think there’s a reason that more virtualisation systems don’t work in this way.  VNC is not great, but I am sure there is a reason that I can’t use a dummy serial port instead.  I’d have preferred RDP, but perhaps there’s a reason I can’t use that too.

I normally use Chicken of the VNC as a mac osx client, because it has a funny name, and has always worked.  However, it crashes and burns (see screen shot) when trying to install Debian on a KVM guest.  Hopefully I can save someone else an evenings’s worth of trying every other mac vnc client, and offer the fix.  Just use VNCViewer.  I tried this after half a dozen others which all failed in a similar way to Chicken.

Any comments on why RDP or Serial might not have been better welcome.

Youtube pushed off the air

In between browsing Facebook and Youtube, the UK economy generates $1,930,000,000 of output a year. Thats $550,000 every two and a half hours. Well if today had been a work day, there’d have been one two and a half hour period where that was much higher. That’s because in a pique of routing excitement, Pakistan Telecom managed to hide Youtube from most of the internet for that length of time.

Pakistan Telecom and Youtube are likely to have no commercial relationship in place to carry Youtube traffic – particularly as around two hours ago, according to Yahoo News, the story broke that the Pakistan Government required ISPs operating in the country to block Youtube. Despite this, Pakistan Telecom were able to cause ISPs all over the world to send traffic that should be destined for Youtube to Pakistan instead.

This is because the protocol that determines how to find my network on the internet, is shaped by how “specific” the announcement of my network is. If I make an announcement of a network of 1,024 addresses, and someone else makes a second announcement of 256 addresses within a subset of my 1,024, then the network which announces the smaller subset win the traffic destined to those hosts. This is a feature – fully by design – of the BGP routing protocol. Almost every time a more specific block of addresses is announced, this is because the administrators of those networks intend for the routing to be different for a subset of a large number of addresses.

Sadly, there are accidents from time to time – another network can announce a subset of my addresses without my knowledge or permission, and they win the traffic that should have gone to me. This happened today – it seems that Pakistan Telecom decided to inject a fake route to their network containing Youtube’s webservers, and accidently then leaked that route to the networks they connect to.

Small networks and end sites can limit the chances that they will leak bad routes by explicitly listing the network addresses that they intend to send to their upstream or peered networks. Larger networks may find it harder to stop themselves propagating someone else’s mistake, because they may have a contract to carry forward any announcement that their customers make. Furthermore, the complexities of their own networks mean that an engineer working under pressure after announcements made by government ministers are more likely to make a typo error and do the wrong thing.

Richard Clayton presented a very interesting set of commentaries at the last LINX meeting. He commented that right now its very obvious indeed when someone hijacks some of my network space in this way, because all of my traffic disappears. Youtube were probably aware that something was very wrong within moments of the announcement. What if someone builds an infrastructure to steal my traffic – or at least some of my traffic – but after doing something with it, they send it back to me, it is much harder for me to spot that anything is wrong.

This is a significant risk to ecommerce infrastructures that competitors or e-pirates could seize upon opportunities to steal customer behaviour data. What if a wizard stole the network containing your web server, proxied your shop, but set up a fake checkout? How quickly would you spot?

Because this problem is inherent to the routing protocol, this is the obvious place to fix it. There are attempts to blend PKI with routing information, so that peers can verify the validity of your announcements. S/BGP (secure BGP) requires me to sign my announcements, and gives my peers a method to check in an impartial internet community database that my announcement is valid. It is the sort of technology that would have prevented Youtube from disappearing off the air today.

Internationalisation of DNS continues

Like most original internet standards, the DNS was designed to initially suit the needs of any section of the world that could communicate using 7-bit ASCII and Latin character sets. Then the internet became really popular. Everywhere. The DNS had to evolve to cope with naming schemes that came from alphabets all over the world.

All successful internet protocols are elegant and simple by design. This makes it possible to retro-fit great ideas someone has one. Internationalisation was proposed in 1992, and it eventually became possible to register Internationalised Domain Names (IDNs) in the .com space in 2003. Standards move slowly on the internet!

IDN is up for discussion again at the 31st ICANN meeting on Monday. This time, the world’s registry community are meeting in New Delhi, one of the most significant IT regions of the non-Latin world, to discuss the remaining “glitch” in the IDN system. An IDN might look like this: .com. Therefore any user still needs to be able to type .com in order to reach the resource they request. There is a proposal at the ICANN meeting to add Internationalised top-level domains, actual complimentary TLDs to .com, that will mean that resources can be reached in any supported alphabet.

This is interesting stuff. One school of thought is that this could significantly assist the development of electronic enterprise in many more pockets of the world. The supremacy of Silicon Valley as the web’s main economy would then be broken. I think differently – I think that .com is now too established as the main ecommerce ‘brand’ TLD, and attempts to localise the meaning of .com will be fruitless. .com means “I trade online”. Despite .biz and similar TLDs being equal in technical terms, they are not equal in the eyes of shoppers or traders. .com now has specific global meaning, and can’t be diluted.

European Internet exchange update slides

I presented a talk on recent European Internet exchange news [download] with Mike Hughes from the LINX last week at UKNOF. Many of the attendees run networks that do not peer publicly, so it was a pleasure to explain the impact that European IXPs have on member traffic. We also then gave a perspective on peering in London.

Many of the statistics came from Serge at Euro-IX who did the leg work for the raw figures.

The highlight points of the talk were

• Euro-IX identify 103 exchanges in Europe, in 31 countries. (3 in 1993)
• 8 Exchanges in the UK (was 9 until BT’s UK6x closed)
• At the end of 2007 networks publicly peered 1.215Tbit/sec at peak.
• More public peering in EU than US (but it’s cheaper to peer in EU thanks to lower x-connect fees, and cheap ubiquitous mutual exchanges)
• London is #1 for network reach – 601 networks peer publicly, 415 peer exclusively in the UK.
• 22% of LINX members peer exclusively at LINX, 31% of LONAP members peer exclusively at LONAP.
• 577 networks peer at more than one IXP, and one network (Colt) is present at 19 exchanges!
• Last year the good weather in April caused an additional summer-time traffic dips in Europe, in addition to the regular dip in July/August

There’s other stuff in the slides too, such as the usual traffic updates for various major exchanges in Europe.

The Network Is The Computer. Again.

Ever since John Gage of Sun first offered the phrase “The Network Is The Computer” to the world, people have been using it as inspiration. Sun use it to explain that they mean Social Networking without actually using the phrase (they prefer the old fashioned “community”).

I think web 2.0 developers are offering a new perspective on the phrase. There was a time that if I wanted my computer to do a job, I would find or buy a piece of software that was engineered to cause that job to be performed. The Internet and the Web 2.0 culture is changing this model completely. Its becoming the case that the computer in front of me is stupid, and some cluster of servers in MarketPost Tower, San Jose actually does all the work.

Enter Mint. Mint is self described as refreshing money management. Sound like what Quicken does if I install it on my computer, but with the word ‘refreshing’ in front? It’s refreshing (well, different) because to use their software I simply need to visit their website and sign up. Mint will learn about your spending via your online banking accounts, and aggregate your personal finance situation into one simple application. It sounds so simple, and solves no problems that weren’t also solved with personal finance software on the ZX Spectrum. And yet on Launch day just a couple of months ago it won $50,000 in the innovation contest, ‘Techcrunch40‘.

There’s two lessons from this. Firstly, there’s a new blueprint for guaranteed success in the web 2.0 world, and that is if you can take all the features of a market leading piece of traditional software, and build that functionality into a website, then you will be heralded an innovator. But you have to be the first person to do that. So hurry.

Secondly, and this is a lesson that wont be learned by the general public for a while, is that putting your private data in a central place that you do not control, is a weak point of attack when someone wants to learn something about you. Maybe someone wants to target your assets. Maybe the government want a nosey in what you’ve been spending. Maybe a consumer profiling company want to run a survey. They only have to get hold of one set of keys to your mint account, and all three of these groups can do what they want with your data. If my computers were stolen, then an expert would be able to find out quite a lot about me. If I put my data online and it gets stolen, the same can happen. And if everyone puts their data online in the same place, then it becomes very attractive to break in and steal it for lots of groups.