my web 0.2 website

DNSSEC and SSL certificates

Dr. Jörg Schweiger of the German domain name registry DENIC posed an interesting question at this morning’s first DENOG meeting, in Frankfurt.

Would domain name users who are concerned about the accuracy of data served pay extra for the ability to sign their DNS zone ?  A handful of people in the room raised their hand in agreement, but the overwhelming majority of operators did not.

His argument was that this compared well with SSL certification authorities who sell certificates that suggest that visitors to a website are interacting with a validated entity, and the technology guarantees privacy between the visitor and the website.  It’s this technology which makes buying and selling online safe.

However, I think that DNSSEC has different aims altogether – simply to guarantee that DNS data is not changed en-route between the authoratative server, through the caches, all the way to users.  Therefore there are significant attack mitigation reasons to deploy DNSSEC, so I hope that operators will begin trials (we are doing so), and that the pace of trials will quicken as the root zone will be signed this year.

If DNSSEC is deployed as designed, then temporary and brief mistakes will not be imported into DNS caches, users will not fall foul to tampered data in caches, and we all receive an authenticated/secure channel for distributing DNS data inside an organisation.

The argument that Dr. Schweiger used is that DNSSEC adds an operational and technical burden to registries (extra communication with registrars, more complex software, additional CPU and bandwidth requirements).

I hope that my colleagues in other organisations agree that there are significant infrastructure advantages to freely allowing DNSSEC to grow, and that Moore’s Law, automation, and the fact that DNS registries normally find it simple to peer widely with ISP networks will offset the needs to consider the commercial signing model.

Extreme Switch / OpenSSH bug

I have been trying to get a patch applied to Debian’s openssh-client packages since February which would fix a bug that prevents me from logging into Extreme switches via ssh:

trials:/usr/src/openssh-5.1p1# ssh hextreeme -l netadmin

Keyboard-interactive authentication

Enter password for netadmin:

channel 0: open failed: resource shortage: Channel open failed

The bug is described in Debian bug 495917, and it also prevents connection to some NetScreen firewalls.  I have this bug with Debian 4 (openssh-client 4.3p2-9etch3) and Debian 5 (openssh-client 1:5.1p1-5).

If anyone else is experiencing the same bug and needs a quick fix, then you can download my Debian packages which replace openssh-client.  You of course need to hold the packages if you don’t want them overwriting by a security fix.

• Debian 4 openssh-client_4.3p2-9etch3_i386.deb
• Debian 5 openssh-client_5.1p1-5_i386.deb

By using this software you agree to hassle both the debian-ssh team and extreme to sort their stuff out!

Openness and telecoms

This is a response to Lee Dryburgh’s article on Skype.  We had a debate on Twitter, but I have not yet mastered the art of debate in 140 characters!

Lee’s premise is that “Certainly Skype is not a walled garden. All things being relative, it’s certainly not overly closed either.”  Lee claims that the accusations of closeness are unfair, because they are levied by commentators who advocate SIP based addressing and dialing rather than any other system.

This is not my premise.  I claim that Skype is closed because calls are signalled and completed using protocols that are entirely secret as a matter of policy.  Skype’s founder presented at Spring VON 2007 and stated that if Skype did not keep their protocols entirely secret, then Skype would be full of spam and attack like email is.  I think this is a poisonous claim, telephone networks have been interconnecting around the world since telephony was conceived.  By not allowing telecoms firms to interconnect between the skype namespace and other networks, Skype have prevented openness to develop and maintain a monopoly position. That’s perfectly acceptable business, but it is not in the slightest bit open.

Randy Bush googled Walled Garden for a recent presentation and found this cartoon.  I like this definition because it’s correct.  Is Skype a Walled Garden ?  Lee says a Walled Garden is a commercial restriction, for example, “sharing of ringtones via Bluetooth, using WiFi from a PDA, having access to all Web sites“.  I think that only allowing interconnection with the purchase of an upgrade like SkypeOut is a restrictive or practice that suggests Skype is a Walled Garden.  Worst of all a call between two VoIP networks using this method requires default PSTN routing, which harms signal quality, and prevents the expansion of next-generation services such as Wideband/High Definition audio.

The meshing of networks, whether they are traditional voice or IP networks, leads to higher audio quality and increased reliability.  Keeping telephony systems and protocols secret in order to prevent meshing may well be a viable business model, but it is not an open business model.

Internet broken for ASN32 speakers today.

Not trying to point fingers or name-and-shame, just to raise the profile of a nasty little bug handling breaches of RFC4893.  This post is basically shaped from a message I posted to nanog earlier.

AS196629 (3.21 in asdot) announce 91.207.218.0/23.  Experienced eyes will notice that this is quite a large as number.  It’s a ‘new’ 4-byte ASN.  When an OpenBGPd speaker with 4-Byte ASN support receives the update for this message, the session is torn down with the daemon logging a ‘fatal error’. Why?
OpenBGPd is checking AS4_PATH to ensure that it contains only AS_SET and AS_SEQUENCE types, as per RFC4893.  When processing the UPDATE for 91.207.218.0/23 it sees :

91.207.218.0/23
Path Attributes – Origin: Incomplete
Flags: 0×40 (Well-known, Transitive, Complete)
Origin: Incomplete (2)
AS_PATH: xx xx 35320 23456 (13 bytes)
AS4_PATH: (65044 65057) 196629 (7 bytes)

See the confederation ASNs in the AS4_PATH ?  Thats forbidden :

To prevent the possible propagation of confederation path segments outside of a confederation, the path segment types AS_CONFED_SEQUENCE and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH attribute. RFC 4893.

The RFC does not suggest how to handle AS4_PATH violations, but if the bad path is learned on every upstream, this will cause a network with obgpd edges to disconnect from the internet…. Modifying the OpenBGPd software to permit AS_CONFED_SEQUENCE, AS_CONFED_SET in an as4_path causes the path to be accepted and the session is not torn down.  This isn’t a great fix.
The impact today is fairly limited as there are relatively few bgp speakers honouring the 4-byte ASN protocol extension rules, but as code that support these features creeps around the internet, the next time this happens the impact could be much greater, so we need to understand which implementation of which BGP software caused this illegal origination.

From a software point of view, I want to see a configurable option to reject the route but keep the session, reject the route and drop the session, accept the route but log/send trap, etc.

In any case we need to publish the arrangement that has led to this mistake so that other networks using the same toolset to originate prefixes can avoid the same situation happening.  I have made contact with an engineer at the NOC who are investigating.

VoIP For Network Operators Tutorial

These are the slides that I presented at NANOG44 in Los Angeles on Sunday, “VoIP For Network Operators“.

This talk was for network operators looking to build voice segments of their network, and the slides cover

• Voice Basics for SPs
• Why Operators should care
• Voice Peering
• Metrics
• VoIP Security

Youtube pushed off the air

In between browsing Facebook and Youtube, the UK economy generates $1,930,000,000 of output a year. Thats $550,000 every two and a half hours. Well if today had been a work day, there’d have been one two and a half hour period where that was much higher. That’s because in a pique of routing excitement, Pakistan Telecom managed to hide Youtube from most of the internet for that length of time.

Pakistan Telecom and Youtube are likely to have no commercial relationship in place to carry Youtube traffic – particularly as around two hours ago, according to Yahoo News, the story broke that the Pakistan Government required ISPs operating in the country to block Youtube. Despite this, Pakistan Telecom were able to cause ISPs all over the world to send traffic that should be destined for Youtube to Pakistan instead.

This is because the protocol that determines how to find my network on the internet, is shaped by how “specific” the announcement of my network is. If I make an announcement of a network of 1,024 addresses, and someone else makes a second announcement of 256 addresses within a subset of my 1,024, then the network which announces the smaller subset win the traffic destined to those hosts. This is a feature – fully by design – of the BGP routing protocol. Almost every time a more specific block of addresses is announced, this is because the administrators of those networks intend for the routing to be different for a subset of a large number of addresses.

Sadly, there are accidents from time to time – another network can announce a subset of my addresses without my knowledge or permission, and they win the traffic that should have gone to me. This happened today – it seems that Pakistan Telecom decided to inject a fake route to their network containing Youtube’s webservers, and accidently then leaked that route to the networks they connect to.

Small networks and end sites can limit the chances that they will leak bad routes by explicitly listing the network addresses that they intend to send to their upstream or peered networks. Larger networks may find it harder to stop themselves propagating someone else’s mistake, because they may have a contract to carry forward any announcement that their customers make. Furthermore, the complexities of their own networks mean that an engineer working under pressure after announcements made by government ministers are more likely to make a typo error and do the wrong thing.

Richard Clayton presented a very interesting set of commentaries at the last LINX meeting. He commented that right now its very obvious indeed when someone hijacks some of my network space in this way, because all of my traffic disappears. Youtube were probably aware that something was very wrong within moments of the announcement. What if someone builds an infrastructure to steal my traffic – or at least some of my traffic – but after doing something with it, they send it back to me, it is much harder for me to spot that anything is wrong.

This is a significant risk to ecommerce infrastructures that competitors or e-pirates could seize upon opportunities to steal customer behaviour data. What if a wizard stole the network containing your web server, proxied your shop, but set up a fake checkout? How quickly would you spot?

Because this problem is inherent to the routing protocol, this is the obvious place to fix it. There are attempts to blend PKI with routing information, so that peers can verify the validity of your announcements. S/BGP (secure BGP) requires me to sign my announcements, and gives my peers a method to check in an impartial internet community database that my announcement is valid. It is the sort of technology that would have prevented Youtube from disappearing off the air today.

Vodafone’s legal challenge to fast porting.

I tried to open some dialogue with colleague members of the ITSPA about Vodafone’s legal challenge to Ofcom’s two-hour number port ruling.  Instead I got a number of offlist replies suggesting Vodafone’s challenge is still news to many in the industry.

Today, if you want to port your number from one service provider to another, it relies on two major coincidences – firstly that your old and new provider have an agreement in place to manage the technical transfer between the two networks, and secondly that your old provider remains fully willing to forward all calls destined from your old number, to your new service provider.

There are several issues with such a system – the first is that your old provider are still very much involved, so their technical or commercial failure causes a problem long after you have ported away, another is that the process is slow and manual, and a third is that not all service providers have agreements to permit number porting (called a Mutual Porting Agreement in the industry).

Vodafone are concerned about the costs of the new system, even though an industry group UKPorting has only just begun to gather information about how the system should work.  I think that it’s a flawed premise to argue that a system is too expensive before a system is selected (and associated costs are announced).  Instead Vodafone should get involved with designing a perfect system.

The UKporting system to facilitate fast, reliable, and simple porting must happen, and must succeed.  We have to protect consumers who port their number from failures caused by their former service provider.

I am concerned that the system may mean all multihomed telephone networks will need to move to any all-call-query model that’s run by one natural monopoly.  If a single entity holds the industry to ransom, we have not moved forward – there’s still a single commercial or technical position that can fail to break your port.  The single All-Call-Query model also lends itself well to governments having access to a single point where recording of most call attempts can be made.

The Network Is The Computer. Again.

Ever since John Gage of Sun first offered the phrase “The Network Is The Computer” to the world, people have been using it as inspiration. Sun use it to explain that they mean Social Networking without actually using the phrase (they prefer the old fashioned “community”).

I think web 2.0 developers are offering a new perspective on the phrase. There was a time that if I wanted my computer to do a job, I would find or buy a piece of software that was engineered to cause that job to be performed. The Internet and the Web 2.0 culture is changing this model completely. Its becoming the case that the computer in front of me is stupid, and some cluster of servers in MarketPost Tower, San Jose actually does all the work.

Enter Mint. Mint is self described as refreshing money management. Sound like what Quicken does if I install it on my computer, but with the word ‘refreshing’ in front? It’s refreshing (well, different) because to use their software I simply need to visit their website and sign up. Mint will learn about your spending via your online banking accounts, and aggregate your personal finance situation into one simple application. It sounds so simple, and solves no problems that weren’t also solved with personal finance software on the ZX Spectrum. And yet on Launch day just a couple of months ago it won $50,000 in the innovation contest, ‘Techcrunch40‘.

There’s two lessons from this. Firstly, there’s a new blueprint for guaranteed success in the web 2.0 world, and that is if you can take all the features of a market leading piece of traditional software, and build that functionality into a website, then you will be heralded an innovator. But you have to be the first person to do that. So hurry.

Secondly, and this is a lesson that wont be learned by the general public for a while, is that putting your private data in a central place that you do not control, is a weak point of attack when someone wants to learn something about you. Maybe someone wants to target your assets. Maybe the government want a nosey in what you’ve been spending. Maybe a consumer profiling company want to run a survey. They only have to get hold of one set of keys to your mint account, and all three of these groups can do what they want with your data. If my computers were stolen, then an expert would be able to find out quite a lot about me. If I put my data online and it gets stolen, the same can happen. And if everyone puts their data online in the same place, then it becomes very attractive to break in and steal it for lots of groups.