Not trying to point fingers or name-and-shame, just to raise the profile of a nasty little bug handling breaches of RFC4893. This post is basically shaped from a message I posted to nanog earlier.
AS196629 (3.21 in asdot) announce 91.207.218.0/23. Experienced eyes will notice that this is quite a large as number. It’s a ‘new’ 4-byte ASN. When an OpenBGPd speaker with 4-Byte ASN support receives the update for this message, the session is torn down with the daemon logging a ‘fatal error’. Why?
OpenBGPd is checking AS4_PATH to ensure that it contains only AS_SET and AS_SEQUENCE types, as per RFC4893. When processing the UPDATE for 91.207.218.0/23 it sees :
91.207.218.0/23
Path Attributes – Origin: Incomplete
Flags: 0×40 (Well-known, Transitive, Complete)
Origin: Incomplete (2)
AS_PATH: xx xx 35320 23456 (13 bytes)
AS4_PATH: (65044 65057) 196629 (7 bytes)
See the confederation ASNs in the AS4_PATH ? Thats forbidden :
To prevent the possible propagation of confederation path segments outside of a confederation, the path segment types AS_CONFED_SEQUENCE and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH attribute. RFC 4893.
The RFC does not suggest how to handle AS4_PATH violations, but if the bad path is learned on every upstream, this will cause a network with obgpd edges to disconnect from the internet…. Modifying the OpenBGPd software to permit AS_CONFED_SEQUENCE, AS_CONFED_SET in an as4_path causes the path to be accepted and the session is not torn down. This isn’t a great fix.
The impact today is fairly limited as there are relatively few bgp speakers honouring the 4-byte ASN protocol extension rules, but as code that support these features creeps around the internet, the next time this happens the impact could be much greater, so we need to understand which implementation of which BGP software caused this illegal origination.
From a software point of view, I want to see a configurable option to reject the route but keep the session, reject the route and drop the session, accept the route but log/send trap, etc.
In any case we need to publish the arrangement that has led to this mistake so that other networks using the same toolset to originate prefixes can avoid the same situation happening. I have made contact with an engineer at the NOC who are investigating.
Yesterday I gave a talk to Sheffield GeekUp on preparing enterprises for IPv6 [download]. The premise of the talk was :
The advice I gave was :
My hope is that this talk is improved upon and delivered internationally to enterprises.
These are the slides that I presented at NANOG44 in Los Angeles on Sunday, “VoIP For Network Operators“.
This talk was for network operators looking to build voice segments of their network, and the slides cover
In between browsing Facebook and Youtube, the UK economy generates $1,930,000,000 of output a year. Thats $550,000 every two and a half hours. Well if today had been a work day, there’d have been one two and a half hour period where that was much higher. That’s because in a pique of routing excitement, Pakistan Telecom managed to hide Youtube from most of the internet for that length of time.
Pakistan Telecom and Youtube are likely to have no commercial relationship in place to carry Youtube traffic – particularly as around two hours ago, according to Yahoo News, the story broke that the Pakistan Government required ISPs operating in the country to block Youtube. Despite this, Pakistan Telecom were able to cause ISPs all over the world to send traffic that should be destined for Youtube to Pakistan instead.
This is because the protocol that determines how to find my network on the internet, is shaped by how “specific” the announcement of my network is. If I make an announcement of a network of 1,024 addresses, and someone else makes a second announcement of 256 addresses within a subset of my 1,024, then the network which announces the smaller subset win the traffic destined to those hosts. This is a feature – fully by design – of the BGP routing protocol. Almost every time a more specific block of addresses is announced, this is because the administrators of those networks intend for the routing to be different for a subset of a large number of addresses.
Sadly, there are accidents from time to time – another network can announce a subset of my addresses without my knowledge or permission, and they win the traffic that should have gone to me. This happened today – it seems that Pakistan Telecom decided to inject a fake route to their network containing Youtube’s webservers, and accidently then leaked that route to the networks they connect to.
Small networks and end sites can limit the chances that they will leak bad routes by explicitly listing the network addresses that they intend to send to their upstream or peered networks. Larger networks may find it harder to stop themselves propagating someone else’s mistake, because they may have a contract to carry forward any announcement that their customers make. Furthermore, the complexities of their own networks mean that an engineer working under pressure after announcements made by government ministers are more likely to make a typo error and do the wrong thing.
Richard Clayton presented a very interesting set of commentaries at the last LINX meeting. He commented that right now its very obvious indeed when someone hijacks some of my network space in this way, because all of my traffic disappears. Youtube were probably aware that something was very wrong within moments of the announcement. What if someone builds an infrastructure to steal my traffic – or at least some of my traffic – but after doing something with it, they send it back to me, it is much harder for me to spot that anything is wrong.
This is a significant risk to ecommerce infrastructures that competitors or e-pirates could seize upon opportunities to steal customer behaviour data. What if a wizard stole the network containing your web server, proxied your shop, but set up a fake checkout? How quickly would you spot?
Because this problem is inherent to the routing protocol, this is the obvious place to fix it. There are attempts to blend PKI with routing information, so that peers can verify the validity of your announcements. S/BGP (secure BGP) requires me to sign my announcements, and gives my peers a method to check in an impartial internet community database that my announcement is valid. It is the sort of technology that would have prevented Youtube from disappearing off the air today.
I presented a talk on recent European Internet exchange news [download] with Mike Hughes from the LINX last week at UKNOF. Many of the attendees run networks that do not peer publicly, so it was a pleasure to explain the impact that European IXPs have on member traffic. We also then gave a perspective on peering in London.
Many of the statistics came from Serge at Euro-IX who did the leg work for the raw figures.
The highlight points of the talk were
There’s other stuff in the slides too, such as the usual traffic updates for various major exchanges in Europe.
I come from an IP engineering background, and now work in a telecoms role with Localphone.com. Huge amounts of crossover exist between the two disciplines, especially now that inter-company telecommunications interconnections are now regularly made over IP, but much of what someone will learn about peering in the voice world will not be mellifluous to someone with a background in IP peering.
I attended a PulverMedia conference on voice peering last week, with some preconceptions about what I imagined voice peering to be. These are some things I learned after talking to people at the conference. Gary Kim gave one of the most useful insights when he complained that he was, “More confused about where peering is going today than he was two years ago.”
“Why can’t I configure a voice peer like I can configure a BGP peer?” is a typical question. The answer is simple. When you peer using IP, the protocol is well known and established, prefixes are in a ubiquitous standard, peering is typically settlement free (and when its not, pricing is transparent and easy to calculate as mostly all traffic is equal – from a billing perspective).
The sad dichotomy is that in the voice world, prefixes (telephone numbers) behaviour is not identical, the protocols different companies use will be different (media codecs, call signalling, dtmf), and thanks to the regulators and history of commercial telephony peering is hardly ever settlement free.
This complexity has led to the emergence of another traditional pattern in telecoms – a barrier to entry. Clearing houses who will abstract peers from each other. They mediate media codecs, signaling differences, and perform CDR mediation. A barrier to entry, because they don’t want to do this for free. This is a model which is not great for many telcos who quite rightly don’t want to yield control of their outbound dialplans to any third party. Abstracting my media might mean callers get lower quality calls, and leave me as a service provider with poor visibility of the route that a call between two parties takes. Abstracting signaling without me being aware means that error messages about calls are lost in translation.
The clearing house model is also a natural monopoly. If a company is a member of one clearing house, and I am a member of another, then there is no way for us to peer using the traditional clearing-house model. Some clearing houses have suggested a protocol that would permit clearing houses to peer (share their registry data) – effectively increasing the reach – but this potentially further increases the layers of abstraction between me as a service provider and a peer.
Before I explain what I think the answer is, let me explain a few of the reasons why peering between telephony companies is good. Peering between competitive telecoms companies reduces their dependency on national incumbent providers. Two large non-incumbent telcos can peer, possibly meaning that TDM legs are removed from a call which is IP at both ends resulting in better quality calls, possibly permitting the use of new ultra-clear wideband-audio codecs, and typically at cheaper rates than connecting through an incumbent party. This gives telecoms customers cheap calls at a higher quality. As a result this is an important strategy for next-gen telcos.
Telecoms companies need to communicate their prefixes and standards bilaterally – that is to say, without a clearing house. I am working with some of the people I met at the conference on a new Internet-Draft to suggest the protocol that would facilitate this (like TRIP, but with enough understanding of commercial logic in the protocol to make it useful). I’m hoping to publish the first draft later this month.