The slides show that the new route-servers perform splendidly well compared with traditional Quagga based MLPs, also that route-servers are now free of ‘first generation code’ bugs, and also that they handle your prefixes transparently – as you would expect.

Interestingly, BIRD and OpenBGPd behave identically ‘on the wire’ so IXPs are encouraged to use multi-vendor MLP on their platform for increased reliability and stability.  The new breed of route-server code is dependable and tested, so networks that would like to connect should draw confidence from this testing, and IXPs wishing to roll out MLP services should feel confident in the software tested.

Happy peering!

The messages are clear and simple.  Working now to get ready for the IPv6 transition will be less expensive and lower risk than waiting for IPv4 starvation to hurt.  I interviewed some key enterprises about their specific grumbles but the great news is that most are transitional and already people are working on fixing them.

Some history in very brief terms – all networks on the internet that participate in BGP need a number to identify themselves.  On the public internet, this number normally needs to be globally unique.  The number can be between 1 and 65,535, and we have close to 50,000 of these numbers in use.  To grow past this number, the BGP standard needs to be modified.  The modification is described in a document called rfc4893, and this document was accepted by the community last May.

The first incarnations of router software that support these large AS numbers is now circulating. Due to flaws in the standards that exist in January 2009, if you install one, you may become disconnected from the internet.

Why? Some more background, first: BGP allows for large networks to configure ‘hints’ in their router configuration, by dividing their network into several small networks (confederations).  The information about the ‘virtual’ divisions of the network should be removed from the BGP messages which are sent to other networks, but if a network supports large ASN in some parts, and not in others, the routers in the legacy part of the network may not know to test the ‘large number’ section of a BGP message for the presence of an internal confederation ID.  The standard tries to take this into account by explicitly forbidding that confederation ID be passed between networks in the asn4 part of the BGP message.

However, should this occur by accident, what are the effects?  Well, elsewhere in the Large ASN standard, it states that the connection between two networks should be severed if Confederation ASN appear to be leaked in the ASN4 part of the BGP message.  This means that networks which do not understand large ASN can forward a broken message to a network which does understand large ASN.  At which point the network which does understand large ASN should tear down the session.

Since this message can be delivered over a transit session, this means the receiving ISP loses their connection to the internet via that ISP.  If it learns the router over every ISP, then the network can lose its connection to the internet entirely.

The message that I reported was leaking in December is still leaking.  AS196629 (AS3.21 in legacy asdot notation) is announcing to AS35320, who are not stripping their confederation information from the large-asn section of BGP messages.  If you learn the prefix via AS196629’s other transit, AS6886, then you are fine.  If you learn the prefix via AS35320, you are (today) receiving a broken message.

We tested out how Cisco IOS is coping with this broken message using the first generation of code for the cisco 7200 router that understands ASN4.  We peered the router to NetSumo’s research and development network, AS15653.  Cisco honours the standard/rfc, and breaks the session.  Since it learns the dirty message on the transit session, the router disconnected our test network from the internet entirely :

• *Jan 16 11:29:58.531: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Up
• *Jan 16 11:30:02.595: %BGP-6-ASPATH: Invalid AS path (65044 65048 65062) 3.21 23456 received from 193.239.32.2: Confederation found in AS4_PATH
• *Jan 16 11:30:02.595: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Down BGP Notification sent
• *Jan 16 11:30:02.595: %BGP-3-NOTIFICATION: sent to neighbor 193.239.32.2 3/1 (update malformed) 27 bytes E0111803 030000FE 140000FE 180000FE 26 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0050 0200 0000 3540 0101 0240 020C 0205 3D25 2114 89F8 5BA0 5BA0 4003 04C1 EF20 02E0 1118 0303 0000 FE14 0000 FE18 0000 FE26 0202 0003 0015 0000 5BA0 175B CFDA

If you work in this field, I implore you to read the more thorough analysis on the nanog list, and participate in the discussion to work out how we should correct the standard, to allow routers to behave differently when a dirty message is received.  If we do not, then there is a simple, easy to understand, and easy to implement mechanism to break the internet, as soon as networks upgrade to the current version of their routing software.

Lee’s premise is that “Certainly Skype is not a walled garden. All things being relative, it’s certainly not overly closed either.”  Lee claims that the accusations of closeness are unfair, because they are levied by commentators who advocate SIP based addressing and dialing rather than any other system.

This is not my premise.  I claim that Skype is closed because calls are signalled and completed using protocols that are entirely secret as a matter of policy.  Skype’s founder presented at Spring VON 2007 and stated that if Skype did not keep their protocols entirely secret, then Skype would be full of spam and attack like email is.  I think this is a poisonous claim, telephone networks have been interconnecting around the world since telephony was conceived.  By not allowing telecoms firms to interconnect between the skype namespace and other networks, Skype have prevented openness to develop and maintain a monopoly position. That’s perfectly acceptable business, but it is not in the slightest bit open.

Randy Bush googled Walled Garden for a recent presentation and found this cartoon.  I like this definition because it’s correct.  Is Skype a Walled Garden ?  Lee says a Walled Garden is a commercial restriction, for example, “sharing of ringtones via Bluetooth, using WiFi from a PDA, having access to all Web sites“.  I think that only allowing interconnection with the purchase of an upgrade like SkypeOut is a restrictive or practice that suggests Skype is a Walled Garden.  Worst of all a call between two VoIP networks using this method requires default PSTN routing, which harms signal quality, and prevents the expansion of next-generation services such as Wideband/High Definition audio.

The meshing of networks, whether they are traditional voice or IP networks, leads to higher audio quality and increased reliability.  Keeping telephony systems and protocols secret in order to prevent meshing may well be a viable business model, but it is not an open business model.

AS196629 (3.21 in asdot) announce 91.207.218.0/23.  Experienced eyes will notice that this is quite a large as number.  It’s a ‘new’ 4-byte ASN.  When an OpenBGPd speaker with 4-Byte ASN support receives the update for this message, the session is torn down with the daemon logging a ‘fatal error’. Why?
OpenBGPd is checking AS4_PATH to ensure that it contains only AS_SET and AS_SEQUENCE types, as per RFC4893.  When processing the UPDATE for 91.207.218.0/23 it sees :

91.207.218.0/23
Path Attributes – Origin: Incomplete
Flags: 0×40 (Well-known, Transitive, Complete)
Origin: Incomplete (2)
AS_PATH: xx xx 35320 23456 (13 bytes)
AS4_PATH: (65044 65057) 196629 (7 bytes)

See the confederation ASNs in the AS4_PATH ?  Thats forbidden :

To prevent the possible propagation of confederation path segments outside of a confederation, the path segment types AS_CONFED_SEQUENCE and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH attribute. RFC 4893.

The RFC does not suggest how to handle AS4_PATH violations, but if the bad path is learned on every upstream, this will cause a network with obgpd edges to disconnect from the internet…. Modifying the OpenBGPd software to permit AS_CONFED_SEQUENCE, AS_CONFED_SET in an as4_path causes the path to be accepted and the session is not torn down.  This isn’t a great fix.
The impact today is fairly limited as there are relatively few bgp speakers honouring the 4-byte ASN protocol extension rules, but as code that support these features creeps around the internet, the next time this happens the impact could be much greater, so we need to understand which implementation of which BGP software caused this illegal origination.

From a software point of view, I want to see a configurable option to reject the route but keep the session, reject the route and drop the session, accept the route but log/send trap, etc.

In any case we need to publish the arrangement that has led to this mistake so that other networks using the same toolset to originate prefixes can avoid the same situation happening.  I have made contact with an engineer at the NOC who are investigating.

• IPv4 addresses are scarse, and at current consumption rates, the IANA pool of free v4 addresses will be gone at the start of 2011.
• This starts a “Post IPv4 world” where the IPv4 internet continues to function as before (certainly initially), but obtaining new addresses becomes harder and expensive.  This inhibits expansion of existing firms, and new entrants to the market.
• Address trading is likely to lead to a larger routing table, meaning that failure-recovery times increase, and the risk of blackholes on the internet increases.
• Large broadband providers may not have enough v4 addresses to give one address per customer.  This means protocol translation techniques need to be used, which break the end to end model.  We rely on the end to end model when innovating new services on the internet.
• If services and consumers gradually roll v4 and v6 (dual stack), the negative impact of markets for addresses, routing problems, and translation can be mitigated.
• Service providers are enabling v6 in the core.  Enterprises need to move next in order to get the world v6 ready.

The advice I gave was :

• Today’s market leaders are already learning v6 lessons in their labs, (e.g. ipv6.google.com).  They are doing this to help them retain market leadership.  If you want to retain your market position, start labbing your applications and service provision with v6.
• Write a policy stating all new purchases of infrastructure and services need to be from providers with v6 support, or a well defined v6 road map.  In other words, make v6 a “life cycle upgrade”.
• Share information, and learn information from your industry peers.
• I also listed some advice to developers with regard to v4 and v6 differences.
• I then delivered a very quick primer to those who have not seen v6 deployed before.

My hope is that this talk is improved upon and delivered internationally to enterprises.

This talk was for network operators looking to build voice segments of their network, and the slides cover

• Voice Basics for SPs
• Why Operators should care
• Voice Peering
• Metrics
• VoIP Security

Pakistan Telecom and Youtube are likely to have no commercial relationship in place to carry Youtube traffic – particularly as around two hours ago, according to Yahoo News, the story broke that the Pakistan Government required ISPs operating in the country to block Youtube. Despite this, Pakistan Telecom were able to cause ISPs all over the world to send traffic that should be destined for Youtube to Pakistan instead.

This is because the protocol that determines how to find my network on the internet, is shaped by how “specific” the announcement of my network is. If I make an announcement of a network of 1,024 addresses, and someone else makes a second announcement of 256 addresses within a subset of my 1,024, then the network which announces the smaller subset win the traffic destined to those hosts. This is a feature – fully by design – of the BGP routing protocol. Almost every time a more specific block of addresses is announced, this is because the administrators of those networks intend for the routing to be different for a subset of a large number of addresses.

Sadly, there are accidents from time to time – another network can announce a subset of my addresses without my knowledge or permission, and they win the traffic that should have gone to me. This happened today – it seems that Pakistan Telecom decided to inject a fake route to their network containing Youtube’s webservers, and accidently then leaked that route to the networks they connect to.

Small networks and end sites can limit the chances that they will leak bad routes by explicitly listing the network addresses that they intend to send to their upstream or peered networks. Larger networks may find it harder to stop themselves propagating someone else’s mistake, because they may have a contract to carry forward any announcement that their customers make. Furthermore, the complexities of their own networks mean that an engineer working under pressure after announcements made by government ministers are more likely to make a typo error and do the wrong thing.

Richard Clayton presented a very interesting set of commentaries at the last LINX meeting. He commented that right now its very obvious indeed when someone hijacks some of my network space in this way, because all of my traffic disappears. Youtube were probably aware that something was very wrong within moments of the announcement. What if someone builds an infrastructure to steal my traffic – or at least some of my traffic – but after doing something with it, they send it back to me, it is much harder for me to spot that anything is wrong.

This is a significant risk to ecommerce infrastructures that competitors or e-pirates could seize upon opportunities to steal customer behaviour data. What if a wizard stole the network containing your web server, proxied your shop, but set up a fake checkout? How quickly would you spot?

Because this problem is inherent to the routing protocol, this is the obvious place to fix it. There are attempts to blend PKI with routing information, so that peers can verify the validity of your announcements. S/BGP (secure BGP) requires me to sign my announcements, and gives my peers a method to check in an impartial internet community database that my announcement is valid. It is the sort of technology that would have prevented Youtube from disappearing off the air today.

Many of the statistics came from Serge at Euro-IX who did the leg work for the raw figures.

The highlight points of the talk were

• Euro-IX identify 103 exchanges in Europe, in 31 countries. (3 in 1993)
• 8 Exchanges in the UK (was 9 until BT’s UK6x closed)
• At the end of 2007 networks publicly peered 1.215Tbit/sec at peak.
• More public peering in EU than US (but it’s cheaper to peer in EU thanks to lower x-connect fees, and cheap ubiquitous mutual exchanges)
• London is #1 for network reach – 601 networks peer publicly, 415 peer exclusively in the UK.
• 22% of LINX members peer exclusively at LINX, 31% of LONAP members peer exclusively at LONAP.
• 577 networks peer at more than one IXP, and one network (Colt) is present at 19 exchanges!
• Last year the good weather in April caused an additional summer-time traffic dips in Europe, in addition to the regular dip in July/August

There’s other stuff in the slides too, such as the usual traffic updates for various major exchanges in Europe.

I attended a PulverMedia conference on voice peering last week, with some preconceptions about what I imagined voice peering to be. These are some things I learned after talking to people at the conference. Gary Kim gave one of the most useful insights when he complained that he was, “More confused about where peering is going today than he was two years ago.”

“Why can’t I configure a voice peer like I can configure a BGP peer?” is a typical question. The answer is simple. When you peer using IP, the protocol is well known and established, prefixes are in a ubiquitous standard, peering is typically settlement free (and when its not, pricing is transparent and easy to calculate as mostly all traffic is equal – from a billing perspective).

The sad dichotomy is that in the voice world, prefixes (telephone numbers) behaviour is not identical, the protocols different companies use will be different (media codecs, call signalling, dtmf), and thanks to the regulators and history of commercial telephony peering is hardly ever settlement free.

This complexity has led to the emergence of another traditional pattern in telecoms – a barrier to entry. Clearing houses who will abstract peers from each other. They mediate media codecs, signaling differences, and perform CDR mediation. A barrier to entry, because they don’t want to do this for free. This is a model which is not great for many telcos who quite rightly don’t want to yield control of their outbound dialplans to any third party. Abstracting my media might mean callers get lower quality calls, and leave me as a service provider with poor visibility of the route that a call between two parties takes. Abstracting signaling without me being aware means that error messages about calls are lost in translation.

The clearing house model is also a natural monopoly. If a company is a member of one clearing house, and I am a member of another, then there is no way for us to peer using the traditional clearing-house model. Some clearing houses have suggested a protocol that would permit clearing houses to peer (share their registry data) – effectively increasing the reach – but this potentially further increases the layers of abstraction between me as a service provider and a peer.

Before I explain what I think the answer is, let me explain a few of the reasons why peering between telephony companies is good. Peering between competitive telecoms companies reduces their dependency on national incumbent providers. Two large non-incumbent telcos can peer, possibly meaning that TDM legs are removed from a call which is IP at both ends resulting in better quality calls, possibly permitting the use of new ultra-clear wideband-audio codecs, and typically at cheaper rates than connecting through an incumbent party. This gives telecoms customers cheap calls at a higher quality. As a result this is an important strategy for next-gen telcos.

Telecoms companies need to communicate their prefixes and standards bilaterally – that is to say, without a clearing house. I am working with some of the people I met at the conference on a new Internet-Draft to suggest the protocol that would facilitate this (like TRIP, but with enough understanding of commercial logic in the protocol to make it useful). I’m hoping to publish the first draft later this month.