This happened following an assignment of around half a million addresses to support the users at the Chinanet Fujian Province Network.

The pool of available addresses to the region including some of the world’s largest populations, such as China, India, Indonesia, and some of the world’s largest economies, such as Japan and Australia, has depleted to such low levels, that the registry responsible for distribution of these addresses will now ration them, such that any ISP requesting space will be given a single block of 1,024 addresses, on a single occasion only.

This is enough space to allow the ISP only to host NAT or ipv4 to ipv6 translation technologies.  It is not enough to address a large content infrastructure, hosting environment, or internet access customer-base.

The rules of the game have today changed for 50% of the world’s population, and they will change in Europe too in a few short months too.  If you do not have an IPv6 plan, then this is your new significant business risk – how will users with v6 only connections reach your content?  And if this is through a translation mechanism, how will you ensure quality, or that your end-to-end protocols (like voice, video, etc.) will work ?

Get in touch to continue the conversation!

According to the RIPE RIS service, the links between MWEB (AS10474) to Telkom South Africa (AS5713) were disconnected on the November 2nd – Telkom being the original transit provider that MWEB used.

MWEB have detected that congestion reduces, therefore service levels increase when traffic bypasses the incumbent and is delivered directly to other ISPs in their region via peering links.  If a network refuses to peer, MWEB simply deliver the traffic to local providers via their international links – possibly just as congested, but available at a fraction of a cost.  If traffic is then delivered to the incumbents via links they themselves pay for, the incumbents also have a financial incentive to peer.

Peering is the best way to encourage enormous capacities between ISPs and other networks, because a direct one-to-one connection can be monitored and well managed in order to guarantee availability for internet traffic.  Peering therefore increases available bandwidth and reduces bandwidth costs.  This will enable high the sort of services that require high-bandwidth availability, like streaming media and high definition video conferencing.

Interestingly, thirty minutes after the adjacency with Telkom was severed, it appeared that MWEB picked up a new transit customer – Yebo, AS12258, with Yebo’s prefixes being advertised to Interoute (again, according to RIPE RIS).  The commercial nature of this downstream relationship is, however, not revealed by the routing table.

The incumbent is perfectly entitled to – and well placed to – sell excellent transit links into the local market, but their strategy to do that, as I explained in my last article, must be to make the transit product in their key regions excellent – this means to peer with the key other local providers (not all providers) in the market, and to ensure that capacities across their backbone and to customers are well managed and available for traffic.

It demonstrates clearly that deregulated markets offer enormous advantages over controlled ones, and should serve well as a reminder to operators and policy makers that simply getting out of the way could be the best way to further their aims for industry in any given region.  This is mainly because:

• Allowing networks to interconnect freely (calculated as number of active ASNs in a region), and the size of the market (calculated from the pool of announced ip addresses in a region) are strongly correlated (slide 8).  My guess is that more organisations get online, because competition leads to price falling, whilst the versatility and relevance of services offered increases.
• When there are a larger number of networks in a region, the global carriers have a greater incentive (more customers!) to run diverse connectivity into the region.  This leads to a huge advantage to firms in a region, their connectivity carries on despite local major fibre breaks. (slide 25, 36)
• Content moves out of the US/Western Europe and into the local market place, creating opportunity (and jobs) for local players, and improving the performance of services for local users.

Incumbent networks in this region have a huge opportunity to grow revenues, as the market expands, as long as they are willing to interconnect widely in this region.  As the number of providers in a region expands, customers will be able to (and, according to this research, actually do) pick between innovative and disruptive new providers with excellent regional (via peering), and international (via transit) capacities.  Peering also makes capacity cheap, because traffic can stay local to the ISP.  An incumbent provider that refuses to peer in order to retain market share will not be able to compete in quality terms with the new providers.  Defending a 100% market share is impossible in a competitive market, so the strategy must change, the aim must become enjoying the fruits of a booming market instead of monopoly.  As the Renesys slides say, there is no dominent IXP in this region yet, with many networks dragging traffic to London, Amsterdam or Frankfurt to exchange, but this will change as the density of providers in the Middle East reaches a critical mass.

Setting up an Internet Exchange point is simple from a technology point of view, but requires significant planning, and community support for the plans.  Read the slides to find out more about what must be planned.

Download:  [Slides + Notes (recommended)] ~ [Slides alone]

View directly from Slideshare (requires flash):

A route-server is a fantastic way for networks to start to peer (swap internet traffic) at Internet Exchanges, and results in instant success after connection.  A network with an open peering policy can connect to the internet exchange, and then get peering with more than half of all the other networks on the exchange by bringing up a single pair of BGP sessions.

When a route-server peering is established, a BGP session is setup between your router and LONAP’s route database.  LONAP advertise all of the prefixes of the other connected members to you, but the traffic between you and the other members flows between you and your peer directly (it does not need to traverse the route-server.)  Members do not need to open their network to their own customers at the route servers, they can send special messages to the route-servers to prevent certain networks from seeing prefixes.

Route-servers are not new, but have had a bad reputation for stability for several years.  With our colleagues at several other community exchanges, including the LINX, we shared bugs, workarounds, and feature requirements with each other and the main open-source route-server vendors.  Eventually, we were able to report considerable improvement in stability last December.  As a result, we at LONAP selected BIRD and OpenBGPd as our route server vendors, and built a support framework to link our configuration with the LONAP configuration system.

Since then we have been advocating the route-servers to our members, and the fact that they are now providing a stable stepping-stone to more than half of our peers shows that this effort was worthwhile.  If you would like to start to peer, but need to be assured of instant success and results, then contact Andy for information about how the route-servers at LONAP can help.

The slides show that the new route-servers perform splendidly well compared with traditional Quagga based MLPs, also that route-servers are now free of ‘first generation code’ bugs, and also that they handle your prefixes transparently – as you would expect.

Interestingly, BIRD and OpenBGPd behave identically ‘on the wire’ so IXPs are encouraged to use multi-vendor MLP on their platform for increased reliability and stability.  The new breed of route-server code is dependable and tested, so networks that would like to connect should draw confidence from this testing, and IXPs wishing to roll out MLP services should feel confident in the software tested.

Happy peering!

The messages are clear and simple.  Working now to get ready for the IPv6 transition will be less expensive and lower risk than waiting for IPv4 starvation to hurt.  I interviewed some key enterprises about their specific grumbles but the great news is that most are transitional and already people are working on fixing them.

They all show that the effects of scarsity of IP addresses will be felt before the final few addresses become assigned to end users.  All consumers of addresses will feel constrained, which means all businesses trading online, whether they are a traditional ISP, a growing e-commerce shop, or a datacentre/hosting firm.  The policies under consideration are :

• 2008-06 – Both new and existing organisations requesting IPV4 addresses shall be given addresses only to support transitioning technologies (i.e. infrastructure services which enable access to IPv6 addressed resources.)  They will only be given one block of IPv4 addresses, and this shall be the smallest possible range of addresses as decided by the community at large.  This is the only range of addresses which shall be given to the end user, even if their needs justify more.
• 2009-03 – From summer 2010, any organisation shall only be given enough new addresses to cater for anticipated need for nine months.  By summer 2011, this will change to three months.  Organisations (registries and end users) MUST have used more than half of their address space within six months of assignment or allocation.
• 2009-04 – Organisations must demonstrate that they are implementing an IPv6 transition policy (RFC5211) in order to be given IPv4 addresses.  Allocations from the RIPE NCC will be /27 (32 addresses for the entire registry, e.g. ISP)

This also means that the effects of black market trading in address space will be seen before the ‘anticipated’ IPv4 dry date in 2011.  There are no magic bullets, you (and your customers, suppliers, and partners) need deep pockets, decades worth of existing resources, or an ipv6 transition plan.  The only sensible option is to plan your v6 deployment today.

I decided to put the timetable to the test.  On Wednesday at 2:30PM uk time, I applied for a /32.  One hour later, we were allocated 2a02:c30::/32.  I straight away assigned a /48 for our network infrastruture, and another for our production hosting lan, another for our development hosting lan.  From these /48s, several /64s were reserved, one for router loopbacks, another for point to point links, more for individual hosting applications.  An hour later, this was implemented on our network – routers had loopbacks, and a v6 IGP was up and running, and working.  I filed a ticket with our upstreams, and the first announcement was turned up minutes later – check BGPlay for exact times.  Around 2 hours after making our application to RIPE, we were participants on the IPv6 internet.

Now this is not a front-to-back implementation, but in just two hours we had something to hand over to our systems teams for testing and training.  If you rely on the internet for your business, this is the stage you need to get to urgently.  In fact, by close of play Wednesday, some of our simpler services were already running dual stack, and additionally we are now running Dual-stack for DNS – Nominet having the simplest method for adding ipv6 glue.

Full disclosure: In reality our v6 rollout started months ago, by monitoring advice on operational mailing lists, attending v6 seminars, and in fact we had been engaged in the rollout of ipv6 on several customer networks to date, so this rollout was not frightening for us.

We are now in the position that we can integrate IPv6 addressing as part of every configuration refresh or maintenance on our services, so that v6 is rolled out in a controlled, monitored, and careful manner.  By moving now, we have bought ourselves time – a luxury, and firms waiting longer to start their v6 rollout will have a harder time, with the whole migration feeling like a ‘y2k bug’.

Some history in very brief terms – all networks on the internet that participate in BGP need a number to identify themselves.  On the public internet, this number normally needs to be globally unique.  The number can be between 1 and 65,535, and we have close to 50,000 of these numbers in use.  To grow past this number, the BGP standard needs to be modified.  The modification is described in a document called rfc4893, and this document was accepted by the community last May.

The first incarnations of router software that support these large AS numbers is now circulating. Due to flaws in the standards that exist in January 2009, if you install one, you may become disconnected from the internet.

Why? Some more background, first: BGP allows for large networks to configure ‘hints’ in their router configuration, by dividing their network into several small networks (confederations).  The information about the ‘virtual’ divisions of the network should be removed from the BGP messages which are sent to other networks, but if a network supports large ASN in some parts, and not in others, the routers in the legacy part of the network may not know to test the ‘large number’ section of a BGP message for the presence of an internal confederation ID.  The standard tries to take this into account by explicitly forbidding that confederation ID be passed between networks in the asn4 part of the BGP message.

However, should this occur by accident, what are the effects?  Well, elsewhere in the Large ASN standard, it states that the connection between two networks should be severed if Confederation ASN appear to be leaked in the ASN4 part of the BGP message.  This means that networks which do not understand large ASN can forward a broken message to a network which does understand large ASN.  At which point the network which does understand large ASN should tear down the session.

Since this message can be delivered over a transit session, this means the receiving ISP loses their connection to the internet via that ISP.  If it learns the router over every ISP, then the network can lose its connection to the internet entirely.

The message that I reported was leaking in December is still leaking.  AS196629 (AS3.21 in legacy asdot notation) is announcing to AS35320, who are not stripping their confederation information from the large-asn section of BGP messages.  If you learn the prefix via AS196629′s other transit, AS6886, then you are fine.  If you learn the prefix via AS35320, you are (today) receiving a broken message.

We tested out how Cisco IOS is coping with this broken message using the first generation of code for the cisco 7200 router that understands ASN4.  We peered the router to NetSumo‘s research and development network, AS15653.  Cisco honours the standard/rfc, and breaks the session.  Since it learns the dirty message on the transit session, the router disconnected our test network from the internet entirely :

• *Jan 16 11:29:58.531: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Up
• *Jan 16 11:30:02.595: %BGP-6-ASPATH: Invalid AS path (65044 65048 65062) 3.21 23456 received from 193.239.32.2: Confederation found in AS4_PATH
• *Jan 16 11:30:02.595: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Down BGP Notification sent
• *Jan 16 11:30:02.595: %BGP-3-NOTIFICATION: sent to neighbor 193.239.32.2 3/1 (update malformed) 27 bytes E0111803 030000FE 140000FE 180000FE 26 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0050 0200 0000 3540 0101 0240 020C 0205 3D25 2114 89F8 5BA0 5BA0 4003 04C1 EF20 02E0 1118 0303 0000 FE14 0000 FE18 0000 FE26 0202 0003 0015 0000 5BA0 175B CFDA

If you work in this field, I implore you to read the more thorough analysis on the nanog list, and participate in the discussion to work out how we should correct the standard, to allow routers to behave differently when a dirty message is received.  If we do not, then there is a simple, easy to understand, and easy to implement mechanism to break the internet, as soon as networks upgrade to the current version of their routing software.