Sign In

Internet Performance Delivered right to your inbox

Global Impacts of Recent Leaks

Recent routing leaks remind us why monitoring Internet routing and performance is important and requires effective tools.  Routing leaks are the ‘benign cousin’ of the malicious BGP route hijack.  They happen accidentally, but the result is the same: traffic to affected prefixes is redirected, lost, or intercepted.  And if they happen to you, your online business and brand suffers.

In this blog, we look at examples of a full-table peer leak, an origination leak, and a small peer leak and what happens to traffic when these incidents occur.  As we will see, some events can go on for years, undetected and hence, unremediated, but extremely impactful never the less.  As you read this blog, keep the following  questions in mind.  Would  you know if the events described here were happening to you?  Would you know how to identify the culprit if you did?

 

iTel/Peer1 routing leak

Starting on 10 October at 10:54 UTC, iTel (AS16696) leaked a full routing table (555,010 routes) to Peer 1 (AS13768).  Normally, iTel exports 49 routes to Peer 1;  however, over the course of several minutes, it leaked 436,776 routes from Hurricane Electric (AS6939) and 229,537 routes from Shaw Communications (AS6327), two of iTel’s transit providers.  Peer 1 then sent these routes on to its substantial peering base, resulting in a brief but widespread disruption in global routing.

Shortly afterwards, Peer 1 announced to its customers that it had suffered a widespread network outage.  While it is difficult to ascertain whether the routing leak was a cause of the network outage or a product of it, whenever a full routing table is leaked — especially by a highly-peered network like Peer 1 — the impacts can be far reaching.

Many thousands of our continuous global traceroute measurements were redirected into iTel through Peer 1 (visible in green in the graph below).


16696
Some of the more prominent companies impacted include Microsoft (AS8075) and Netflix (AS2906).  The following illustrates the propagation profile of two Microsoft routes around the time of the leak.  The leak manifests itself as a Hurricane Electric bulge, as more peers begin accepting leaked routes to Microsoft that go through Peer 1, iTel and then Hurricane Electric before reaching Microsoft.  In each case, nearly the entire Internet elected to send traffic destined for these address ranges through the leaked routes.

104.44.69.0_24_1444474800 65.54.215.0_24_1444474800

Interestingly, routes to Netflix (AS2906) behaved quite differently.  As the leaked routes propagated, they were subsequently withdrawn, suggesting some sort of automated leak detection and remediation process attempting to counter such events.

23.246.14.0_24_1444474500 23.246.15.0_24_1444474500

As an example of a more distant impact, this leak amazingly resulted in Shaw Communications of Canada becoming a major upstream of China Telecom for a number of routes for almost an hour.  As we’ve explained in earlier blogs, routing leaks like these can often have widespread global impacts.  The following two graphs show how our routing peers reached China Telecom for 58.216.9.0/24 and 180.97.156.0/22 over time.  Shaw Communications (AS6327, light blue) isn’t normally in the path for these routes (for any provider other than Shaw or its customers) and only made an appearance during the leak.

58.216.9.0_24_1444473900 180.97.156.0_22_1444473900

Consider the following traceroute, originating in Tanzania and directed toward China, about 9,000km away as the crow flies.  We would not expect the path to traverse Canada, of all places, but in this example that’s exactly what happened.  First, Tata (AS6453) takes the traffic from Dar es Salaam, though Paris, London, New York, Chicago and Seattle before arriving at iTel Networks in Vancouver.  Then the path continues on to Los Angeles and finally to China by way of Taipei, Taiwan.  The nearly 500ms required here is just under that seen for geosynchronous satellite communications, satellites which orbit over 42,000 km above earth’s surface!


trace from Dar es Salaam, TZ to 218.87.160.1 (China Telecom) at 11:05 Oct 10, 2015
1  *
2  41.79.68.25       core.dar-es-salaam.aptus.co.tz                    0.885
3  41.206.162.9      if-0-0-1-110.core1.41B-Dar-Es-Salaam.as6453.net   0.610
4  80.231.165.5      Pos-channel3.core1.WYN-Marseille.as6453.net     119.783
5  80.231.165.102    if-0-2-2-27.tcore1.WYN-Marseille.as6453.net     300.206
6  80.231.217.6      if-8-1600.tcore1.PYE-Paris.as6453.net           310.017
7  80.231.130.85     if-3-6.tcore1.L78-London.as6453.net             308.617
8  80.231.131.1      if-2-2.tcore2.L78-London.as6453.net             307.179
9  216.6.99.13       if-20-2.tcore2.NYY-New-York.as6453.net          300.803
10 216.6.99.46       if-12-6.tcore1.CT8-Chicago.as6453.net           304.046
11 64.86.79.1        if-22-2.tcore2.CT8-Chicago.as6453.net           290.148
12 64.86.124.33      if-15-0-0.core1.00S-Seattle.as6453.net          281.590
13 66.110.25.10      if-4-1-1.core3.VCW-Vancouver.as6453.net         287.499
14 *
15 199.192.104.1     iTel Networks Inc (Vancouver, Canada)           339.794
16 *
17 *
18 *
19 *
20 202.97.49.105     CHINANET backbone network (Los Angeles)         307.980
21 202.97.52.177     CHINANET backbone network (Taipei, TW)          385.910
22 202.97.91.9       CHINANET backbone network (Taipei, TW)          405.530
23 202.97.33.17      CHINANET backbone network (Shanghai, CN)        459.055
24 202.97.39.70      CHINANET backbone network (Shanghai, CN)        481.006
25 220.177.199.202   CHINANET jiangxi province network (Ji’an, CN)   480.762
26 218.87.160.1      CHINANET jiangxi province network (Ji’an, CN)   475.348

The impacts of the leak continue to reverberate.  Ten prefixes of Canadian provider ViaNet are still routed (at least in part) along the leaked AS Path as follows:

… 6453 13768 16696 16696 16696 16696 16696 16696 16696 16696 6939 5690 24.138.100.0/22

 

Freifunk Rheinland leak

The day after the iTel/Peer 1 leak, Freifunk Rheinland e.V. (AS201701) of Germany initiated an origination leak — one in which the guilty party masquerades as the origin of the route, unlike the previous incident which left the origin intact.  Just after 23:00 UTC on 11 October 2015, AS201701, which normally announces four prefixes, originated (leaked) 4,423 prefixes.  For the most part, the leak was brief in duration and the routes didn’t propagate very far across the Internet, so the impact was fairly minimal.

The following graphs show the portion of our peering base that observed AS201701 as the origin for two prefixes (shown in red).  The graph on left shows a Venezuelan prefix (201.210.0.0/16) normally originated by Venezuelan incumbent CANTV (AS8048) and minimally impacted by the leak.  On the right, we see a Vietnamese prefix originated by Netnam (AS24173) that was greatly affected.  Why drives the dramatic difference in route propagation in these two cases?

201.210.0.0_16 101.96.66.0_23

As we have described in previous blog posts about routing leaks such as the Indosat leak of 2014 and the Telekom Malaysia leak of 2015, there are prefixes in the global routing table that are heavily prepended to everyone and hence are at extreme risk in these situations.  For what is most certainly a very bad oversight, the prefix 101.96.66.0/23 is prepended six(!) times to all of its upstreams and is routed along the following AS Path.

… 6453 45903 24173 24173 24173 24173 24173 24173 24173

In these situations, the routes from the leaker will almost assuredly have a shorter AS Path and will always be selected when path length is the deciding criteria for BGP route selection.

 

Korea Telecom and Broadband Systems Corporation of Rwanda

Unlike the previous two examples, routing mishaps can also occur on a smaller scale.  Take for example the case of Broadband Systems Corporation of Rwanda.  For whatever reason, Broadband Systems Corporation (AS37288) peers with Korea Telecom (AS4766).  (How much traffic really flows between Rwanda and South Korea?)  However, Korea Telecom alternates how it routes the three prefixes it receives from Broadband Systems throughout the day.  It transits them for several hours, then it treats them as peer routes and doesn’t send them to its peers and transit providers.  This oscillation has been going since 14 September 2013 when 197.243.126.0/24 first came into existence.  Many important Rwandan government web resources appear to be hosted in this address space including the following.

197.243.48.67  mail.presidency.gov.rw
197.243.16.115 www.presidency.gov.rw
197.243.16.111 www.parliament.gov.rw
197.243.16.111 www.usa.embassy.gov.rw
197.243.16.111 www.russia.embassy.gov.rw
197.243.16.104 www.rdfcsc.mil.gov.rw

Next, we see a graph of the propagation profile of each of the three routes exported to Korea Telecom over the past two weeks.  In the case of 197.243.126.0/24, it is a more-specific of a /22 and only is in circulation when Korea Telecom is routing it — guaranteeing all Internet traffic to this address space will have to pass through South Korea before making its way into Rwanda. The other two routes are at least partially available through normal Africa transit paths when Korea Telecom is transiting them.

197.243.126.0_24_1443657600

41.74.160.0_20_1443657600 197.243.0.0_17_1443657600

The result is that for Internet users around the world, including those in East Africa, when these routes are being propagated, traffic to these sites is directed through South Korea before reaching Rwanda.  Consider the following traceroute from Kenya to Rwanda, via South Korea!   The straight line distance between the origin and destination is only 755 km, but these neighboring countries might as well be communicating via satellite, as the round trip times exceed 500ms.

trace from Nairobi, Kenya to 197.243.126.30 at 21:13 Oct 11, 2015
1  *
2  172.17.28.10     RFC 1918 (private use)                           1.192
3  196.201.222.25   Safaricom Kenya                                  8.094
4  196.201.222.206  Safaricom Kenya                                  7.558
5  195.219.174.1    if-0-0-1.core1.N71-Fujairah.as6453.net          57.756
6  195.219.174.14   if-7-0-2-0.tcore1.WYN-Marseille.as6453.net     160.797
7  80.231.217.6     if-8-1600.tcore1.PYE-Paris.as6453.net          259.128
8  80.231.154.86    Tata (Paris, FR)                               368.336
9  213.155.131.14   prs-bb2-link.telia.net (Paris)                 360.714
10 80.91.253.126    nyk-bb2-link.telia.net (New York)              343.712
11 62.115.112.191   chi-b21-link.telia.net (Chicago)               259.029
12 62.115.137.85    sea-b1-link.telia.net  (Seattle)               415.526
13 62.115.54.86     ktc-ic-310746-sea-b1.c.telia.net               307.934
14 112.174.87.125   Korea Telecom (Seongnam-si, KR)                428.873
15 112.174.84.33    Korea Telecom (Seongnam-si, KR)                428.804
16 112.174.84.226   Korea Telecom (Seongnam-si, KR)                430.371
17 121.189.3.118    Korea Telecom (Seoul, KR)                      521.208
18 197.243.126.30   Content Network (Kigali, Rwanda)               521.676

Quartz ran a story in August entitled “Why your internet connection is slow wherever you are in Africa” based largely on a presentation made by Dyn Chief Scientist Jim Cowie at Africa DNS Forum.  The story makes the case that lack of local content hosting, and not constrained international links, is the leading cause of slow Internet experience in East Africa.  (See video posted at the bottom of this post.)  In the case we illustrated here, the locally hosted content is being severely undermined by poor BGP routing, even though it is readily available in the region.

Conclusion

It is a wonder that the Internet works as well as it does despite the continuous noise of routing goof-ups and the like.  For tools to help monitor for these types of incidents, please take a look at Dyn’s Internet Intelligence family of products.  For further discussion about dangers of peer leaks, see our blog post from last year entitled Use Protection if Peering Promiscuously.


Share Now

  • frnkblk

    As the percentage of traffic being delivered to end-customers is increasingly being served by local/regional CDNs, has the relative impact of these routing mishaps decreased?

  • It seems so far several hosting companies have problems with attackers or internal issues like rackspsce, go daddy, peer 1, etc. The web is full of these incident reports 🙁
    There are million dolllar infrastructures for cooling and power but where they always have problems seems to be 2 factors: human errors and networking somehow.
    Hope one day we move to the trully 100% uptime.
    Javier @vndhost.

Whois: Doug Madory

Doug Madory is a Director of Internet Analysis at Dyn where he works on Internet infrastructure analysis projects. Doug has a special interest in mapping the logical Internet to the physical lines that connect it together, with a focus on submarine cables.