Internet Performance Delivered right to your inbox

Routing Leak briefly takes down Google

This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).

Bharti in turn announced these routes to the rest of the world, and a number of ISPs accepted these routes including US carriers Cogent (AS174), Level 3 (AS3549) as well as overseas incumbent carriers Orange (France Telecom, AS5511), Singapore Telecom (Singtel, AS7473) and Pakistan Telecom (PTCL, AS17557). Like many providers around the world, Hathway peers with Google so that their customers have more direct connectivity with Google services. But when that private relationship enters the public Internet the result can be accidental global traffic redirection.

Last fall, I wrote two blog posts here and here about the issues surrounding routing leaks such this one. Routing leaks happen regularly and can have the effect of misdirecting global traffic. Last month, I gave a talk in the NANOG 63 Peering Forum entitled “Hidden Risks of Peering” that went over some examples of routing leaks like this one.

Below is a graph showing the timeline of the incident for one of the 336 prefixes involved. Bharti (AS9498) should never have been seen as an upstream of Hathway (AS17488) for any Google prefixes. As the graph shows, only a portion of the Internet accepted these routes: the providers who peer with or sell to Bharti, and who failed to filter Bharti’s BGP announcements.


216.58.223.0_24_1426150200-2
Below is a traceroute from one of our servers in Bratislava, Slovakia earlier today showing traffic to Google redirected to India.


trace from Bratislava, Slovakia to 72.14.210.134 (Google) at 09:09 Mar 12, 2015
1  *
2  *
3  *
4  149.11.48.1      te0-0-2-3.nr11.b027220-0.bts01.atlas.cogentco.com   1.95
5  154.25.3.181     te0-0-2-0.agr11.bts01.atlas.cogentco.com            1.908
6  154.54.37.229    te0-3-0-5.ccr21.bts01.atlas.cogentco.com            1.574
7  130.117.1.50     be2222.ccr21.vie01.atlas.cogentco.com               3.552
8  130.117.49.1     be2200.ccr21.muc01.atlas.cogentco.com               9.818
9  130.117.0.250    be2023.ccr21.zrh01.atlas.cogentco.com               14.892
10 130.117.50.165   be2024.ccr21.mrs01.atlas.cogentco.com               27.371
11 149.6.155.182                                                        33.255
12 182.79.237.125   (Airtel Limited, India)                             158.796
13 *
14 202.88.147.66    (Hathway, Mumbai, India)                            283.586
15 *
16 72.14.235.29     (Google, Mumbai, India)                             282.664
17 209.85.255.131   (Google, Mumbai, India)                             294.956
18 *

Highly peered content networks such as Google are uniquely vulnerable to this type of accidental traffic misdirection. Once routes are handed off to a peer, that peer can make a mistake and re-route your traffic. Vigilance is critically important: we know that Hathway was a risky peer for Google because just 22 hours previously, Dyn observed Hathway leaking 134 Google prefixes to Bharti for less than a minute. Careful monitoring of global routing is the only way for enterprises to detect these situations before they become front page news.


Share Now

Whois: Doug Madory

Doug Madory is a Director of Internet Analysis at Dyn where he works on Internet infrastructure analysis projects. Doug has a special interest in mapping the logical Internet to the physical lines that connect it together, with a focus on submarine cables.