Sprint, Windstream: Latest ISPs to hijack foreign networks

Last year my colleague Jim Cowie broke a story about routing hijacks that resulted in Internet traffic being redirected through Iceland and Belarus. Unfortunately, little has changed since then and the phenomenon of BGP route hijacking continues unabated and on an almost daily basis.

In the past three days, the situation has gone from bad to downright strange as we have observed a flurry of this activity. Now, for the first time, we’re seeing major US carriers, Sprint and Windstream, involved in hijacking, along with the return of an operation out of Poland targeting Brazilian networks. We see router misconfigurations regularly in BGP data – could these incidents also be explained by simple command-line typos?

Route hijacking continues

In May my colleague, Earl Zmijewski gave a presentation about routing hijacks at the LINX 85 meeting, describing a comprehensive system that can be used to identify suspicious hijacks on a global basis and without any prior knowledge about the networks involved. While we now detect suspicious routing events on an almost daily basis, in the last couple of days we have witnessed a flurry of hijacks that really make you scratch your head.

To mention a few recent events, last month we tweeted about the Korea Meteorological Administration mistakenly hijacking a prefix from the US National Climatic Data Center, which hosts websites like www.climate.gov and www.drought.gov, thereby making these sites and many others unreachable to most of the Internet. And then we saw a company in South America trying to hijack back its address space from another entity that was squatting on it.

US carriers hijacking foreign networks

From 13:56 UTC on Tuesday (9-September) to 15:56 UTC on Wednesday (10-September), US wireless carrier Sprint started hijacking a prefix (95.128.184.0/22) from Telesmart, an ISP in Macedonia. About 65% of our peers believed that Sprint was the origin of this prefix and so redirected Macedonian traffic for it to Sprint.

What was interesting was that once traffic arrived at Sprint, it continued onto Cogent and finally onto its intended destination at Telesmart in Skopje. Was this an accidental man-in-the-middle or something else? That is, probes from most of our servers around the world suddenly started going through Sprint to reach Internet resources hosted at Telesmart, such as news and municipal websites. Below is one of the more striking examples of the impact on traffic paths. Before the hijack, traffic from Sofia would reach Skopje in about 6ms, as these cites are fairly close.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 03:15 Sep 02, 2014
1 *
2 87.120.6.9    (Neterra, Sofia, Bulgaria)      0.339ms
3 83.217.227.33 (NTT Europe, Bulgaria Facility) 9.446ms
4 83.217.227.66 (NTT Europe, Bulgaria Facility) 6.883ms
5 95.128.184.1  (Telesmart, Skopje, Macedonia)  6.216ms


During the hijack, traffic took the scenic route through Frankfurt, Paris, Frankfurt (again), Munich, Vienna, Sofia (again), then on to Skopje and consequently took about 10 times longer to reach its intended destination.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 21:31 Sep 09, 2014
1 *
2 87.120.6.9        (Neterra, Sofia, BG)                        10.682ms
3 77.67.67.113      ae0-431.sof10.ip4.gtt.net                   0.512ms
4 141.136.110.173   xe-9-2-0.fra23.ip4.gtt.net                  34.056ms
5 213.200.64.54     (GTT, Frankfurt, DE)                        53.026ms
6 213.206.129.65    sl-crs1-par-0-0-5-0.sprintlink.net          49.62ms
7 217.118.224.54    sl-bb21-par-0-13-0-0.sprintlink.net         50.087ms
8 130.117.14.145    po7-0-2.ccr01.par03.atlas.cogentco.com      46.84ms
9 130.117.51.146    te0-8-0-25.ccr42.par01.atlas.cogentco.com   48.43ms
10 154.54.62.78     be2278.ccr42.fra03.atlas.cogentco.com       48.859ms
11 154.54.38.58     be2229.ccr22.muc01.atlas.cogentco.com       52.757ms
12 130.117.49.138   be2223.ccr21.vie01.atlas.cogentco.com       59.008ms
13 130.117.1.21     be2046.ccr21.sof02.atlas.cogentco.com       77.427ms
14 154.54.59.126    te2-1.ccr01.skp01.atlas.cogentco.com        80.141ms
15 95.128.184.1     (Telesmart, Skopje, Macedonia)              48.916ms

For 26 hours, traffic to this Telesmart network in Skopje entered the Sprint network at various global locations (from Frankfurt to San Francisco) and was then handed off to Cogent in Paris en-route to Skopje. As we often are in these cases, we’re left wondering, what the heck was that? Sprint is a big network, but AS1239 doesn’t originate many prefixes outside the US and doesn’t originate anything starting with 95.x.x.x or 59.x.x.x or anything else that is typographically close to this prefix. Thus, this doesn’t look like an obvious and easily explained typo, something that we see all too often. Not only did traffic to this prefix get much slower, such diversion subjected a small amount of regional traffic to the risk of foreign interception — something on the minds of nearly everybody in the post-Snowden era.

But Sprint wasn’t the only US carrier to begin an unusual hijack on 9 September: US provider Windstream (AS7029) began announcing 212.118.142.0/24 (SaudiNet), which is normally announced by Saudi Arabian incumbent, Saudi Telecom. Unlike the previous Sprint example, traceroutes to this prefix along the Windstream route died within Windstream, effectively knocking this network off the Internet for anyone accepting the bogus route. Then on Wednesday, Windstream announced a handful of strange routes for about 10 hours including one from Gaza, one from Iceland, and three from China — all more-specifics of existing routes, ensuring their global propagation and acceptance. These are listed below.

Hijack route    Organization label and location    Victim Route   Victim AS
37.8.96.0/19    Hadara Gaza BSA - 3ed subnet, PS   37.8.0.0/17    Hadara Technologies, AS15975
82.221.102.0/24 Advania hf., Reykjavik, IS         82.221.96.0/19 Thor Data Center, AS50613
116.10.191.0/24 CHINANET Guangxi province network  116.8.0.0/14   China Telecom, AS4134
117.21.191.0/24 CHINANET Jiangxi province network  117.21.0.0/16  China Telecom, AS4134
61.174.51.0/24  CHINANET Huzhou node network       61.174.0.0/16  China Telecom, AS4134


Of the four examples above, only Advania of Iceland (AS30818) tried to get their traffic back by announcing the same more-specific as Windstream. Advania’s attempt to reclaim their network started at 15:06 UTC, at which point Windstream pulled their errant announcement. In all other cases, the hijacks continued until 15:49 UTC. Unlike the previous Sprint example, any traffic headed to computers in these IP address ranges would have been directed to Windstream during this time and then dropped, effectively knocking these networks off the Internet.

There is a potentially innocent explanation to this example. Perhaps, these address ranges were ones that Windstream deemed to be sources of bad traffic and so was “blackholing” them internally, a relatively common practice. In this scenario, we could have simply witnessed Windstream inadvertently leaking internal routes to the global Internet for 10 hours.

Polish hijacking of Brazilian IP address space

In another recent hijacking incident, starting at 03:16 UTC on 6 August, two different ISPs in Poland each started hijacking a prefix each from two different ISPs in Brazil. These hijacked routes were only circulated to a small number of other ISPs in Eastern Europe.

The two prefixes, 177.39.184.0/23 and 187.106.32.0/19, were originated by INEA S.A. (AS33868) and INOTEL S.A. (AS44514), respectively. Once originated, they were both routed through the same upstream provider, another INEA S.A. ASN (AS13110) before going to Hurricane Electric (AS6939) and then onto a handful of providers in Eastern Europe. Routes took the following forms.

 

	Normal route	Hijacked route
177.39.184.0/23	… 3549 52770	… 6939 13110 33868
187.106.32.0/19	… 4230 28573	… 6939 13110 44514

 

Traceroutes from Eastern Europe show traffic paths redirected into INEA S.A. before continuing onto their proper destination.

Before:

trace from Warsaw, Poland to Revo Telecomunicações at 00:35 Aug 05, 2014
1 *
2 217.153.202.161   (GTS Poland Sp. z o.o., Warsaw, PL)             0.952
3 193.85.195.94     (GTS Czech s.r.o.,Prague, CZ)                   17.065
4 *
5 4.69.154.200      ae-4-90.edge4.Frankfurt1.Level3.net             17.222
6 4.68.63.242       glbx-level3-10G.Frankfurt1.Level3.net           25.666
7 189.125.13.178    (Revo Telecomunicações, Porto Alegre, BR)       248.358
8 177.39.185.10     (Revo Telecomunicações, Guarulhos, BR)          251.24
9 177.39.190.8      (Gm Telecom Ltda, Palmitinho, BR)               251.609

During (note the new INEA S.A. hops through Poznań, Poland):

trace from Warsaw, Poland to Revo Telecomunicações at 17:10 Aug 06, 2014
1  *
2  217.153.202.161  (GTS Poland Sp. z o.o., Warsaw)                 1.119ms
3  157.25.248.142   (GTS Poland Sp. z o.o., Warsaw)                 0.397ms
4  185.1.4.2        inea-gw.pix.net.pl                              5.407ms
5  62.21.99.161     rt1-owsiana-vlan503.core.icpnet.pl, Poznań, PL) 5.407ms
6  62.21.99.6       rt1-oswiecenia-vlan407.core.pl, Poznań, PL)     5.608ms
7  212.73.253.137   (Level 3, Frankfurt, DE)                        27.76ms
8  4.69.153.69      ae-9-9.ebr4.Frankfurt1.Level3.net               27.74ms
9  4.69.163.22      ae-74-74.csw2.Frankfurt1.Level3.net             27.65ms
10 4.69.154.72      ae-2-70.edge4.Frankfurt1.Level3.net             27.743ms
11 4.68.63.242      glbx-level3-10G.Frankfurt1.Level3.net           27.854ms
12 189.125.13.178   (Revo Telecomunicações, Porto Alegre, BR)       249.385ms
13 177.39.184.194   (Revo Telecomunicações, Porto Alegre, BR)       248.138ms
14 *

At this year’s Defcon conference, Luca ‘kaeso’ Bruno and Mariano ’emdel’ Graziano gave a talk about the vulnerabilities of some ISPs’ public looking glass utilities that would allow an attacker to remotely modify router configurations. INEA S.A. (AS33868) was on their list of vulnerable ISPs.

The hijacks described above ceased at the end of August. However, on Wednesday we saw some new suspicious activity from AS13110. At 00:14 UTC on 10 September, AS13110 hijacked a prefix from the US Department of Defense (128.19.65.0/24, normally announced by AS27064). At 00:31 UTC the hijack had stopped, but then a few minutes later an ASN downstream of AS13110 began hijacking a new Brazilian prefix (177.200.2.0/23, TecleNet Solucoes Tecnologicas) at 00:35 UTC. Was the DoD hijack an initial test before the real target was selected?

The above graph shows the percent of our peers accepting the bogus origin (AS57807) compared to the legitimate Brazilian origin (AS57807) over a day and a half of this activity. Then this hijack disappeared yesterday morning at 14:13 UTC while I was writing up this blog post.

And there is IP squatting to worry about

On 31 August, responding to an inquiry on the NANOG mail list, I wrote a description of a rather unique IP squatting operation out of Russia that briefly announces (mostly unused, sometimes not) IP address space assigned to others and originates these via two unused AS numbers. To be sure, IP squatting is not a new phenomenon. Nick Feamster and Anirudh Ramachandran from Georgia Tech presented on the use of IP squatting for spam operations at NANOG 36 in 2006.

Starting in late June, we have observed dozens of prefixes announced along AS paths of the following forms (origins are the rightmost ASN) each for no more than a couple hours at a time.

… 39792 57756 {197329 or 43239}    prefix

… 44050 57756 {197329 or 43239}    prefix

Since 30 August, these routes have taken the following form:

… 44050 197598 {49121 197794 or 201910}    prefix

Below is a graph of the bogus route originations of this IP squatting operation on a recent day. Prefixes and origin ASNs are only up for a few hours at a time. AS201910 is the newest formerly unused ASN being used as a bogus origin. On Wednesday of this week, it announced three prefixes of unused IP address space:

193.228.170.0/23  Bundesministerium fuer Land- und Forstwirtschaft  AT
185.33.16.0/22  DuoDecad IT Services Luxembourg S.a r.l.  Luxembourg
194.76.166.0/23  Amadeus Data Processing GmbH  Bayern DE

While this IP squatting operation is rather unique in its methodology, there are far more examples of IP squatting that don’t involve brief prefix announcements from odd ASNs. Perhaps, an amusing on-going example is Bitcanal (AS197426) of Portugal, which originates unused IP address space from a number of locations ranging from Syria (95.212.192.0/18, AYA Internet Service Provider) to Texas (206.218.64.0/22, Office of the Attorney General, State of Texas). (Would someone please alert Texas Governor Rick Perry about that last one?)

The above graphs illustrate that the Syrian prefix has been globally routed out of Portugal since 25 August, while the new home of the Texas prefix has been seen by about 40% of our peers since 19 June.

Conclusion

So what is the story with all of this?

Well, for one, sloppiness abounds. Humans are at the controls and are always capable of mucking things up. For example, British Telecom Latin America (BT Latam, AS7908) has been globally announcing 125.125.125.0/24 since April 2012. The route is technically a hijack of 125.112.0.0/12, which is announced by China Telecom but looks like a leaked internal route. Or take Beltelecom of Belarus, who has been globally announcing 13 prefixes within RFC6598 address space (such as 100.88.0.0/16 and 100.89.0.0/16) intended for internal carrier grade NAT operations since January of this year.

It is interesting to note that the Bitcoin hijack from earlier this year originated its bogus routes using two formerly unused ASNs with a common upstream AS path. The Russian IP squatting operation also typically uses two unused ASNs for its origination. Finally, the Polish hijacking of the Brazilian routes mentioned above also employed two different ASNs, although not unused this time. Perhaps it is a coincidence, or perhaps it is a known tactic: when hijacking multiple routes, it is better to spread out one’s bogus routes across multiple origins to lower the profile of any single misbehaving ASN.

Regardless of the cause of each of these incidents, the problem is a very real and growing one. Perhaps documenting these incidents will promote a greater understanding of the extent and nature of the problems around the trust-based Internet routing system in global use today.