Internet Performance Delivered right to your inbox

Last Month in Internet Intelligence: October 2018

The level of significant Internet disruptions observed through the Oracle Internet Intelligence Map was lower in October, though the underlying reasons for these disruptions remained generally consistent compared to prior months. For enterprises, the importance of redundant Internet connectivity and regularly exercised failover plans is clear. Unfortunately, for state-imposed Internet outages, such planning and best practices may need to include failsafes for operations while periodically offline.

Directed disconnection

On October 10, Ethiopian Prime Minister Abiy Ahmed met with several hundred soldiers who had marched on his office to demand increased pay. The Ethiopian Broadcasting Corporation (formerly known as ETV) did not cover the soldiers marching but noted that Internet connectivity within the country had been shut off for several hours to prevent “fake news” from circulating on social media. This aligned with residents’ reports of a three-hour Internet outage. The figure below shows that the disruption began around 12:00 GMT, significantly impacting both traceroutes to, and DNS query traffic from, Ethiopia for several hours.

The impact of the Internet shutdown is also clearly evident in the figure below, which shows traceroutes into Ethio Telecom, the state-owned telecommunications service provider. Similar to the country-level graph shown above, the number of completed traceroutes into Ethio Telecom dropped significantly for several hours. However, a complete Internet outage was not observed in either case.

Exams

On October 14, the Ministry of Communications in Iraq announced the latest round of Internet shutdowns within the country in conjunction with nationwide exams. According to the translation of an article posted on the Iraqi Media Network, “The ministry’s spokesman Hazem Mohammad Ali said in a statement that the Internet service will be interrupted for two hours a day from 11 am to 1 pm for ten days from Sunday, 2018/10/14 until Wednesday, 2018/10/24, This piece came at the request of the Ministry of Education and will stop the Internet service for the days of examinations exclusively.”

As shown in the figures below, multiple Internet shutdowns were observed during the specified period, but they did not appear to take place on October 19, 23, or 24 as expected. As has been seen during previous Internet disruptions in Iraq, the government’s actions cause a decline in the number of completed traceroutes to targets in the country, a reduction in the number of routed networks from the country, and lower levels of DNS traffic seen from resolvers on Iraqi networks.

As noted in the past, during these nationwide disruptions, telecoms with independent Internet connections through the north of Iraq often stay online, as do those with independent satellite links. However, the figures below illustrate the impact of these state-imposed disruptions on two major Iraqi network providers, ITC and Earthlink. These graphs show that the observed disruptions within these networks appear to be near-complete, as well as the lack of anticipated outages on the 19th, 23rd, and 24th.

Power outage

On October 15, the Tweets above (among others) provided insight into the scope of a widespread power outage in Venezuela. This October blackout follows similar events in July and August. As seen in the figure below, the traceroute completion ratio metric saw a sharp but partial drop during the evening of October 15, aligned with the time the power outage reportedly began. (Venezuela is 4 hours behind GMT.) The metric recovered gradually over the following 24 hours, though it took a few days for it to return to normal. The blackout did not have a meaningful impact on the BGP routes metric, which is not surprising, because the relevant routers are generally located in data centers with backup/auxiliary power, such as generators. Interestingly, the power outage appeared to have something of a delayed impact on the DNS query rate metric; while the request traffic followed a pattern roughly similar to that seen on preceding/following days, the volume of requests was slightly lower.

The impact of the power outage was also visible in the Traffic Shifts  graphs of a number of Venezuelan network providers, as shown in the figure below. It is particularly evident in the graphs for Net Uno and Inter, seen in the top row, with noticeable declines in the number of completed traceroutes to targets in those networks. The impact was less pronounced in other networks such as Digitel and CANTV, with the number of completed traceroutes seeing a more nominal decline.

Severe weather

After getting battered by Typhoon Mangkhut in September, the Northern Mariana Islands were devastated on October 24-25 by Super Typhoon Yutu, which hit as a Category 5 typhoon.  The figure below shows an Internet disruption starting mid-morning (GMT) on October 24. (The Northern Mariana Islands are 10 hours ahead of GMT.) As the graphs show, both the traceroute completion ratio and DNS query rate metrics dropped concurrent with the arrival of the storm, but the routing infrastructure remained stable. The other figure below illustrates the impact of Yutu on IT&E Overseas, a Guam-based telecommunications firm that also provides Internet connectivity in the Northern Mariana Islands.  As seen in the figure, the number of completed traceroutes reaching endpoints in IT&E declined as the storm hit, with upstream connectivity through Hutchinson, Level 3, Tata, and Cogent evidently impacted.

Network malfunction

On October 22, East Timor (also known as Timor-Leste) suffered a multi-hour Internet disruption, reportedly due to a failure at an upstream provider of the country’s largest telecommunications operator. Coverage of the issue noted that Timor Telecom’s link to upstream provider Telkomsel failed at around 17:45 local time (08:45 GMT), and that a failover to satellite provider O3b did not occur as expected. Services were reportedly restored just before 23:00 local time (14:00 GMT). The figure below shows how the link failure impacted connectivity at a country level in East Timor. The traceroute completion ratio metric was lower for the 5-plus hour duration of the disruption, and the number of routed networks dipped lower for the period as well. The impact is harder to see in the in the DNS query rate graph, likely due to the skew caused by the spikes on October 21 and 23, but the graph does appear to flatten during the period of the disruption.

The traffic shifts graphs below illustrate how the Telkomsel link failure impacted network providers in East Timor. Published reports quoted a representative of Timor Telecom, and the first figure corroborates their report of the problems with Telkomsel and failed shift of upstream traffic to O3b. (Telkomsel is a subsidiary of Telekomunikasi Indonesia International, which is shown in the graphs below.) The graph shows that that the majority of the traceroutes to targets in Timor Telecom go through satellite Internet provider O3b, with a fraction going through Telekomunikasi Indonesia International (T.L.), which is presumably a network identifier that the Indonesian provider uses for Internet services in East Timor. However, when the link to Telekomunikasi Indonesia International failed, it appears that the link to O3b did as well, dropping the number of completed traceroutes to near zero, and spiking the latency for those that did complete.

The second figure shows that Telekomunikasi Indonesia International (T.L.)  gets nearly all of its upstream connectivity through PT Telekomunikasi Indonesia, and the link failure is clearly evident in that graph. Finally, the third figure illustrates the impact of the link failure on Viettel Timor Leste, which also uses Telekomunikasi Indonesia International (T.L.) as an upstream provider. The graph shows that when the problems with Telekomunikasi Indonesia International (T.L.) occurred, traceroutes to targets in Viettel found alternate paths, with increasing numbers going through Asia Satellite Internet eXchange (ASIX) and PT. Sarana Mukti Adijaya.

Conclusion

In addition to the Internet disruptions reviewed above, notable irregularities were observed in Mayotte, Mali, Botswana, and St. Vincent and the Grenadines during October. However, the root causes of these disruptions remain unknown. Observed network-level disruptions aligned with the country-level ones, but no public information was found that explained exactly why these Internet outages occurred. And in addition to these, the Oracle Internet Intelligence Map surfaced hundreds of brief and/or minor Internet disruptions around the world over the course of the month.

Regardless of the underlying causes, the importance of redundant Internet connections and the need to regularly test failover/backup infrastructure cannot be understated, as we saw in East Timor. While this may be challenging, and even expensive, in remote locations dependent on submarine cables or satellite connectivity for Internet access, the growing importance of the Internet for communication, commerce, and even government services means that wide-scale Internet disruptions, even brief ones, can no longer be tolerated.


Share Now

David Belson
Whois: David Belson

David Belson is a Senior Director of Internet Research & Analysis at Oracle Dyn Global Business Unit, a pioneer in managed DNS and a leader in cloud-based infrastructure that connects users with digital content and experiences across a global internet.