During August 2018, the Oracle Internet Intelligence Map surfaced Internet disruptions around the world due to familiar causes including nationwide exams, elections, maintenance, and power outages. A more targeted disruption due to a DDoS attack was also evident, as were a number of issues that may have been related to submarine cable connectivity. In addition, in a bit of good news, the Internet Intelligence Map also provided evidence of two nationwide trials of mobile Internet services in Cuba.
On August 15, the Oracle Internet Intelligence Twitter account highlighted that a surge in DNS queries observed the prior day was related to a nationwide test of mobile Internet service, marking the first time that Internet services were available nationwide in Cuba’s history. The figure below shows two marked peaks in DNS query rates from resolvers located in Cuba during the second half of the day (GMT) on the 14th. Paul Calvano, a Web performance architect at Akamai, also observed a roughly 25% increase in their HTTP traffic to Cuba during the trial period.
This testing was reported by ETECSA (the Cuban state telecommunications company) in a Facebook post in which they noted:
The Telecommunications company of Cuba S.A. Etecsa, reports that as part of the preparatory actions for the start of the internet service via mobile, it has carried out some tests to verify the functioning of all elements involved in it.
A Timely test with customers of the prepaid cell service has been developed today, which have been able to make internet connections at no cost. This and other tests make it possible to assess available capacities and different experiences of use achieved in correspondence with the characteristics of the access network present at the time and place of connection in order to make some adjustments. Further tests shall be carried out in successive days.
The start date of the service, as well as its rates and other details of interest, shall be informed through the official media and official channels of the company.
A little more than a week later, another surge in DNS queries from Cuba was observed, resulting from ETECSA performing a second test of national mobile Internet service. As seen in the figure below, it appears that service was enabled around 12:00 GMT on August 22, but was shut off around 05:00 GMT the following day. A notice posted by ETECSA on their Facebook page on the 22nd informed customers of prepaid cellular services that they had the opportunity to acquire a one-time 70 MB data “package” free of charge that could be used at any time during the test period. A subsequent Facebook post noted that customers that exceeded their 70 MB allocation would no longer be able to browse the Internet.
On August 27, the Bank of Spain Web site was targeted with a DDoS attack that published reports indicate temporarily disrupted access to the site. A DNS lookup on the site’s hostname, www.bde.es, shows that it resolves to a single IP address (220.127.116.11). This IP address is part of a block of addresses routed by AS20905, registered to Banco de España. The figure below shows a significant increase in latency in traceroutes to endpoints within that autonomous system during the reported time of the attack. Such increased latency is a concomitant effect of a DDoS attack that floods a target with traffic.
Presumably in response to being targeted by a DDoS attack, the Bank of Spain activated a DDoS mitigation service the following day, as the figure below shows that on August 28, the majority of traceroutes to endpoints in AS20905 started going through Akamai’s Prolexic service (in green on the graph).
Following a number of similar disruptions at the end of July, the Internet in Syria was down from 01:00-05:30 GMT (4:00 am-8:30 am local) on August 1 & 2 as part of an effort to prevent cheating on high school exams, as seen in the figure below.
Four similar exam-related disruptions occurred the following week as well, as the figure below shows traceroute completion ratios and the number of routed prefixes dropping to zero on August 5, 6, 7, and 8. (As noted in last month’s post, we believe that the concurrent spikes in DNS queries are due to the Internet shutdowns being implemented asymmetrically – that is, traffic from within Syria can reach the global Internet, but traffic from outside the country can’t get in. These spikes in DNS traffic are likely related to local DNS resolvers retrying when they don’t receive the response from Oracle Dyn authoritative nameservers.)
Ahead of presidential run-off elections in the country on August 11, authorities in Mali reportedly disrupted Internet access in the country. While not as obvious as disruptions observed in other countries, the figure below shows a decrease in the number of routed networks within the country for several hours in the middle of the day (GMT) on August 10. A published report notes that the Internet shutdown was confirmed by on-the-ground reports from Internet users in Bamako and Gao. In addition, a study by advocacy group NetBlocks found that access to social media and messaging platforms was also disrupted during this period.
Another Internet disruption occured in Mali after the election took place, but before the results were publicly announced. On August 16, advocacy group Internet Without Borders Tweeted:
Total blackout du réseau de télécommunications civiles au #Mali avant la proclamation des résultats de l’élection présidentielle.
Acte final d’un cas atypique de censure en cascade d’Internet.https://t.co/DJ5OFyUqz6 #PresidentielleMali2018 #Keepiton
— Internet_SF (@Internet_SF) August 16, 2018
As the figure below shows, the both the Traceroute Completion Ratio and BGP Routes metrics experienced noticeable multi-hour decreases starting later on August 15 (GMT). While not as obvious, the DNS Query Rate metric also appears to be at a slightly lower level than at similar times during the previous days. Published reports (Bourse Direct, La Nouvelle Tribune) indicated that this second disruption may have primarily targeted mobile networks in the capital city of Bamako, with local reports of connectivity working over fixed-line/Wi-Fi, but problems connecting over 3G.
On August 20, Sure Telecom of Diego Garcia in the British Indian Ocean Territory posted a notice on their homepage alerting users of a multi-hour “maintenance outage” that would impact availability of Internet services offered by the company.
The impact of this maintenance outage can be seen in the figure below, with the Traceroute Completion Ratio and DNS Query Rates dropping to zero during the specified maintenance window. (British Indian Ocean Territory is GMT+6.) The BGP Routes metric was also lower during the maintenance period, but didn’t drop to zero. Sure Telecom appears to be the sole commercial Internet Service Provider in the British Indian Ocean Territory, so it is not surprising that this maintenance had such a significant impact on Internet availability in the region.
On August 17, a massive power outage affected the Sindh and Balochistan provinces in Pakistan for several hours. The blackout also had an impact on Internet availability within the country. As the figure below shows, the Traceroute Completion Ratio metric declined sharply as the power outage occurred, remained low for a few hours, and then gradually recovered. This indicates that traceroute endpoints in impacted locations were likely unreachable due to the power outage. However, because it occurred later on Friday evening local time, Internet usage was likely ramping down anyway, so there was no clear impact on the DNS Query Rate metric; the BGP Routes metric was unaffected because the routers announcing routes to IP address prefixes in the affected regions are either located in data centers with backup power, or are located in Pakistani cities not impacted by the power outage.
Closing out the month, on August 31, an explosion at an electrical substation in Maracaibo, Venezuela, plunged much of the city into darkness, and had a visible impact on the country’s Internet connectivity as well. The figure below shows a decline in the Traceroute Completion Ratio metric at approximately 05:30 GMT, coincident with the explosion, which published reports state occurred at 1:36 AM local time. A minor increase in the unstable network count can be seen at approximately the same time as well.
Internet disruptions due to issues with submarine cables are not uncommon, but are often hard to confirm as cable operators are often reticent to publicize faults in the cables that they are responsible for. However, sometimes the issues cause are significant enough to be covered in the news, and other times impacted service providers will expose such issues as the root cause of problems that their customers are experiencing.
The latter scenario occurred on August 28/29 in the Maldives, as illustrated by Tweets from two local providers:
we are experiencing internet traffic instability on international routes due to a technical issue on submarine cable. We are working with our cable system provider to resolve the issue, apologies for the inconvenience. Thankyou
— Raajje’ Online (@ROLMaldives) August 29, 2018
We’re experiencing interruption in our international routes due to an unforeseen circumstance from our supplier’s end. While we are working to fix this, our customers may face difficulty in using Internet on Mobile, SuperNet & Faseyha, as well as making IDD calls during the time.
— Ooredoo Maldives (@OoredooMaldives) August 29, 2018
As the figure below illustrates, at approximately 01:00 GMT on August 29, both the Traceroute Completion Ratio and BGP Routes metrics for the Maldives declined from their ‘steady-state’ values. As evidenced by the Tweets from local Internet Service Providers shown above, these declines are likely due to the referenced submarine cable issue.
Two cables carry international traffic for the Maldives: the WARF Submarine Cable, which runs to Sri Lanka and India; and the Dhiraagu-SLT Submarine Cable Network, which runs to Sri Lanka. Based on analysis of routing path data collected by Oracle’s Internet Intelligence team, we can posit that issues with the WARF cable likely caused the observed/reported Internet disruption, as upstream providers for both Ooredoo Maldives and Raajjé Online include companies listed as owners of the cable.
Several additional unattributed Internet disruptions were observed during August in areas that have a significant reliance on submarine cables for international Internet connectivity.
On August 14, the Internet Intelligence Map showed an Internet disruption in Vanuatu, a nation made up of approximately 80 islands in the South Pacific. As seen in the figure below, the values of all three metrics declined for a multi-hour period.
Vanuatu is connected to Fiji via Interchange Cable Network 1 (ICN1). The figure below shows that nearly all traceroutes to Telecom Vanuatu reach the network through Vodafone Fiji Limited, and that the number of completed traceroutes to Telecom Vanuatu dropped to near zero during the same period shown in the figure above. Nearly all traceroutes to Vodafone Fiji Limited reach the network through Telstra Global, an Australian provider, and the figure below also shows that the number of completed traceroutes to Vodafone Fiji Limited dropped to near zero concurrently. Fiji connects to Australia via the Southern Cross Cable Network (SCCN). As such, the observed disruption may be related to issues with one of these two cables, possibly cable maintenance, as the disruption occurred in the middle of the night local time. Interestingly, this disruption occurred while the 2018 Asia Pacific region Internet Governance Forum (APrIGF 2018) was taking place in Port Vila, Vanuatu.
On August 4, concurrent Internet disruptions were observed in Angola, Cameroon, and Gabon, as seen in the figures below. Although brief, all three metrics were affected across the three impacted countries. While no specific publicly available information about the cause of the disruption could be found, research shows that all three countries are connected to both the SAT-3/WASC and Africa Coast to Europe (ACE) submarine cable systems. Damage to, or maintenance on, a segment of one of these cables could potentially have caused the observed disruption. This blog has previously covered the impact of damage to the ACE cable on Internet connectivity in African countries.
In the Caribbean, a brief Internet disruption was observed on August 5 in Grenada and St. Vincent & the Grenadines. Both countries connect to the Eastern Caribbean Fiber System (ECFS) and Southern Caribbean Fiber cables. On August 21, a brief Internet disruption was observed in Saint Barthelemy, which is also connected to the Southern Caribbean Fiber cable (on a spur from Anguilla). These disruptions are evident in the figures below. As island nations, all three countries are heavily reliant on submarine cables for international Internet connectivity, meaning that the observed disruptions could have been caused by damage to, or maintenance of, these cables.
Although we have historically focused on the value of the Internet Intelligence Map in identifying disruptions to Internet service, it was encouraging to also see it offer evidence in August of the nationwide mobile Internet service tests conducted in Cuba. Limited Internet access has been available on the island through paid public hotspots, but if ETECSA makes mobile access widely available (and affordable), then disruptions to Internet connectivity in Cuba (ultimately visible in the Internet Intelligence Map) will have a much more significant impact.