15:03 UTC: the Dyn Operations team was notified of an issue with Dyn Standard DNS nameservers. The team then immediately began investigating the issue and identified it as a Distributed Denial of Service (DDoS) attack against all five Dyn Standard DNS nameservers. Compounding this issue was a series of wide scale Internet stability issues caused by a software bug in a major networking vendor’s routing code, which affected BGP routing for the a good majority of the Internet. This added complexity in identifying the DDoS vector, ultimately delaying our efforts to begin mitigation of the attack.
15:20 UTC: the nature of the attack was identified and our DynStatus site was updated. Operations began deploying our well-practiced DDoS countermeasures and mitigations. At 15:40 UTC, a majority of Dyn Standard DNS nameservers were offline due to complete exhaustion of server resources attempting to migitate the attack. At 16:10 UTC, all Dyn Standard DNS nameservers went offline as server resources were completely exhausted.
16:32 UTC: our ns2.mydyndns.org nameserver returned to service, protected by a variety of anti-DDoS mitigation systems including router ACLs, firewalls and DDoS scrubbing devices. At 16:50 UTC, the ns3.mydyndns.org nameserver returned to service. Due to complexities of fully reloading edge nameservers, it took until 17:50 UTC to return ns1.mydyndns.org and ns4.mydyndns.org to service. Finally, ns5.mydyndns.org was back in service at 18:15 UTC.
An additional complicating factor was that our DynStatus site became overwhelmed with traffic at 16:30 UTC. At this time, we opted to use both Twitter feeds to communicate with our users (primarily @DynDNS and @DynInc) while we altered the configuration of the DynStatus site to handle additional load. At 17:23 UTC, the DynStatus site was online again.
So today, for the first time since 2001, we experienced a full 22 minute outage of our Dyn Standard DNS nameservers, which means that we reset our Dyn Standard DNS uptime counters back to zero. For that, we’re disappointed and we apologize to our customers that were affected by both the outage and the hiccup with our DynStatus site that prevented us from communicating to the extent that we wanted to do. We believe that transparency is critical in keeping our customers informed and will be taking efforts to harden our DynStatus site to ensure it is always available, even if or DNS servers are not. We appreciate your decision to use our services and we thank you for the patience during this issue.
As Dyn is constantly dealing with DDoS attacks, we have a tradition of naming them similar to the way hurricanes are named in the US. Today’s event was named Fiona. Attached to the name is a post mortem analysis of the event to identify the area of weakness in our network and systems, so that immediate improvements can be made. That process has already started.
For customers utilizing Dyn’s DynECT Managed DNS platform, served from 17 global datacenters, no issues or outages were observed during the course of the event.