Tom Daly Archive - Dyn

12.02.2011 By Tom Daly

Observed DNS Anomaly: Bumps in DNS ANY Query Activity

For the past 5 days, we’ve been seeing a 20-25% bump in DNS QPS for random 5-10 minutes intervals that started back up this morning. It’s certainly not a pattern that we’re accustomed to seeing, so we classified this as a traffic anomaly and sent our Operations Team searching for answers on what the source of this might be.

Here’s the particulars of what we’re seeing:

11.16.2011 By Tom Daly

AllOps: Evolution From The DevOps Culture Movement

For those in the software engineering, system administrator and generalist IT fields, you may have heard about a culture shift known as DevOps.

Designed to break down the interdepartmental barriers that exist between business technology divisions at strategic and tactical levels, the tactics behind DevOps bring together traditionally partitioned IT resources, such as software engineering and development, with IT operations, including system administration and infrastructure, by enhancing communication, collaboration and integration. The long-term strategy yields a more productive IT organization that is able to deliver more business value throughout the company.

For too long, IT organizations have been plagued with the blame game, which is what the concept of AllOps aims to eliminate. Tell me if you’ve heard of or been part of one of these scenarios before…

11.09.2011 By Tom Daly

Evaluating The Growth Of Internet Traffic

At the opening of NANOG 53, Kevin McElearney of Comcast commented that within Comcast Regional Area Networks (CRANs), the company is regularly pushing 40+ Gbps of traffic out to the global Internet. This is a massive amount of traffic and in many cases, it’s more traffic than entire countries around the world push out to the Internet.

It got me thinking about just how much traffic there is on the Internet and the rate at which that traffic will grow over the coming years.

11.07.2011 By Tom Daly

Post Mortem: Attack To Dyn Standard DNS Nameservers

15:03 UTC: the Dyn Operations team was notified of an issue with Dyn Standard DNS nameservers. The team then immediately began investigating the issue and identified it as a Distributed Denial of Service (DDoS) attack against all five Dyn Standard DNS nameservers. Compounding this issue was a series of wide scale Internet stability issues caused by a software bug in a major networking vendor’s routing code, which affected BGP routing for the a good majority of the Internet. This added complexity in identifying the DDoS vector, ultimately delaying our efforts to begin mitigation of the attack.

15:20 UTC: the nature of the attack was identified and our DynStatus site was updated. Operations began deploying our well-practiced DDoS countermeasures and mitigations. At 15:40 UTC, a majority of Dyn Standard DNS nameservers were offline due to complete exhaustion of server resources attempting to migitate the attack. At 16:10 UTC, all Dyn Standard DNS nameservers went offline as server resources were completely exhausted.

16:32 UTC: our ns2.mydyndns.org nameserver returned to service, protected by a variety of anti-DDoS mitigation systems including router ACLs, firewalls and DDoS scrubbing devices. At 16:50 UTC, the ns3.mydyndns.org nameserver returned to service. Due to complexities of fully reloading edge nameservers, it took until 17:50 UTC to return ns1.mydyndns.org and ns4.mydyndns.org to service. Finally, ns5.mydyndns.org was back in service at 18:15 UTC.

An additional complicating factor was that our DynStatus site became overwhelmed with traffic at 16:30 UTC. At this time, we opted to use both Twitter feeds to communicate with our users (primarily @DynDNS and @DynInc) while we altered the configuration of the DynStatus site to handle additional load. At 17:23 UTC, the DynStatus site was online again.

So today, for the first time since 2001, we experienced a full 22 minute outage of our Dyn Standard DNS nameservers, which means that we reset our Dyn Standard DNS uptime counters back to zero. For that, we’re disappointed and we apologize to our customers that were affected by both the outage and the hiccup with our DynStatus site that prevented us from communicating to the extent that we wanted to do. We believe that transparency is critical in keeping our customers informed and will be taking efforts to harden our DynStatus site to ensure it is always available, even if or DNS servers are not. We appreciate your decision to use our services and we thank you for the patience during this issue.

As Dyn is constantly dealing with DDoS attacks, we have a tradition of naming them similar to the way hurricanes are named in the US. Today’s event was named Fiona. Attached to the name is a post mortem analysis of the event to identify the area of weakness in our network and systems, so that immediate improvements can be made. That process has already started.

For customers utilizing Dyn’s DynECT Managed DNS platform, served from 17 global datacenters, no issues or outages were observed during the course of the event.

11.07.2011 By Tom Daly

Snowtober: Managed DNS, Email Delivery Aren’t Much Different than Power & Water Utilities

Just over a week ago, the northern New England corridor was hit with a historic October Nor’easter, duly named “Snowtober”. On Saturday, October 29^th, more than a foot of snow was dropped across southern New Hampshire and northern Massachusetts, causing numerous power outages, road closures and general discomfort for residents throughout the region.

When the power goes out, the ramifications are endless. Pumps cannot be run to supply water, furnaces cannot run to supply heat and refrigerators cannot run to keep food cold and/or frozen. When the water supply is impacted, it prevents people from cooking, cleaning and bathing, basically meaning that general human hygiene can be affected.

The interesting part to me is that these basic functions are rarely thought of, except during a utility outage like the ones caused by the Snowtober Nor’easter. For the most part, most people simply expect power to be on and water to be flowing as these are utility services that are provided to the home and one really doesn’t think about these services until an outage occurs.

The same parallel can be drawn to DNS and Email Delivery systems –- people don’t always spend time thinking about them until an outage or issue arises.

10.13.2011 By Tom Daly

NANOG: Stewards Of The Internet

I was at NANOG 53 in Philadelphia, PA, this week and while on the flight home, I started thinking about the roots of the Internet, how the network was constructed and the ways in which it has evolved in our daily lives. I spent some time thinking about what has changed with the Internet and what things remains the same today.

Thinking about this immediately post-NANOG yields interesting thoughts with one key theme emerging: one thing that hasn’t changed about the Internet is that it takes a closely-knit group of evangelists and stewards to keep it running.

09.13.2011 By Tom Daly

DNS Infrastructure Upgrades Coming In U.S., Europe, Australia

As we alluded this past spring, we’ve been making significant headway on our year-long plan of continued improvements and upgrades to the DynECT Managed DNS edge infrastructure. The results of these upgrades are twofold: to continue to provide our growing list of clients and partners with rock-solid managed and outsourced DNS services and to continue our ongoing mission to win the battle against network latency.

Please join me on a little jaunt around the U.S. to review what we’ve been up to and what’s ahead for our worldwide Anycast network.

08.30.2011 By Tom Daly

Hurricane Irene Fallout: Is Your Data Center Ready For Major Weather Events?

For those living on the eastern seaboard of the U.S., you have probably heard all about Hurricane Irene pounding the East Coast with heavy rain and high winds over the past weekend. The storm system brought Category Two conditions along the Carolinas, Category One conditions through Maryland and Delaware and tropical storm conditions through New York City and the Northeast where our base of Manchester, NH, got especially hit hard Sunday.

For professionals running Internet applications worldwide, a key concern is what happens to infrastructure when a power outage occurs, a condition of high likelihood when faced with impending hurricane events.

06.28.2011 By Tom Daly

DNS Talk: Forging Towards A Secure Internet Control Plane

In the past few weeks, I’ve been learning more about macro-level steps to secure the global Internet’s control plane from a BGP routing perspective. The perspective of this new technology — RPKI and BGPSEC combined with DNSSEC — and another new technology called DANE presents an opportunity to secure the Internet’s most basic control plane functions.

To be perfectly clear, when I say “Control Plane of the Internet,” I’m referring to the core protocols used to route and direct traffic between networks on the Internet, specifically Border Gateway Protocol (BGP) and the Domain Name System (DNS). Given the recent NANOG 52 meeting in Denver, CO, it’s timely to provide a quick overview of these technologies and how they may be leveraged in the future.

Author Archive