Dyn Inc. Prevents BIND Denial of Service Vulnerability from Affecting Customers
Yesterday was an exciting day in the world of DNS. Lots of activity. Not all of it good.
So, when my cell phone rings 30 minutes after I left the office and I see it’s Dyn Inc. CTO Tom Daly, I know it’s going to be a long night. I was on my way to a softball game, but had my laptop with me just in case. He caught me before I left the driveway.
A vulnerability in BIND, the software we rely on at Dyn Inc. for our managed DNS services, was announced. A specially-crafted dynamic update message could cause a nameserver to crash. Was this what was behind some major DNS outages at ISPs and web hosting providers that broke name resolution for thousands of web sites yesterday?
Dyn Inc. customers were not affected. When we became aware of the vulnerability, we mobilized our network operations and development teams to make sure we would not be. Accompanying the vulnerability announcement was an implementation of the exploit, increasing the chances that one of our nameservers could fall victim to the exploit. We had to move fast to protect ourselves and prepare for any attack that might come our way.
At Dyn Inc. we run a customized version of BIND that lets us handle the volume of dynamic updates and zone changes that millions of customers depend on every day. It’s the beauty of open source that allows for that kind of innovation. Our primary developer behind these efforts likes to work in the shadows. But a patched version of the latest and greatest version was available to us within the hour.
Let the patching begin.
So begins the upgrade process which, following best practices, included some comprehensive testing. This is when one of our team members, who cut his tennis game short to come do some code-diving, realized that the issue was specific to master nameservers. A few quick tests to demonstrate the issue, and Tom was sharing this information with the community. The vulnerability update was modified to acknowledge this new information.
We also all let out a breath. We were quite relieved to discover that some specific architectural decisions we made in deploying BIND kept us safe from this particular exploit. We patched anyway. That’s how we roll.
Some may say that BIND is the problem. We disagree. The very nature of open source, where a distributed team of independent experts and stakeholders around the world could investigate problems, led to the quick and effective ability to squelch this bug. The Internet Systems Consortium (ISC), the organization responsible for coordinating the open source efforts on BIND and distributing the results, had patches to affected BIND versions available quickly. A timely announcement on the dns-operations mailing list got the information out to BIND system administrators paying attention. In our opinion, the net result of this process is more robust software (which is a good thing!), and we are all a little better off for having gone through the experience.
Over a post-mortem pint, Tom and I reflected on the number of nameservers our team patched over the course of the evening. Less than five hours after the ISC notice was posted, Dyn Inc. was fully patched. Not just our authoritative nameservers, either, but also the nameservers for our Spring Server VPS platform.
While all this was going on, the DNSCog.com developers were quickly putting into place a vulnerability check. A previously mentioned tennis game was a victim here too. I encourage everyone to take a look. Are your nameservers vulnerable? Has your hosting company patched yet?
You can be sure that if your DNS is with Dyn Inc., the servers resolving your queries will not fall prey to this DoS vulnerability.
Many personal plans were interrupted to make sure we continue resolving the domains we’re responsible for. At Dyn Inc. “uptime is the bottom line”, and we all take personal responsibility for that.
Here’s a quick run-through of how to get BIND patched for those of you following along at home:
Download new BIND:
Get the build line from existing BIND:
Configure and install BIND:
tar -zxvf bind-9.6.1-P1.tar.gz
./configure (add build line from old BIND)
Shutdown old BIND and wait for exit:
Startup new BIND: