The whole reason for caching DNS records is to reduce unnecessary DNS queries; many webservers don’t change their IP addresses all that often. Every DNS record that your operating system requests has what is called a “Time to Live” (TTL), which is a number (in seconds) that determines how long a particular DNS record is cached by your Operating System (OS).
As the Internet has grown, web browsers have gotten faster and better at providing a good user experience. One of the ways they do this is to cache DNS records for a short time on top of the OS level cache.
Here’s what web browser DNS caching is all about, how our Internet Guide “Continue to Site” feature helps alleviate it, and why the overall practice can be frustrating.
How It Works
When you visit a website like Dyn.com, your web browser needs to resolve that domain name to an IP address before it can load the page. When your web browser makes a request for that DNS record, it first checks the operating system resolver. If the IP for the domain name isn’t cached there, your OS queries your nameserver for the DNS record, which then gets passed to the browser. When the TTL has passed, your OS purges that resolved domain from the cache and subsequent requests to that domain would cause a new query to your nameserver.
TTL values are set per DNS record. In the past, a very common value of 86400 (24 hours) was used. As the Internet has grown, this value has become too large for some sites. Load balancing, active failover, disaster recovery and many other things work and benefit from setting a deliberately low TTL in their DNS record.
A simple program I wrote to query the top 1000 websites (according to Alexa) shows 212 hits with a TTL value of 300 (5 mins), 192 hits with a TTL of 3600 (1 hr), 116 hits with a TTL of 600 (10 mins) and 79 hits with a TTL of 86400. The rest of the results had hits in the 50s and less, ranging anywhere from a TTL of 5 (1 hit) to a TTL of 864000 (1 hit).
You may be wondering why web browsers would even cache DNS entries if the operating system already does this. Supposedly, this happens to reduce DNS server load and speed up response time, although I can’t imagine a hit on your OS DNS cache is all that expensive of an operation. Mozilla has even documented this behavior here. Most other browsers do this too, though their times seem to vary somewhat, ranging from 15 to around 60 seconds with Opera caching (seemingly) indefinitely until browser restart and Internet Explorer for 30 minutes.
Internet Guide’s Continue To Site
If you saw our article on our Internet Guide “Continue to Site” feature back in April, you might be starting to see how this relates. With this feature, our DNS servers actually give back the real DNS record when you click the “Continue” button. This is an important distinction from other services that appear to do the same thing. These services usually cost a lot of money, and traffic going to a “soft-blocked” site is proxied through their servers which can break a lot of functionality and important security.
I can’t go into much more detail here, but in our first “Continue to Site” iteration, users had to wait 5 to 75 seconds depending on the browser, but the upper end at 75 seconds was unacceptable to us. Through some clever development and lots of testing, we managed to get most browsers (I’m looking at you, Internet Explorer and Opera) down to 15 seconds or less. We think 15 seconds is short enough that most people are willing to wait for it.
The chart below illustrates our TTL test results from Wireshark and the observed behavior of different browsers and operating systems.
I understand why web browser vendors make the choice to cache DNS, but there are edge cases like this that do more harm than help. It seems to me that an elegant way around browser DNS caching would be a header that a webserver could respond with, telling the browser, “Hey, I might not want my DNS entry to be cached by you. Respect the OS DNS cache please.”. Something like:
X-DNS-Cache: no could work beautifully. The only catch is whether web browser vendors would adopt this. It would certainly make some of our lives easier.
As more and more web technologies are pushing the limits of what can be done, snags like this are bound to happen.