Back in 1998, a bunch of engineers sat down and examined the way that we address computers on the Internet. In those days, we had one addressing and numbering scheme, which you may have heard of, called IPv4 addressing. This addressing scheme is used worldwide to identify computers, printers, routers, and many other devices which connect to the Internet. In fact, the IPv4 addressing scheme, in many ways, defines the Internet today.
There’s a problem with IPv4 however, in that it only contains enough addresses (IPv4 uses 32-bits) to support about 4 billion directly connected devices on the Internet. This is a problem, because we have come to a point where there will be more than 4 billion directly connected devices connected to the Internet within the next couple years. (Go read Geoff Houston’s analysis of the problem , it is quite excellent.)
So, what’s the answer? Enter IPv6, a 128-bit addressing scheme, which yields us a lot more IP addresses. The problem with IPv6 is switching to it. Switching to IPv6, or even running IPv4 and IPv6 at the same time is hard. To make it work, everyone needs to upgrade software in computers, printers, switches, and routers. And to make matters worse, IPv6 isn’t the Internet, yet! It is just another interconnected network of computers, that doesn’t have enough content or eyeballs to be classified as the Internet yet. Until there is solid reasoning (i.e. content and eyeballs) to switch to IPv6, most have put it off.
Dyn Inc has already taken steps to start down this road to IPv6. In fact, you may have seen or used our IPv6-enabled Spring Server VPS, which supports both IPv4 and IPv6 addressing. Our office LAN (in Manchester, NH) is IPv6 enabled, so everyone can enjoy IPv6 access to their Spring Server VPSes (which are in Chicago, IL) and many other IPv6 sites, like ipv6.google.com, nanog.org, icann.org, and iana.org.
Here’s a little network map of how our office gets IPv6 connectivity:
(Whiteboarding props to Cory!)
Very simple, really. We buy an IPv6 pipe from an ISP at our Chicago, IL datacenter, which feeds our router and the Spring Server VPSes with IPv6. Furthermore, we run an IPv6 over IPv4 encapsulation tunnel (much like a VPN) into our Manchester, NH office. From there, we number our office with IPv6 addresses, similar to already existing IPv4 LANs.
So on with my story: One day last week, we had a problem with our IPv6 tunnel and one of our IPv6 ISPs, which effectively cut off IPv6 access for our office. I was surprised to walk into the office and have everyone looking at me like this:
Yeah, IPv6 was down. But what’s the big deal? If the Internet is all running over IPv4, why did anyone care? The Internet still worked, right?
The answer technically, is yes, but another problem was masking the issue. You see, in the domain name system (DNS), we have these things called record types. Record types are used to define how your client should interpret the data that is returned in the DNS record. For example, a record type of “A” means that the data returned represents an IPv4 address, whereas an “AAAA” record means an IPv6 address. Simple, right?
In an effort to spearhead an IPv6 Internet more and more websites have begun to enable themselves with IPv6. To do so, they are publishing AAAA records for their domains. For example, dyndns.com, which isn’t IPv6 enabled (yet), has a DNS lookup that looks like this:
tom@zinc: dig +noall +answer dyndns.com
dyndns.com. 1175 IN A 220.127.116.11
Whereas, a website like nanog.org, which is IPv6 enabled, returns:
tom@zinc:dig +noall +answer nanog.org
nanog.org. 1732 IN A 18.104.22.168
nanog.org. 1779 IN AAAA 2001:48a8:6880:95::21
Note the difference, that the DNS records for nanog.org contains both an A record and AAAA record. On a modern operating system (in my case Windows XP), with IPv6 connectivity, we’re going to try to access the website via the AAAA record. Normal, right? Yes. And if IPv6 doesn’t work, we fail over to the A record and IPv4, right? Wrong! This is the problem we experienced. Because DNS was giving us a AAAA record back, we were trying to use IPv6 to reach nanog.org while IPv6 was down. The result, no nanog.org for us.
The problem was that DNS had no idea that our IPv6 connectivity was down, so it happily returned the AAAA record. Taking the AAAA record, Windows XP tried to access the IPv6 webserver, which it couldn’t reach. Moreover, the DNS requests that were being made were happening over IPv4, so there was no reason that they should fail!
What was the answer? Simply to disable IPv6 on all of our computers until we could confirm that our IPv6 tunnel to Chicago, IL was back up and working properly, which it was, within a couple of hours.
So, where do we go from here? First, lessons learned:
1. When you have both IPv4 and IPv6 enabled on your network, problems that would be easily diagnosed can get masked between the protocols. You and your support staff need to know how to debug both protocols and need to understand the implications of how each client which accesses the network will behave (i.e. not failing back to IPv4 when IPv6 fails).
2. When you have both IPv4 and IPv6 enabled on your network, you need to examine the DNS carefully to know which protocol you’re going to prefer when trying to access a website (i.e. is DNS returning A or AAAA?)
3. Finally, as a company using IPv6, we probably should have multi-homed our IPv6 connectivity to two ISPs, over separate tunnels, to ensure that a problem with one tunnel didn’t cause problems.
Second, questions asked:
1. Remember how I said that the DNS queries we made were being made over IPv4, yet we got back answers (AAAA) for IPv6? This doesn’t make sense to me. For the sake of analogy, this is like saying “the best way to Boston is via bus, but the bus line is broken. You can go by car, which I’ll loan you, but you just can’t have the keys.” And like this analogy, the only way to get to Boston is to hot wire the car, which is similar to disabling IPv6, in this case. To me, it would make much more sense if you make a DNS request using IPv4, you should get back IPv4-compatible answers only, and if you make the same request on IPv6, you get back IPv6-compatible answers only. Anyone know why we did it the other way?
2. Even with the assumption above, that IPv4 queries get IPv4 answers, and IPv6 queries get IPv6 answers, what do you do in hybrid networks, where some clients are IPv4 only or IPv6 only or IPv4/IPv6 dual stack? If your LAN has IPv6, and you query DNS over IPv6, but the query goes out to the world over IPv4, and returns IPv4 addresses, then what? If we hold #1 above to be true, then you’d get no answer. But if you had simply made the query in IPv4, you would have been fine – but the DNS and the LAN have no idea about the conditions of the network once it leaves you – no visibility “end-to-end” as to which protocol will work.
3. Finally, at least for Windows XP (which I did my experimenting on), when IPv6 failed, why didn’t we fall back to IPv4?
If anyone wants to take a stab at these questions, please do so in the comments below, and we’ll talk about them. In the meantime, if you’re going to enable IPv6, please think about our recent situation, and how it could affect you.