Sign In

Internet Performance Delivered right to your inbox

Two Strikes For the I-root

Here we go again. In March we wrote a blog entitled Accidentally Importing Censorship which described how incorrect DNS answers were returned in response to certain queries to the I-root. The problem was tracked down to a single instance of the I-root located in China. Queries to this server for domains blocked in China, such as Facebook, would return seemingly arbitrary answers. As we noted, countries, and even companies, can impose their own standards on the Internet and block anything they want. This story was only noteworthy because those blocks (via bad DNS answers) became visible outside of China. Well, guess what? We are once again seeing the Beijing I-root from outside of China.

Background

Let’s start with a few disclaimers and some background. First and foremost, the sky is not falling. Getting the wrong DNS answer, even when querying the Chinese I-root instance is an extremely rare event. Go back and read our earlier blog to see the exact alignment of the stars that would be necessary. The fact that it is so rare is what kept the problem from being detected for weeks. However, as we noted in that earlier blog, given the broad swath of the Internet potentially querying the Chinese I-root instance, someone was bound to stumble on a bad DNS answer and, as a result, not be able to friend their pals. This is exactly what happened and is what brought the problem to light.

Second, the fine folks at Netnod, who provide the exceptional and free I-root service, vigorously defended their services in China, asserting they provide the same DNS answers regardless of location. We have no reason to think otherwise.

Third, it’s quite easy to see incorrect answers from DNS servers in China yourself, whether or not you happen to live there. This has nothing to do with any of the root name servers. Just pick your favorite DNS server based in China and ask it about Facebook. Here is an example of repeated queries from the Linux command line from a US-based machine to a China Telecom DNS server.

dig @dns1.chinatelecom.com.cn. www.facebook.com.
...
www.facebook.com.       11556   IN      A       37.61.54.158
www.facebook.com.       24055   IN      A       78.16.49.15
www.facebook.com.       38730   IN      A       203.98.7.65

None of these IP addresses has anything to do with Facebook. In fact, addresses starting with 37 haven’t even been allocated by IANA as of this writing.

Of course, if you don’t live in China, you probably don’t use a Chinese DNS server directly. The problem is that we all use the root name servers and they are spread throughout the world. Thanks to the vagaries of Internet routing, you may end up querying any of them, regardless of where you live and where they are hosted. Thus, if you live outside of China and just happen to query a root name server hosted in China, your queries will pass through what is known as the The Great Firewall, and hence will be subject to any restrictions it imposes.

Details, Details

While doing some research for next week’s NANOG meeting in San Francisco, we revisited the time line for the March I-root announcements from China and couldn’t help but notice the problem resurfacing on June 3rd. The I-root resolves to 192.36.148.17, which is announced by AS 29216 (which is dedicated to the I-root) as both 192.36.148.0/23 and 192.36.148.0/24. From there, these prefixes travel via Netnod’s AS 8674 and then onto the general Internet. Since Netnod anycasts these prefixes from dozens of locations around the world, we expect to see them via any number of BGP adjacencies to AS 8674 and, in fact, we do. Around 80 different ASes adjacent to Netnod’s AS 8674 see the two I-root prefixes and, in turn, propagate them onward.

What we do not expect to see are mainland Chinese ASes adjacent to AS 8674 propagating these prefixes outside of China. This is what we did see in March 2010 and it implies Internet users outside of China could be directed to the I-root instance inside of China. Unfortunately, this problem has returned. We see AS 8674 passing just 192.36.148.0/24 off to AS 24151 and then AS 7497, both of which are associated with the China Internet Network Information Center. From there, the prefix travels via Pacnet (AS 10026), formerly Asia Netcom, and PCCW (AS 3491) out to the general Internet. This started just before 10:20 UTC on June 3rd and is still ongoing as of the date of this blog.

As we noted last time, to get a bogus DNS response outside of China, you not only have to query the I-root, you have to query the Chinese instance of it. To measure potential impact, we looked at the originating country of all prefixes downstream of any provider selecting the Chinese I-root. We then computed the percentage of these relative to the total number of prefixes in the country. A graph of the top dozen from the March incident is shown below, followed by those from this current (and ongoing) incident.

China-Iroot.png
China-Iroot-new.png

Not surprisingly, most of the affected countries are in Asia, as before, but there are important differences from the last event. Russia, India and Taiwan all entered the top twelve, while Pakistan, New Zealand and Bangladesh have dropped out. The impact on the countries in both lists is roughly similar, except that US impact went up by a factor of 10. Potentially impacted US states include Florida and California, making up approximately half of the US total. In addition, Singapore increased from 73% to 96%.

Conclusions

Censorship is a fact of life on the Internet today. But unfortunately, given the open, trust-based nature of the network, such censorship can easily spread beyond its intended boundaries. While individuals can do little to avoid such issues, there are actions network and system administrators can take. Filtering root name server announcements with Chinese ASes on the path is one approach. Never querying the I-root is another. Such actions would guard against this particular problem, but probably not the next one — whatever it might be. Ultimately, we are all in this together. We depend on each country or organization not to inadvertently or intentionally interfere with any other. All other paths lead down a very slippery slope.


Share Now

  • To answer my question: There is no evidence that responses from I-root in Beijing are tampered.
    Editor’s note: Here is one from the day after our blog posting, where http://www.facebook.com returns an invalid IP address from Korea Telecom. We will be presenting our complete results at NANOG49 in San Francisco. See you there!
    # dig @i.root-servers.net. http://www.facebook.com. A
    ; >> DiG X.X.X >> @i.root-servers.net. http://www.facebook.com. A
    ;; global options: printcmd
    ;; Got answer:
    ;; ->>HEADER

  • Jaap Akkerhuis

    That one sees the China node from I-root outside china shouldn’t be a suprise to anyone. Things like that happen old the time. For quite a while I used to see the J-node from Korea instead of the much closer one from Frankfurt or here in NL.
    I fully agree that DNSSEC is the proper way to find out that there has been tampered with DNS reponses.

  • Thanks for clarifying. Hoped to see that detailed data inside of your report rather than in the comments later on though. 😉

Whois: Earl Zmijewski

Earl leads a peerless team of data scientists who are committed to analyzing Dyn’s vast Internet Performance data resources and applying their expertise to continually improve upon Dyn’s products and services.