It’s been an interesting year in many ways, not least of which for the Internet. This year, I started to contribute in earnest to the Renesys blog and back in January I was wondering “How am I going to find anything interesting to talk about on a regular basis? Nothing much happens on the Internet, right?” Well, it certainly did this year and now I’ve got many more ideas than I have time to research and write about. In hindsight, I guess it isn’t too surprising. As the world becomes more interconnected and more Internet-dependent, we’re bound to bump into each other more and expose the limitations of the current system. So let’s review what 2008 brought us and take a guess at what is in store for the new year.
The year started quietly enough and here in New Hampshire, we were largely trying to stay warm. Then the Mediterranean cables starting snapping and suddenly things got much more interesting. Entire countries and providers were cut off from the Internet and there was a mad scramble to restore connectivity to the area. As always, there were winners and losers, but the main lessons countries and businesses in the area learned were: “stuff happens”, don’t put all your eggs in one basket, and you get what you pay for. Of course, these cliches are not very sexy, so the conspiracy theorists, apparently having never seen bad weather or a boat anchor, were only too quick to claim that all these cable breaks were a prelude to war, only they weren’t. No new wars were started in January, the cables were ultimately repaired, and a lot of folks in the region started planning for failure by building some redundancy.
Just after the Mediterranean cables were repaired and we were starting to wonder if it would ever stop snowing, Pakistan had another bad hair day and their local incumbent (Pakistan Telecom) inadvertently knocked YouTube off the global Internet for about 2 hours. This incident got a lot of press attention, and there was considerable hand-wringing about this grave new threat to the Internet, despite the fact that such incidents have been taking place for over a decade and are both inevitable and unavoidable in a trust-based system with no central authority. So unfortunately, no one fixed the Internet in 2008.
It’s still snowing and the locals are starting to come a bit unglued. New Englanders are buying seeds and starting to walk around in shorts in a vain attempt to defy the weather gods. And our cooperative trust-based Internet dramatically showed another shortcoming when TeliaSonera and Cogent depeered, partitioning the Internet for those dependent on one or the other of these providers. To name just two of the many victims, this event affected both Martha Stewart’s corporate headquarters, single-homed behind Cogent, and Blizzard Entertainment’s World of Warcraft servers, single-homed behind TeliaSonera. In other words, the Internet was broken in lots of little ways: Martha wouldn’t have been able to play a computer game, Europeans might not have been able to reach Cogent-hosted content, and so forth. But then after two weeks, this particular episode of Internet chicken ended when the two carriers reestablished connectivity, and the Internet was once again whole.
Mud season. The snow is melting fast now and those quaint dirt roads we are famous for are turning to rivers of mud. It was a slow Internet month, but thankfully the start of Todd Underwood’s musical career gave us something to write about. Mercifully, both the season and the career were short-lived.
The ice is breaking up on the lakes and Joe’s pond had their official iceout at the end of April. And while that happens every year, we did report something truly bizarre on the Internet, brought to our attention by ICANN. ICANN, responsible for one of the 13 DNS root name servers, had recently changed the IP address of the server under its stewardship, a switch undoubtedly missed by most of humanity. Then bogus root name servers started appearing on the old IP and were happily providing answers for up to six months before anyone noticed. Although we are far from certain, it appears that no harm was done in this case, but the potential for mischief was considerable. Another contest of egos? As Rodney King once said “Why can’t we all just get along?”
All the traces of snow and ice are now gone, so it’s time for black flies to make their annual appearance. These pests feed on human blood and like to rip out chunks of your flesh near a hairline, which you notice only after blood is streaming down your face. Having the ground covered up for the half the year suddenly doesn’t seem so bad. We’re still wondering what exactly happened with the ICANN root name server, but unfortunately have more questions than answers. At least I start to monitor all of the root name servers from my Renesys Routing Intelligence account, as I might need new blog material one day! Besides, someone should be watching over our critical infrastructure and it might as well be me. The year isn’t even half over and we’ve seen a pretty incredible assortment of major headline-grabbing events, all pointing to serious flaws in how the Internet actually works. On the business front, Cogent became transit-free this month, meaning all the big boys really do need to peer with them to prevent a partitioning of the Internet, like it or not.
Ahh, summer. We get exactly one month of summer in New Hampshire and we aren’t going to spend it writing blogs. Besides, not much happened. Oh, wait, there was Dan Kaminsky’s revelation of a DNS implementation vulnerability so bad that DNS servers all over the planet were trivially subject to DNS cache poisoning. Despite the frantic vendor response and software upgrades, the vulnerability is not resolved, nor resolvable given the current domain name system. Maybe we can finally get widespread interest in DNSSEC. Otherwise, it was a quiet month and we worked on our tans.
Our long “summer” is finally winding down and by the end of the month we could come close to the freezing mark again. While no one was looking, Tony Kapela told the Defcon 16 crowd how to steal the Internet, a technique for transparently intercepting anyone’s incoming Internet traffic before eventually passing it on to the rightful owner, and doing so in a way that can be largely invisible to the victim. As if that wasn’t bad enough, Russia also invaded Georgia. We expected this to have a rather negative impact on Georgian Internet connectivity. Only it didn’t. Sure there were website defacements and some DoS attacks, but that pretty much happens every day in every country. Despite constant calls from the media, we really couldn’t find any changes to Georgia’s Internet presence or any significant or prolonged losses of connectivity. Georgia’s meager Internet infrastructure simply wasn’t blown up.
The days are getting noticeably shorter and much cooler. Thankfully the bugs will all soon be dead, as this was a particularly bad year for winged pests. But as New England cools, the Gulf coast is just heating up with one hurricane after another. Gustav started out looking like a replay of Katrina, but in the end, Louisiana wasn’t hit as hard and was much better prepared for Katrina’s little brother. The real loser in the hurricane lottery for 2008 was Texas, which was hit very hard by Ike. After rolling through Texas, Ike still packed enough punch afterward to cause extensive Internet outages throughout the US. We ended the destructive month with a good old-fashioned public stoning: the notorious Intercage was knocked off the Internet by their irate providers, only to briefly rise from the dead before being taken down for good.
It’s leaf peeper time and while the bugs might be dead, a new scourge has arrived on the land: tour buses. As if these flatlanders weren’t entertaining enough, Sprint depeered Cogent at the end of the month, resulting in a partitioned Internet for the second time this year and impacting a few thousand customers of each carrier. Perhaps Sprint upset the wrong customer, since the peering link was quickly restored over the weekend, when upper management and their lawyers were unlikely to be working. I can’t help but wonder where the phone call came from and to whom in Sprint’s organization, but Sprint clearly didn’t hesitate long to blink. I guess some customers are always right. Also this month, Outpost24 claims to have discovered a way to trivially DoS anyone by exploiting some TCP vulnerabilities, but details remain murky.
No leaves, No snow, No-vember. At least the tourists are gone and the growing darkness gives us time for reflection. Even if you ignore all of the security concerns, the Internet is still heading for a train wreck. We’re running out of usable IP space and the providers are all going broke. If you follow any of the links in this blog, it should be the preceding three for Todd Underwood’s insightful comments on the industry and difficulties that lie ahead. After reading these, you won’t be surprised when the Internet doesn’t work. You’ll be thankful every day that it does. Speaking of which, we almost had another Internet-sized blowout when Companhia de Telecomunicacoes do Brasil Central leaked a “full table”. Had this spread, Brazil would have ended up on the receiving end of much of the traffic on the global Internet, which would not have been a good thing for anyone. As it turned out, we dodged the bullet this time and the damage was very contained. And the month saw another public flogging of an Internet bad boy; this time it was McColo’s turn.
Darkness. The earliest sunset is now at 4:12pm, but when it’s cloudy, it’s more like 3:12pm or even earlier as the sun doesn’t get very high in the sky now. We spend way too much time indoors, looking at computer screens. This problem is partially solved for us when a ferocious ice storm knocks out power to more than half of New Hampshire. Three Renesys employees were without power over a week later. Jim Cowie, was one of them, but he refused to despair over his own homelessness, not to mention the growing global economic gloom, and wrote a hopeful and interesting blog about a possible economic stimulus that does not involve bailing out dying industries. Since the year was drawing to a close, those of us with power also took the opportunity to review how the top global Internet service providers fared in 2008. In short, if you figured out that more than half the planet lives in underserved Asia and built your business plan around that part of the world, you probably did alright, at least in terms of gaining market share. (And we were happy to see that most of the winners were also users of Renesys’ Market Intelligence product.) Finally, it was Deja Vu All Over Again when more Mediterranean cables broke to provide a nice symmetry to the year.
The Year Ahead
I predict that New England will have a banner ski season this winter. Although it is only December as I write this blog, I’ve observed three leading indicators: I can reach my mailbox from above, the picnic table in my backyard is a ski hazard, and the local firefighters have been observed patrolling the neighborhood looking for lost fire hydrants. In other words, we’ve had a lot of snow for this early in the season.
As far as the Internet goes, it’s a different story. If you managed to get through all of the above and maintain your sense of optimism, I applaud you. It’s really not a very pretty picture, nor am I able to find many silver linings. So what will 2009 bring for the Internet? It’s probably pretty safe to say that things are going to get worse before they get better. With the world economy in turmoil, don’t expect your service to improve. And with all the incentive for ill-gotten gain in a deteriorating economic situation, don’t expect much relief on the security front either. Major new vulnerabilities will continue to surface and older ones will be more fully exploited, as the inherent trust-based model of the Internet continues to crumble. Nation-states will beef up their cyber warfare capabilities. And I wouldn’t be surprised to see a major telco declare bankruptcy and at least another major de-peering event. In other words, I don’t expect 2009 to look much different than 2008. To get out of this mess, we’re going to need some global leadership and a game-changing shift in both the economic and security models surrounding the Internet. If things get bad enough, maybe we can at least start a serious discussion around that in 2009.