Internet Performance Delivered right to your inbox

Peering—The Fundamental Architecture of the Internet

Peering matters. Earlier in the year, Cogent and Level (3) had a wee tiff over peering. Level (3) turned off its connections to Cogent for a few days as part of a strategy of negotiating new terms of interconnections. Politicians clamored to offer new, and ill considered regulation of large-carrier interconnection. In the process, politicians and much of the media revealed the depths of their ignorance about how the Internet works.

The organization, character and structure of net interconnections affect everything about how the Internet works (and sometimes doesn’t work), and it is at the core of the work that we do here at Renesys. Peering (settlement-free interconnection) is a significant part of that. Without making sense of peering now, almost nothing else I write about or refer to in the future will make sense. So what is peering and why does it matter so much? What does it have to do with Internet architecture?

The Internet is disorganized. It’s messy. It’s cluttered. There is no dial-0-for-operator, no single “Internet company”, no one true map of how it is put together and how it works. Whole companies (Yahoo!, Google) are built out of the seemingly simple task of trying to list what is actually on a single application of the Internet (the web). Many people know something about DNS, the Domain Name System, which is hierarchically organized, and mistakenly assume that the rest of the Internet works that way, too. It doesn’t. This is a human-designed system that shows very little signs of intelligent design.

The Internet works by voluntary association of private parties–it’s a Libertarian’s dream. Individuals and companies run networks. They build them and sell (or give away) access to them to their customers, according to some business model. They then interconnect their networks with other companies’ networks, either for free or for money. When the links are free, those are called “settlement-free interconnections” or “peering”. When the interconnections are paid for by someone, these are called “transit”.

With no governing regulation, everyone connects to others, either for free or for money, whenever it is mutually beneficial. That’s it. The whole story. It’s a very simple, clear system. And It works. Many of the big-government crowd are shocked to realize that out of this chaotic system a single, fully connected Internet has arisen: almost everyone can reach almost everyone else, almost all the time. The sheer value of the connected Internet forces providers to offer full connectivity to the rest of the Internet. If they didn’t, any customer with a choice would leave in droves for providers who were offering “the real Internet”. And presumably customers without choice would create a market demand for a provider who did provide access to the whole Internet.

Providers used to offer partial networks, back in the dark ages of packet-switched communications. Compuserve, Delphi, GEnie, and hundreds of stand-alone BBS (Bulletin Board Servers) offering service to hundreds of thousands of customers. All of these organizations offered a dialup, online experience that was completely disconnected from every other company doing the same thing. And over time, they all got the Internet religion. Because their customers demanded it.

The evidence is that the market will take care of maintaining the full connectivity of the Internet, but it won’t always do so on your timetable or mine. Witness the Cogent/Level (3) peering dispute. Level (3) peered with Cogent, apparently settlement-free. Level (3) is a “default-free” network, which just means that it is so large that it connects to every other “default-free” network for free. There is no company that Level (3) pays for Internet transit. Cogent is in the challenging position of being almost “default-free”. They have settlement-free interconnections from lots of big carriers, but they still pay Verio (AS2914) for transit to the ones that they’re missing (mostly Sprint (AS1239)). So when Level (3) disconnected Cogent, in theory, Cogent could have reached Level (3) via Verio. They just chose not to. In theory Level (3) could have contracted for transit to reach Cogent via someone else, too. They also chose not to.

And so during that period, customers who were single-homed on the Level (3) network (that is, customers who had no connectivity from any other provider) could not reach customers who were single-homed on the Cogent network. That was a fair number of people. We presented a report about this at NANOG 35 quantifying the damage. It only lasted for a few days and during that time both parties were subject to intense pressure, by the public and most importantly by their customers, to fix the situation.

And so they did. Level (3) and Cogent have since resolved their dispute. Peering disputes are part of the life-cycle of theInternet. Many of the people writing nonsense about this dispute simply lack history and perspective. This dispute was not unprecedented; it was not new; it was not all that catastrophic. It will also not be the last time two big networks go head-to-head to decide what value each of them gets out of the connection between them. This isn’t an aberration, folks. This is how it is supposed to work. The only way for end-sites to protect themselves is to multi-home (that is, connect to more than one provider with routing redundancy). That’s a subject for another time.

Would it be possible to design a network along different organizing principles? Sure. Just make different choices. Centralize control over naming and interconnection. Regulate quality of service and the financial relationships among the parties. For simplicity, you may even want to just have state-owned corporations run the infrastructure, since you’ll be regulating the service offerings, capital investment and pricing anyway. That would certainly work. In fact, it does work: it’s called the telephone system. There are some huge advantages that the phone system has over the Internet: reliability, stability and predictability among them. But those advantages come at a cost–a very high cost. The phone system is expensive and inflexible. It does what it was designed to do but little more.

Right now we have a single, connected Internet. But the only thing that keeps that so is the ongoing, voluntary, self-interest of all of the participants. Sure, I could go off and start a non Internet, IP network, with my own rules and my own customers. I’ll call it ToddNET. It’ll have email and instant messaging to other ToddNET customers and partners. You’ll be able to buy stuff (from me). And it’ll be the same price (or maybe slightly cheaper) than access to that other Internet. Want to sign up for service? I didn’t think so. Self-interest is a powerful force–far more powerful than the kind of regulation that has been suggested.

With these very broad outlines of peering on the record, it should be easier to understand some of the fascinating stuff ahead. For example, two mega-mergers are in progress: SBC (AS7132 and others) has bought AT&T (AS7018 and many others); and Verizon (AS19262 and many others) is buying MCI (AS 70[123] and many others)). The merger conditions contained explicit requirements for peering. I think that this is the first time that the US Federal Communications Commission has regulated the conditions of network interconnection for Internet services. We’ll look at these conditions, the choices that they present to AT&T and Verizon, and the likely impact on networking in the future.

Addition: I was remiss in failing to mention Geoff Huston’s excellent work on this subject. That’s probably a better article to read to understand peering than this one.


Share Now

Whois: Dyn Guest Blogs

Oracle Dyn is a pioneer in managed DNS and a leader in cloud-based infrastructure that connects users with digital content and experiences across a global internet.