Just when it seemed things couldn’t get any worse for President Obama’s recently-appointed FCC Chairman Tom Wheeler, the fight over “net neutrality” and the fate of the private Internet has now become a political football in gridlocked Washington.
Days after the mid-term elections, the President posted a video strongly advocating his own “plan” for net neutrality rules, upping the volume on this year’s loud, confusing, and often intentionally misdirected debate on how best to maintain the open Internet.
The FCC, which by law is independent of the White House, has been considering new rules since a federal appeals court in January largely voided on jurisdictional grounds an earlier effort from 2010. In May, based on guidance from the court, the agency proposed a revised version that differed only slightly from the 2010 rules.
In the meantime, the FCC’s proceeding has been overshadowed by a frenzied media circus that casts the agency, and Chairman Wheeler, as conspirators in a plot to destroy the Internet.
Now, unrefuted evidence has emerged that the unraveling FCC proceeding, almost certain to land the agency back in court for a third time, initially spun out of control based on wildly misread data on early 2014 network performance issues—an error that at least one leading industry analyst suspects may not have been accidental.
The Plot Thickens
The story begins back in March, when the strictly legal debate over the FCC’s limited authority over broadband ISPs first turned toxic with inflammatory statements from Netflix CEO Reed Hastings on the company’s blog.
The 2010 rules, even if reinstated, were too “weak,” Hastings wrote. “A stronger form of net neutrality is required,” he insisted, to “prevent ISPs from charging a toll for interconnection to services like Netflix” and other dominant content providers. To protect the Internet, he wrote, the FCC must force ISPS to provide Netflix “sufficient access to their network without charge.”
At the time, Netflix was concluding a series of agreements with leading ISPS to directly interconnect Netflix’s proprietary content delivery technology with their networks, as other large content providers had long done. Before that, and to keep up with rapid growth, Netflix has been paying third-party transit providers including Cogent and Level 3 and general purpose content delivery networks, which are provided by companies such as Akamai and Limelight.
Hastings, dissatisfied with the negotiations, urged the FCC to redefine net neutrality, transforming it from a set of last-mile consumer protections to detailed government control of connections at the Internet’s back-end. Rather than pay the transit providers, Netflix wanted to connect directly to the ISPs and do so “without charge.”
And Hastings demanded that the FCC make such arrangements a matter of federal law.
To emphasize the need for FCC oversight, Hastings insisted that ISPs were intentionally “constraining” Netflix traffic to force the company to upgrade its connections, “sacrific[ing] the interests of their own customers to press Netflix and others to pay.” (Netflix did not respond to requests for comments on this story.)
That claim quickly upended the on-going FCC proceeding. Soon after, comedian John Oliver launched his satirical tirade against cable company interference with Internet traffic, prominently featuring the Netflix-supplied data.
Comments began flooding into the FCC’s pending Open Internet proceeding, derailing efforts by Wheeler to respond quickly, as he promised, to the appeals court’s “invitation” to clear up the few remaining issues with the on-going rulemaking and move on to more urgent work.
But Netflix had fatally misread the data.
Earlier this month, Frost & Sullivan’s Dan Rayburn, a leading media industry analyst, reported damning evidence that Hastings’s claims of ISPs throttling were untrue all along, based on a fundamental misidentification of the cause of measurable traffic congestion being experienced at the time across the Internet.
There was intentional throttling going on, Rayburn reports. But it was not being done, as Netflix claimed, by Comcast or other large ISPs, intentionally or otherwise.
The congestion, rather, resulted from a calculated choice made by Cogent, Netflix’s own Internet transit provider. Cogent, it turns out, had implemented a practice of prioritizing the traffic of its retail customers over that of its wholesale customers, including Netflix, during times of heavy network usage that strained Cogent’s capacity to deliver the traffic being pulled by end-users.
Faced with irrefutable evidence reported by Rayburn and others, Cogent quickly admitted to intentionally slowing the traffic of all its wholesale customers, a practice that may still be in place.
Cogent explained in a blog post that “retail customers were favored because they tend to use applications…that are most sensitive to congestion” and that in response they implanted a “structure” that “impacts interconnections during the time they are congested.”
According to Rayburn, Cogent never publicly disclosed that it was intentionally prioritizing outgoing traffic of its retail customers, in violation of industry practice (and possible contractual responsibilities with its wholesale customers).
And its own policy: Cogent’s website proudly proclaims “Cogent practices net neutrality. We do not prioritize packet transmissions on the basis of the content of the packet, the customer or network that is the source of the packet, or the customer or network that is the recipient of the packet.”
The failure to disclose the practice even as the FCC proceeding spun out of control because of it was particularly damaging. As Rayburn notes, “What Cogent did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates.”
It was Cogent’s undisclosed actions, in other words, that resulted in a widely-reported slowdown of Netflix’s outgoing Internet traffic.
The slowdown had nothing to do with the lack of enforceable net neutrality regulations. And it said nothing about the potential danger of so-called “paid prioritization” arrangements that have been the rallying cry in this round of the ten year old net neutrality debate.
Quite the opposite. It proved that network management, absent intervention from federal and state regulators, rapidly resolved its own problems through private agreements and new technologies. If anything, it demonstrated the value of an FCC rule, upheld by the court, that required transparency for such practices—rules that have previously not extended to transit providers such as Cogent.
Enter Netflix, Screaming
Though the mainstream media and average consumers have only just joined the story, “net neutrality” has been for years a favorite rallying cry for self-proclaimed consumer advocates who have used it as shorthand for a range of efforts to introduce government regulation—or outright nationalization—for fast-changing Internet technologies, both at home and abroad. (See my earlier post, “The Biggest Net Neutrality Lie of All”.)
Yet as every consumer knows, despite regular claims of imminent doom, those technologies have continued to improve and evolve, superbly regulated by the engineering groups who maintain and update the underlying standards and protocols that define it.
Under a legal structure enacted by a bi-partisan coalition of Congress during the Clinton Administration, both the FCC and the Federal Trade Commission provide backup for potential anti-competitive practices under a “light touch” model of regulation.
Against that backdrop, Netflix’s dramatic entry into the net neutrality maelstrom earlier this year was both sudden and, so far, largely counter-productive.
Back in 2010, the last time the FCC was testing the limits of its legal authority over broadband ISPs, Netflix was largely absent from the debate. The company was just becoming the dominant source of Internet traffic that it has since solidified. Netflix was then in the process of winding down its original service: mailing DVDs via a sweetheart deal with the U.S. postal service that, in an irony that may be lost on the company, has since been held to have unlawfully discriminated against similar services.
In 2011, the company announced plans to spin off the DVD business into a separate company, but consumers and investors revolted against the idea, and the plan was quickly dropped.
Since then, the company wisely shifted its strategy to focus on digital distribution, a move that proved wildly successful. Today, the company has over 35 million subscribers in the U.S. alone. At peak viewing times, the company’s streaming movies and television shows make up as much as a third of all Internet traffic in the world.
As the need to provide more and more video at higher resolution strained the company’s Internet infrastructure, Netflix developed Open Connect, the company’s proprietary content delivery system. And it began flexing its growing competitive muscle by insisting that ISPs either co-locate Open Connect for free or be denied access to HD programming. Like its earlier preferred deal with the postal service, and unlike every other CDN large and small the company has insisted that its equipment be accommodated “without charge.”
In the U.S., however, some large ISPs balked at the idea of offering Netflix preferential access. In many cases, Rayburn told me, the interconnection takes place in third party co-location “hotels,” a service for which both parties must pay.
When the January court decision reopened the question of whether the open Internet required regulatory intervention, Netflix injected itself into the new FCC proceeding, hoping to gain leverage in the stand-off. It’s in that context that Hastings argued the FCC rules should mandate free interconnections and accommodation of unlimited capacity, if not for all content providers, than at least for Netflix.
The Interconnection Misdirection
Netflix’s call for regulatory intervention to help manage its costs is nothing unusual, nor illegal. Indeed, as startups offering ride-sharing, temporary housing, commercial drones, driverless cars, Internet TV, genetic testing and crowdfunding platforms have all found, it is an accelerating and alarming trend among incumbents faced with disruptive innovations.
Rather that compete evenly with the startups, they urge regulators to stifle or stop the innovators by forcing them to abide by laws written for earlier technologies.
In filings with the FCC, Netflix argued the FCC should assert authority it still maintains over the declining switched telephone network, authority granted nearly a century ago for what was then a legal monopoly granted to the former Bell System.
In a legally-dubious process that advocates euphemistically refer to as “reclassification,” the Internet would be turned into a public utility, granting the FCC and state regulators vast powers to oversee any and all aspects of its deployment and operation. (In his plan, Obama explicitly calls for the public utility approach, without addressing any of the downsides that are well known based on the failings of existing utilities and their regulators.)
But Netflix IT costs aside, the case—legal and economic—for an FCC transformation of the Internet into a public utility is extremely weak. The Internet transit market has worked brilliantly with almost no government oversight since the beginning. According to the multinational OECD, over 99% of all peering agreements are so simple they aren’t even reduced to writing.
“Net neutrality” is also a dangerously simplistic term, one concocted by legal academics rather than network engineers.
In fact, during the almost twenty years when the FCC had no enforceable net neutrality rules in place, new network management technologies including co-located servers, content delivery networks, virtual private networks and other specialized services have evolved to handle traffic that has increased exponentially.
Networks must be flexible in order to handle rapid changes in consumer behavior, especially as the Internet is dominated more and more by video and other applications that require high bandwidth and low latency to maintain their quality.
Comcast CEO Brian Roberts
Misunderstandings of modern network engineering by lawyers explains much of the current public confusion. When Netflix announced that Comcast’s alleged throttling had forced it to switch its transit from Cogent to a direct connection with Comcast, for example, Columbia law professor Tim Wu, who coined the phrase “net neutrality” in 2003, claimed it was “the first-ever direct interconnection deal between a broadband provider, like Comcast, and a content company, like Netflix or Google.” The beginning, once again, of the end.
But that too proved to be wildly inaccurate. As Rayburn noted in May, nearly every major content provider, including Apple, Amazon, Facebook, eBay, and Google had long since established such deals with nearly every large ISP and backbone provider. Not because they were forced to, but because such deals made good technical and business sense.
Direct Interconnections as of May, 2014
(Source: Streaming Media)
(Rayburn’s chart shows the extent of direct interconnection between large content providers, who generate significant Internet traffic, much of it video. Rayburn notes “I didn’t look at every ISP out there or every content owner, simply some of the larger ones.”)
Indeed, despite its public rhetoric, Netflix privately acknowledged the unremarkable nature of these deals--and their cost. At a June event sponsored by the Aspen Institute in Washington, as reported by fellow Forbes contributor Hal Singer, a Netflix representative admitted that the price the company was paying Comcast to connect directly to its network was too trivial to report, or to serve as a source of competitive marketing. (Content licensing from copyright holders accounts for the vast majority of the company’s expenses.)
Tarred with Cogent’s Brush
Neither Netflix nor Comcast released details of their particular deal, but Rayburn estimated at the time that Netflix may actually be paying Comcast less than it previously paid to Cogent, and under a long-term contract that protects Netflix from price increases. (Netflix did not respond to a request for details of the deal.)
Still, before the ink had dried on its agreement with Comcast, Netflix’s Hastings began trashing it, claiming in March they had been forced to pay a special “ISP toll” or face continued and intentional degradation of their traffic.
In a follow-up blog post in April, the company’s vice president of content delivery went farther, writing that “Netflix agreed to pay Comcast for direct interconnection to reverse an unacceptable decline in our members’ video experience on the Comcast network. These members were experiencing poor streaming quality because Comcast allowed its links to Internet transit providers like Level3, XO, Cogent and Tata to clog up, slowing delivery of movies and TV shows to Netflix users.” (emphasis added)
The post included a now-infamous chart purporting to show how Comcast had intentionally slowed Netflix traffic prior to the switch from Cogent to the direct connection. “Comcast’s ability to constrain access to Netflix can be clearly seen in the following chart,” the company wrote, “which shows how Netflix performance deteriorated on the Comcast network and then immediately recovered after Netflix started paying Comcast in February.”
(Source: Netflix)
But the Netflix data proves something very different. It proves the catastrophic effect of Cogent’s unilateral decision to prioritize the traffic of its retail customers over its wholesale customers, including Netflix. As Rayburn’s research reveals, the chart reflects the intentional prioritization scheme Cogent now admits to.
Netflix quality improved after establishing direct interconnection with Comcast, in other words, just as the chart shows. But only because Cogent had been disfavoring wholesale traffic en route to the ISP. Once Netflix took Cogent out of the loop, the problem went away. Immediately.
Netflix repeated the false claim against the ISPs as recently as a July filing with the FCC, which argued that the only way the FCC could protect consumers from continued “interconnection congestion” was to transform the Internet into a public utility. When asked if Netflix wished to revise its claims that it was ISPs rather than its own contractor who was intentionally creating the “clog up,” the company did not respond.
Cogent also did not respond to a request for comment for this story, but told a reporter for Ars Technica earlier this month that its decision to intentionally disfavor wholesale customers during times of congestion on its network was “put in place only because Internet service providers refused to upgrade connections to Cogent to meet new capacity needs.”
Rayburn finds the timing of Cogent’s prioritization suspicious. “It seems extremely unusual that Cogent implemented this traffic change during the very same week that Comcast announced the Netflix deal,” he wrote.
The Damage Done
Since Rayburn’s report became public earlier this month, Netflix, for its part, has said nothing.
But even if the company was unaware of what was really happening earlier this year, the company’s call for FCC intervention in its network business dealings has nonetheless profoundly skewed the net neutrality debate, perhaps in ways the company neither intended nor desired.
In particular, since Hastings’s original post and follow-up outreach through public interest proxies, there is now widespread confusion over what’s at stake in the FCC’s on-going Open Internet proceeding.
Whether by design or accident, most mainstream reporters and consumers commenting in the FCC docket erroneously point to what has now been revealed as Cogent’s intentional degradation of Netflix and other wholesale traffic as proof that ISPs have both the power and intention to force content providers to pay for prioritization, or what is sometimes inaccurately characterized as Internet “fast lanes.”
(President Obama’s plan, for example, calls for an explicit prohibition on paid prioritization. Obama explains: “No service should be stuck in a ‘slow lane’ because it does not pay a fee. That kind of gatekeeping would undermine the level playing field essential to the Internet’s growth. So, as I have before, I am asking for an explicit ban on paid prioritization and any other restriction that has a similar effect.”)
But paid prioritization has nothing to do with the kind of network management Netflix accused ISPs of exploiting—and which turns out not to have been taking place in the first place.
Paid prioritization has always referred instead to a theoretical service in which content providers would pay ISPs to have their packets delivered to consumers more quickly than those of other providers (including competitors) when traveling the last mile from the ISP’s equipment to the consumer’s device. Such a service would prioritize traffic, in other words, only after the packets had reached the ISP—not at the point of interconnection with other networks, including those of transit providers such as Cogent.
To date, no ISP has ever offered such a service, and several have long indicated they have no interest or intention to do so in the future.
Under the terms of its merger with NBC Universal, in fact, Comcast is still legally bound to a stricter version of the non-discrimination rule from the 2010 Open Internet order that the court rejected. AT&T has volunteered to a prophylactic ban on paid prioritization, and has offered to commit to it legally by making it a condition of its pending merger with DirecTV.
Though the 2010 rules did not explicitly prohibit paid prioritization in the future (it may, in fact, prove useful for future applications requiring such priority, including emergency applications and remote health services), the D.C. Circuit court ruled in January that the FCC’s wording of the “non-discrimination” rule exceeded the agency’s limited authority.
So in May, the agency sensibly proposed slightly different terminology already approved by the same court for rules in effect for data roaming. Still, and despite self-serving claims that the agency was instead proposing to “authorize” or “mandate” Internet “fast lanes,” the Commission made clear its view that a prohibition on network management practices that were “commercially unreasonable” (the revised phrase) would include paid prioritization.
Consumer groups, however, seized on the unrelated issues of transit and co-location raised by Netflix’s complaints about intentional “clog ups” as proof that the revised rules were insufficient—or worse, a conspiracy to “kill net neutrality.” For most of this year, reports on the FCC proposal have hopelessly and opportunistically conflated prophylactic last-mile “prioritization” bans with back-end transit arrangements.
We now know not only that Netflix’s traffic management issues had nothing to do with paid prioritization, but that they were also the fault not of any ISP but of its own business partner.
The damage, however, has already been done. And if the end result is the insertion of the FCC into every link in the Internet’s efficiently-engineered architecture, Netflix may soon regret its calls for government help.
Unfortunately, and even more likely, so will Internet users. Whether Netflix customers or not.
My new book, co-authored with Paul Nunes, is “Big Bang Disruption: Strategy in the Age of Devastating Innovation” (Portfolio 2014). Follow me on Twitter and Facebook for more on the accident-prone intersection of technology and policy.
