The internet’s best job — traffic cop


It may not be at the top of a police department’s org chart, but on the internet, traffic cop is one of the best and most coveted positions there is.

There are somewhere around one billion websites at present, with some three billion people accessing them across more connected devices than there are human beings on the planet. To say the internet is a vast place is an understatement. As a result, the people or companies that help others find what or get to where they want on the web are in positions of immense authority. And if they figure out ways to monetize that authority, they turn into Google (with an estimated worth around $350+ billion) or Facebook ($200 billion) in a relatively short amount of time.

Susan Athey is the Economics of Technology Professor at the Stanford Graduate School of Business, where she studies industrial organization, microeconomic theory, and applied econometrics. Athey has been focusing on how the internet, as one of the world’s largest repositories for information, is mediated by traffic directors and the effect this has had on the news media, which she discussed at Harvard Business School on April 8, 2015 as part of the Digital Initiative’s Digital Seminar series.

“Early on, we thought that the internet would make it so easy to find things.”

“Early on, we thought that the internet would make it so easy to find things,” Athey said. “Search costs would go down, life would be good. What we collectively missed was yes, there is so much available, but it’s impossible to find any of it without someone helping you. For example, you don’t read a single piece of news without someone or something recommending it to you.”

It used to be that large news organizations like the New York Times or the Washington Post controlled steady, centralized traffic through their websites. They were the ones doing the recommending on their homepages. But with the proliferation of social media and search those news organizations lost much of their authority, and with it much of their revenue, even as they continue to bear the cost and work of producing that information today.

Users now come to news sites as much or more through links on Facebook, or through news aggregators like Google News or the Huffington Post, and from many different devices. Tracking them has become difficult or impossible, which in turn makes it difficult for news organizations to sell effective advertising. Why buy ads on the New York Times to reach a fraction of users when you can buy customizable ads on Google and reach most or all of them?

“If you want to reach everyone in the U.S. for sure, you’re now going to go to Facebook or Google,” Athey summed up. “That’s what they push to advertisers and what makes them so successful: reach and tracking.”

This is why news organizations have had such a tough go of it in recent years; most are no longer positioned in the middle of the internet’s intersections.

This is why news organizations have had such a tough go of it in recent years; most are no longer positioned in the middle of the internet’s intersections. At best, they are service stations at which users stop along their online travels, and few follow their directions. Athey points out, however, that a curious thing happens when this process is allowed to play out in full?—?aggregators and search giants profit in the short term, but effectively kill off the very news organizations on which they rely in the longer term (Athey recalls Rupert Murdoch once referring to aggregators as parasites; in thriving themselves, they eventually kill off their hosts).

No one has yet discovered a foolproof solution to the problem. Spain took the bold step of requiring Google to share revenue with the news organizations it was crawling and including in its results. When it refused, Google News went offline in the country, which allowed Athey to do some compelling research. She found that without Google News, users did more direct navigation to news sites and had a higher reliance on search to find information, and the top news websites saw a corresponding uptick in traffic. It wasn’t all good news, though, as news consumption on the whole fell, leading Athey to believe that when you take away aggregators, people will actually read less.

“Google News certainly helps you find information,” Athey summed up. “But it also potentially changes the incentive structure to produce good news.”

Athey’s deductions did not lack for sample size. The raw data she used was measured in petabytes, which she reduced through careful calculations and elimination to terabytes and then gigabytes. Its scope also allowed her to train her proverbial microscope on social media, which led to some interesting findings. Among them, social media tended to favor articles that were more positive, compassionate, controversial, or targeted at women or parents.

“So, an article about a person hiding in the basement from the tornado would do better on social media, not the piece with the facts about the tornado,” Athey quipped. “People read more objective news when they go to a news website, but when they’re on Facebook they click what their friends share, which is often more opinionated.”

The findings were important because while they showed that social media isn’t necessarily changing the distribution of topics in the news on the whole, it may over time lead to more biased news consumption on a person-to-person basis. It also reified the power that larger, algorithmically-based tech giants like Google and Facebook have, “which could be good if it engages you more in news,” Athey said. “But it could also be bad if it engages you more in only a specific type of biased news.”

On the whole, Athey said that traffic giants and news organizations both bear a responsibility to look at the short and long term effects of their actions and carefully choose the direction they want to head in. Becoming more data-driven is not necessarily a bad thing for a news organization, for example, but it will necessarily change the scope and breadth of their coverage, and poses different challenges than it would for a strictly algorithmic organization like Google.

“The more data-driven you are, the more you’re going to respond to short term needs,” Athey said. “At that point, the challenge becomes how you’re going to manage those needs against longer term considerations like brand equity and image. The math says you should put more weight on things that are precisely measured, but that has a lot of unintended consequences that are sometimes tough to measure. If you’re just producing things people are going to click on, you become undifferentiated. And if you’re undifferentiated, that’s going to affect your profits in the long term.”

Authored By

  • Christian Camerota

    Christian is the director of editorial content strategy and creation at Wesleyan University.


  • Susan Athey

    Susan a professor at Stanford Graduate School of Business.