Written by Caitlin Petre

Abstract

Drawing on Caitlin Petre’s ethnographic study of Chartbeat, Gawker Media and The New York Times, this chapter explores the role of metrics in contemporary news production and offers recommendations to newsrooms incorporating metrics into editorial practice.

Keywords: metrics, analytics, newsrooms, journalism practice, ethnog- raphy, editorial practice

On August 23, 2013, the satirical news site The Onion published an op-ed purporting to be written by CNN digital editor Meredith Artley, titled “Let Me Explain Why Miley Cyrus’ VMA Performance Was Our Top Story This Morning.”1 The answer, the piece explained matter-of-factly, was “pretty simple”:

It was an attempt to get you to click on CNN.com so that we could drive up our web traffic, which in turn would allow us to increase our advertisingrevenue. There was nothing, and I mean nothing, about that story that related to the important news of the day, the chronicling of significant human events, or the idea that journalism itself can be a force for positive change in the world. . . . But boy oh boy did it get us some web traffic. (Artley, 2013)

The piece went on to mention specific metrics like page views and bounce rates as factors that motivated CNN to give the Cyrus story prominent home page placement.

Of course, Artley did not actually write the story, but it hit a nerve in media circles nonetheless—especially since a story on Cyrus’ infamous performance at the MTV Video Music Awards had occupied the top spot on CNN.com and, as the real Meredith Artley later confirmed, did bring in the highest traffic of any story on the site that day. The fake op-ed can be interpreted not only as a condemnation of CNN, but also as a commentary on the sorry state of news judgement in the era of web metrics.

Media companies have always made efforts to collect data on their audiences’ demographics and behaviour. But the tracking capabilities of the Internet, as well as the ability to store and parse massive amounts of data, mean that audience metrics have grown far more sophisticated in recent years. In addition to the aforementioned page views and bounce rates, analytics tools track variables like visitors’ return rates, referral sites, scroll depths and time spent on a page. Much of this data is delivered to news organizations in real time.

Metrics dashboards are now virtually ubiquitous in contemporary newsrooms, and heated debates about how and when they should be consulted are nearly as widespread as the metrics themselves. It is not surprising that metrics have become a hot-button issue in journalism. Their presence invites a number of ever-present tensions in commercial news media to come crashing into the foreground. Among them: What is the fundamental mission of journalism, and how can news organizations know when they achieve that mission? How can media companies reconcile their profit imperative with their civic one? To the extent that the distinction between journalist and audience is still meaningful, what kind of relationship should journalists have with their readers? Audience metrics have become ubiquitous in news organizations, but there has been little empirical research on how the data is produced or how it affects newsroom culture and journalists’ daily work.

With the support of Columbia University’s Tow Center for Digital Journalism, I undertook a long-term ethnographic research project to understand how the use of metrics changes reporters’ behaviour and what this means for journalism. My key research questions included the following:

First, how are metrics produced? That is, how do the programmers, data scientists, designers, product leads, marketers and salespeople who make and sell these tools decide which aspects of audience behaviour should be measured and how to measure them? What ideas—about both those whose behaviour they are measuring (news consumers) and those who will be using their tool (journalists)—are embedded in these decisions? How do analytics firms communicate the value of metrics to news organizations?

Second, how are metrics interpreted? Despite their opposing stances, arguments that metrics are good or bad for journalism have one thing in common: They tend to assume that the meaning of metrics is clear and straightforward. But a number on its own does not mean anything without a conceptual framework with which to interpret it. Who makes sense of metrics, and how do they do it?

Third, how are metrics actually used in news work? Does data inform theway newsrooms assign, write and promote stories? In which ways, if any, is data a factor in personnel decisions such as raises, promotions and layoffs? Does data play more of a role in daily work or long-term strategy? And how do the answers to these questions differ across organizational contexts?

To answer these questions, I conducted an ethnographic study of the role of metrics in contemporary news by examining three case studies: Chartbeat, Gawker Media, and The New York Times. Through a combination of observation and interviews with product managers, data scientists, reporters, bloggers, editors and others, my intention was to unearth the assumptions and values that underlie audience measures, the effect of metrics on journalists’ daily work, and the ways in which metrics interact with organizational culture. In what follows I will summarize some of my central discoveries.

First, analytics dashboards have important emotional dimensions that are too often overlooked. Metrics, and the larger “big data” phenomenon of which they are a part, are commonly described as a force of rationalization: That is, they allow people to make decisions based on dispassionate, objective information rather than unreliable intuition or judgement. While this portrayal is not incorrect, it is incomplete. The power and appeal of metrics are significantly grounded in the data’s ability to elicit particular feelings, such as excitement, disappointment, validation and reassurance. Chartbeat knew that this emotional valence was a powerful part of the dashboard’s appeal, and the company included features to engender emo- tions in users. For instance, the dashboard was designed to communicate deference to journalistic judgement, cushion the blow of low traffic and provide opportunities for celebration in newsrooms.

Second, the impact of an analytics tool depends on the organization using it. It is often assumed that the very presence of an analytics tool will change how a newsroom operates in particular ways. However, I found that organizational context was highly influential in shaping if and how metrics influence the production of news. For instance, Gawker Media and The New York Times are both Chartbeat clients, but the tool manifests in vastly different ways in each setting. At Gawker, metrics were highly visible and influential. At The Times, they were less so, and often used to corroborate decisions editors had already made. This suggests that it is impossible to know how analytics are affecting journalism without examining how they are used in particular newsrooms.

Finally, for writers, a metrics-driven culture can be simultaneously a source of stress and reassurance. It is also surprisingly compatible with a perception of editorial freedom. While writers at Gawker Media found traffic pressures stressful, many were far more psychologically affected by online vitriol in comments and on social media. In a climate of online hostility or even harassment, writers sometimes turned to metrics as a reassuring reminder of their professional competence. Interestingly, writers and editors generally did not perceive the company’s traffic-based evaluation systems as an impediment to their editorial autonomy. This suggests that journalists at online-only media companies like Gawker Media may have different notions of editorial freedom and constraint than their legacy media counterparts.

By way of conclusion, I make the following recommendations to news organizations. First, news organizations should prioritize strategic thinking on analytics-related issues (i.e., the appropriate role of metrics in the organization and the ways in which data interacts with the organization’s journalistic goals). Most journalists were too busy with their daily assignments to think extensively or abstractly about the role of metrics in their organization, or which metrics best complemented their journalistic goals. As a result, they tended to consult, interpret and use metrics in an ad hoc way. But this data is simply too powerful to implement on the fly. Newsrooms should create opportunities—whether internally or by partnering with outside researchers—for reflective, deliberate thinking removed from daily production pressures about how best to use analytics.

Second, when choosing an analytics service, newsroom managers should look beyond the tools and consider which vendor’s strategic objectives, business imperatives and values best complement those of their newsroom. We have a tendency to see numbers—and, by extension, analytics dash- boards—as authoritative and dispassionate reflections of the empirical world. When selecting an analytics service, however, it is important to remember that analytics companies have their own business imperatives.

Third, when devising internal policies for the use of metrics, newsroom managers should consider the potential effects of traffic data not only on editorial content, but also on editorial workers. Once rankings have a prominent place on a newsroom wall or website, it can be difficult to limit their influence. Traffic-based rankings can drown out other forms of evaluation, even when that was not the intention.

Finally, although efforts to develop better metrics are necessary and worthwhile, newsrooms and analytics companies should be attentive to the limitations of metrics. As organizational priorities and evaluation systems are increasingly built on metrics, there is danger in conflating what is quantitatively measurable with what is valuable. Not everything can—or should—be counted. Newsroom, analytics companies, funders and media researchers might consider how some of journalism’s most compelling and indispensable traits, such as its social mission, are not easily measured. At a time when data analytics are increasingly valorized, we must take care not to equate what is quantifiable with what is valuable.

Footnotes

1. This piece has been excerpted and adapted from “The Traffic Factories: Metrics at Chartbeat, Gawker Media, and The New York Times,” originally published by the Tow Center for Digital Journalism at the Columbia University Graduate School of Journalism in 2015. Republished with permission.

Works Cited

Artley, M. (2013, August 26). Let me explain why Miley Cyrus’ VMA performance was our top story this morning. The Onion. www.theonion.com/let-me-explain-why-miley-cyrus-vma-performance-was-our-1819584893