A journalist’s guide to US opinion polls

The 2020 election has its own troubling factors.

“Getting accurate poll data this year is being complicated by the pandemic, wide-spread mail-in voting, hyper-polarised constituencies and daily news surprises. Journalists shouldn't make matters worse by superficial and careless reporting,” wrote former Los Angeles Times editor Frank O. Sotomayor in an article “Reporting on polls? Here’s how to do it responsibly.”

Improving the quality of coverage isn’t rocket science if you know a few essentials, said Sotomayor. “Too many reports, for example, ignore that each poll carries a margin of error—or explain what that means. Adding fine print at the bottom of a graphic doesn’t cut it.”

Here is the good news. Boning up on polling fundamentals and concepts is easier than ever using online tools. There are webinars and videos on polling, online courses and guidelines from veteran journalists. Most are free and accessible.

For journalists, reporting on polls is serious business, a time-honoured responsibility that gives voice to ordinary citizens about their leaders and the social, political and economic issues that impact their lives. Reliable polls are regarded as the most accurate way to measure public opinion, but that comes with a caveat.

“Not all polls are created equal, and it’s a challenge for reporters to put polling results in proper perspective,” cautions Louis Jacobson, senior correspondent for PolitiFact, a Pulitzer Prize-winning fact-checking website. “Don’t assume the previous election is going to be the model for what’s happening now.”

Building confidence in polling

Courtney Kennedy, director of survey research at Pew Research Center, has a ringside seat for opinion polling in the United States. She is optimistic about this election cycle. There are more polls in key states, and some are of higher quality.

In several battleground states like Michigan, there are double or more polls than in 2016, and many are addressing the technical issues that went wrong last time, said Kennedy, but polls are not infallible. Overall, national polls have performed well, she says. Errors in state-level polls in 2016 were “large and problematic.”

In an article for Pew Research Center, Kennedy stressed the value of polling to a democracy. “A robust public polling industry is a marker of a free society. It’s a testament to the ability of organisations outside the government to gather and publish information about the well-being of the public and citizens’ views on major issues,” she wrote.

There also was a list of what the public should know about polling heading into the 2020 presidential election. Here is a sampling:

• Different polling organisations conduct their surveys in quite different ways. “Currently, CNN and Fox News conduct polls by telephone using live interviewers, CBS and Politico field their polls online using opt-in panels, and The Associated Press and Pew Research Center conduct polls online using a panel of respondents recruited offline. These different approaches have consequences for data quality, as well as accuracy in elections.”

• Huge sample sizes sound impressive, but sometimes they don’t mean much. “Students learning about surveys are generally taught that a very large sample size is a sign of quality because it means that the results are more precise. While the principle remains true in theory, the reality of modern polling is different. As Nate Cohn of the New York Times explained, `Often, the polls with huge samples are actually just using cheap and problematic methods.’”

• Transparency in how a poll was conducted is associated with better accuracy. “The polling industry has several platforms and initiatives aimed at promoting transparency, including the American Association for Public Opinion Research’s `Transparency Initiative’ and the Roper Center Archive. FiveThirtyEight’s Nate Silver found that polling firms participating in these organisations have less error on average than those that don’t.”

• The barriers to entry in the polling field have disappeared. “Technology has disrupted polling in ways similar to its impact on journalism: by making it possible for anyone with a few thousand dollars to enter the field and conduct a national poll...there has been a proliferation of polls from firms with little to no survey credential or track record.”

Pew Research Center's website offers a field guide to polling and instructive videos under the heading, “Methods 101,” exploring topics such as random sampling, wording for survey questions, and how polling is conducted around the world.

PolitiFact’s Louis Jacobson also paints a more positive scenario for opinion polls this year. In 2016, “Polls were all over the map, zigzagging from start to finish. They appear much more stable this time,” said the former deputy editor for Roll Call, a Washington, D.C., newspaper that covers Capitol Hill.

“People know a lot about Trump. They have had their minds made up for or against him for a long time. The choice [between Biden and Trump] appears to be more fixed,” said Jacobson. He cites another factor: Americans are voting early by mail or in-person in record numbers.

As of October 28, more than 75 million votes had been cast around the US, surpassing the 58.3 million total pre-election votes in 2016. That is more than 54 percent of the overall turnout for the 2016 election, according to The New York Times. Data science consultant Rob Arthur believes the transformation in how people are voting could be a snag for polls.

“Pollsters are doing their best to cope with this, but it’s really hard to know how all these different factors – coronavirus, new ways of voting, the constant stream of news – will impact an election,” said Arthur. “Even the best pollsters may run into problems they haven’t anticipated. PolitiFact has posted a primer on what voters should pay attention to when reading polls, including weighting for education, poll wording, cherry-picking poll results, and cell phone/Internet polling. "We are entering a new golden age for polling if the tools are used properly," Anthony Salvanto, CBS News elections and surveys director, told PolitiFact.

Writing about polling data

The London-based Market Research Society (MRS) is a hub of quality control for opinion polling with more than 5,000 members in 50 countries, according to the website. “We have an ongoing commitment to supporting members’ professional development and providing them with tools they need to uphold the highest standards,” said CEO Jane Frost.

She points to resources available free on the MRS web page created for journalists and the media. The goal, she says, is to train the trainers, especially journalism educators because, “That’s the best way, to catch them early. We want these materials to be disseminated and used.”

A training module on the MRS website, “Interpreting polls and election data – guidance for media and journalists,” provides an overview on polling that is adaptable to workshops, seminars and classroom instruction. Newsrooms might also use it for professional development.

MRS partnered with IMPRESS, the UK’s independent press regulator, to create “Using surveys and polling data in your journalism,” a training tool on how to spot unreliable research and ensure that stories on polls are statistically sound. There is a focus on common mistakes reporters make, such as a lack of understanding of statistics. For instance, not knowing the difference between UK and Great Britain for statistical purposes.

MRS also works closely with the British Polling Council (BPC), another opinion poll watchdog. The BPC’s quick guide for reporting on polls, posted on its website, begins with a hypothetical:

“The results of a poll have just landed on your desk. You have to write a report about it in a matter of hours. But, can you trust it? What should you be looking out for? And what details should you include?” Here is a quick guide for you need to know and do – in just five minutes.”

It explains how polls are conducted, which ones should be avoided, what can go wrong and the limits of polling and ends with a checklist of five questions reporters should answer before writing their story.

Another resource worth noting. The Quinnipiac University Poll, a household name in the U.S. polling industry, offers a tip sheet called, “The importance of covering poll data with clarity and accuracy.” It exhorts reporters to answer who, what, when, why and how before writing about a poll.

Quinnipiac states on its website: “Before journalists can report the statistical findings of a poll, it is essential they understand the methodology involved in the development of the poll and the data collection. The first thing to determine is whether the poll is transparent about its methods. There are many factors that go into creating a poll, and each one can have a big impact on the results.”

The bottom line: The more journalists know about polls, how they work and how to evaluate their quality, the closer they come to clarity and accuracy in reporting. That is vitally important during an election campaign where lines between truth and disinformation often have been blurred.

Prepping for election night

Political pundits predict election night may be rife with chaos. A Poynter webinar brought together media experts to ruminate about the potential problems that could arise.

Counting ballots will take longer with a record number of mail-in votes. States have divergent rules on when they can start counting ballots. Social media and extreme partisanship will stroke misinformation. The winner might not be known for days or weeks, depending on the outcome and legal entanglements that might follow.

Those are only some of the issues that could beleaguer Election Night 2020 said newsroom leaders from National Public Radio, The New York Times, CNN’s Washington bureau, and Associated Press, among others.

After the webinar, PolitiFact’s Jacobson co-authored an article listing recommendations for journalists assigned to cover the election. They included:

• Figure out a coronavirus safety plan for your newsroom. “All the preparations in the world won’t matter if you’re too sick to work. Reporters will be out on the street talking to voters, and at polling places and election supervisor headquarters. Provide relevant equipment, such as masks, to keep your staff safe.”

• Counter misinformation when you see it. “Debunk or clarify the claim as early as possible in the story, and make sure not to give a misleading impression with headlines and in social media. Online tools such as Crowdtangle can help you find out what’s gaining traction online and reverse-image search RevEye can help verify and track down viral images. Google’s Factcheck Explorer can lead you to fact-checking work of other journalists.”

• It’s OK to say you don’t know the answer yet. “When there haven’t been enough votes counted to be sure who’s ahead, or when the margins are too close to make a call, don’t rush. It’s more accurate and responsible to say there’s `no clear leader’ than to focus on who is leading with a small margin.”

• Emphasise the local view. “The big advantage that local media outlets have over national ones is that they already have journalists on the ground. If a dispute arises in a specific state, reporters in local newsrooms are often best positioned to sort out the facts.”

Jacobson’s article also stated a reality: Whether it’s orderly, or whether it proves to be the weirdest election night ever, it is certain to be historic. That appears to be a foregone conclusion. Foreign Policy magazine has dubbed the Biden-Trump race. “The Most Important Election. Ever” in American history.