Today, CNN is launching new standards for reporting on polling. These standards will serve to elevate research conducted by the most skilled pollsters and will make it easier for readers and viewers to understand what a poll says.
The news is flooded with polls every election year. Those looking to raise their profile have discovered that frequent polling on newsworthy topics can get a lot of attention. But not all of those polls deserve to be taken as seriously as polls conducted by researchers doing surveys to contribute to the public’s understanding of the American people.
It doesn’t take much to conduct a poll these days, but it does take quite a lot of care to conduct one well.
CNN will continue to hold pollsters to high standards. Our new approach opens the door for recently developed methods that did not previously meet the benchmark, provided they are being done carefully, while holding those using long-trusted methods to a higher bar that better reflects the steps pollsters need to take to conduct accurate surveys today.
The standards rest on responses to 16 questions which assess the methodology and the assumptions which form the backbone of any publicly-released poll. Pollsters who meet the standards are those who have answered all 16 questions and have done so in a way that shows they are using the best practices the industry has and methodologies which have been proven to work.
CNN’s 16 questions will be easy to answer if a pollster has taken the necessary steps to reliably estimate the American public’s opinion. Any pollster who can’t — or won’t — answer them all may not have put in the time and effort necessary to reach a representative sample today.
And getting the views of a representative group of Americans is harder than ever. Polling has changed quite a lot, and rapidly. For decades, the gold standard for polling was to survey people reached by a random sample of telephone numbers for landline phones. Today, relying on that approach alone produces skewed results because the majority of Americans who only use cellphones are ignored. But calling cellphones as well as landlines has grown more expensive and more difficult. Online research has become dominant for market research and is becoming more prominent in public opinion research as costs have risen for phone polling, even though there hasn’t been clear consensus about the best way to do it.
Probability sampling — the kind where everyone in the group of people being surveyed has a chance at being interviewed — remains the best place to start when it comes to polling methodology. It is still the gold standard, and research based on other methods is lost without benchmarks established using those methods. But it isn’t the only way to get a representative slice of the American public.
By implementing these new standards, CNN will cover some polling that uses non-probability sampling, but only if pollsters can demonstrate that their modeling assumptions were carefully constructed using criteria that are meaningful to the topic and their samples were chosen from reliable sources to represent the target universe.
That’s where CNN’s list of 16 questions comes in. Evaluating the validity of a poll is only possible if pollsters are transparent about what they are doing. Transparency has become a critical part of how those in the survey research world assess the validity of other research, with the American Association for Public Opinion Research and the Roper Center both implementing formal means to gauge a pollster’s transparency. CNN’s new polling standards means readers and viewers can be confident that polling covered on CNN has been done by pollsters who are willing to show their work.
There is not one correct way to answer the 16 questions that make up our standards questionnaire, but there are some things that have not met CNN’s standards for reporting in the past and will not now. And there are some new, clear lines that define the way polls should be done today. We will not cover:
- Polls that are conducted by campaigns or by those who have a financial or advocacy interest in the outcome
- Polls that are conducted by telephone using robocalls rather than live interviewers
- Polls that are conducted without any type of sampling, where anyone who chooses to can participate
- Polls that are conducted solely using unrepresentative sample sources
- Polls that are conducted without accounting for people who take surveys on their cellphones, either over the phone or by web
- Polls on some topics which do not ensure that respondents of all education levels are adequately reflected
We take seriously our responsibility to report polling carefully.
When journalists weigh whether to report information shared by a source, one of the most important considerations is how that source knows what they say they know. A journalist must be satisfied that the source’s information is solid before it goes out to the public.
Our evaluation of polls and surveys should meet that same standard, and the initial approach ought to be skeptical: A poll’s existence alone does not make it news, and journalists shouldn’t be any more credulous about numbers than they are about words. CNN’s standards mean that polls covered by CNN have been thoroughly checked out.
