Last December, a joint survey by the Economist and the polling organization YouGov claimed to reveal a striking antisemitic streak among America’s youth. One in five young Americans thinks the Holocaust is a myth, according to the poll. And 28 percent think Jews in America have too much power.
“Our new poll makes alarming reading,” declared the Economist. The results inflamed discourse over the Israel-Hamas war on social media and made international news.
There was one problem: The survey was almost certainly wrong. The Economist/YouGov poll was a so-called opt-in poll, in which pollsters often pay people they’ve recruited online to take surveys. According to a recent analysis from the nonprofit Pew Research Center, such polls are plagued by “bogus respondents” who answer questions disingenuously for fun, or to get through the survey as quickly as possible to earn their reward.
In the case of the antisemitism poll, Pew’s analysis suggested that the Economist/YouGov team’s methods had yielded wildly inflated numbers. In a more rigorous poll posing some of the same questions, Pew found that only 3 percent of young Americans agreed with the statement “the Holocaust is myth.”
These are strange times for survey science. Traditional polling, which relies on responses from a randomly selected group that represents the entire population, remains the gold standard for gauging public opinion, said Stanford political scientist Jon Krosnick. But as it’s become harder to reach people on the phone, response rates have plummeted, and those surveys have grown exponentially more expensive to run. Meanwhile, cheaper, less-accurate online polls have proliferated.
“Unfortunately, the world is seeing much more of the nonscientific methods that are put forth as if they’re scientific,” said Krosnick.
Meanwhile, some pollsters defend those opt-in methods — and say traditional polling has its own serious issues. Random sampling is a great scientific method, agreed Krosnick’s Stanford colleague Douglas Rivers, chief scientist at YouGov. But these days, he said, it suffers from the reality that almost everyone contacted refuses to participate. Pollsters systematically underestimated support for Donald Trump in 2016 and 2020, he pointed out, because they failed to hear from enough of those voters. While lax quality controls for younger respondents, since tightened, led to misleading results on the antisemitism poll, YouGov’s overall track record is good, said Rivers: “We’re competitive with anybody who’s doing election polls.”
Nonetheless, headlines as outrageous as they are implausible continue to proliferate: 7 percent of American adults think chocolate milk comes from brown cows; 10 percent of college graduates think Judge Judy is on the Supreme Court; and 4 percent of American adults (about 10 million people) drank or gargled bleach to prevent Covid-19. And although YouGov is one of the more respected opt-in pollsters, some of its findings — one third of young millennials aren’t sure the Earth is round, for example — strain credulity.
Amidst a sea of surveys, it’s hard to distinguish solid findings from those that dissolve under scrutiny. And that confusion, some experts say, reflects deep-seated problems with new methods in the field — developed in response to a modern era in which a representative sample of the public no longer picks up the phone.
The fractious evolution in polling science is likely to receive fresh attention as the 2024 elections heat up, not least because the consequences of failed or misleading surveys can go well beyond social science. Such “survey clickbait” erodes society’s self-esteem, said Duke University political scientist Sunshine Hillygus: It “undermines people’s trust that the American public is capable of self-governance.”
Veteran pollster Gary Langer compares traditional randomized polling methods, known as probability polling, to dipping a ladle into a well-stirred pot of minestrone soup. “We can look in and see some cannellini beans, little escarole, chunks of tomato,” he said. “We get a good representation of what’s in the soup.”
It doesn’t matter if the pot is the size of Yankee Stadium, he said. If the contents are thoroughly mixed, one ladle is enough to determine what’s in it. That’s why probability surveys of 1,000 people can, in theory, represent what the entire country thinks.
The problem is that getting a truly representative sample is virtually impossible, said YouGov’s Douglas Rivers, who pointed out that these days a good response rate to a randomized poll is 2 percent.
Pew expends a great deal of effort to maintain a randomized panel of about 10,000 people willing to take surveys. For the most recent annual recruitment, the organization mailed letters to a random selection of 13,500 residential addresses obtained from the U.S. Postal Service, receiving around 4,000 responses according to Pew researcher Courtney Kennedy. They only invite one-quarter of responders to the panel. Otherwise, Kennedy explained, the panel would be overrun with the types of people most amenable to taking surveys. Eventually, they wound up with 933 new recruits.
Some groups — in particular young people, people of color, and those who didn’t go to college — are generally more reluctant to take surveys, said Kennedy: That’s where they lose the perfect representative. Like every other pollster, she said, Pew adjusts their data, giving more weight to the responses of those underrepresented in the sample, so that the results represent the country in terms of demographics such as age, gender, race, education level, and political affiliation.
But those weighting methods are imperfect. And the people in a poll are still unrepresentative in at least one way: They are the Americans who are willing to answer a pollster’s message. Those difficulties have prompted a quiet revolution in survey research over the past two decades.
In 2000, nearly all pollsters simply called people on the phone, according to a 2023 Pew study of polling methods. But use of calls alone plummeted starting in 2012, while online opt-in surveys like the Economist/YouGov survey, one of the main forms of what are known as nonprobability polls, soared.
Nonprobability surveys don’t stir the pot so that each ingredient has an equal chance of being selected. Instead, they scoop up what’s referred to as a convenience sample of respondents, typically recruited online. Opt-in pollsters differ in how they recruit and select participants, and they are not always transparent about their methods. Once they have assembled a group of participants, pollsters can weight the sample so that it matches the broader U.S. population. But it’s much harder to accurately weight nonprobability polls, since there is less information on how people who opt into polls compare to the public at large.
“Probability sampling tends to yield more representative samples that nonprobability approaches,” Kennedy wrote in an email.
However, nonprobability surveys are typically much cheaper than probability polls. As Americans have ditched their landlines and stopped answering their cell to unknown callers, contacting people takes far more time and effort than it used to. As a result, according to Duke University political scientist Sunshine Hillygus, while it can cost as little as $1 per response to run a short online opt-in poll, it can cost 50 to 500 times that for a high-quality random-sample survey.
To create a pool of people to take opt-in surveys, polling companies recruit through ads that pop up on social media, internet search engines, and even during video games, offering cash or rewards to complete surveys, said Kennedy. YouGov, for example, pays people in points — 500 to 1,000, for example, to take a short survey. At 25,000 points, you can cash in for a $15 gift card; 55,000 points earns $50 in cash.
Pew and other pollsters who do randomized polling also pay people a small amount to take the occasional survey. But with opt-in polling, survey taking can become a full- or part-time job for many people. The job search website Indeed, for example, lists companies that pay for surveys in its career guide. And in the Reddit community Beer Money, which has 1.3 million members, people frequently discuss the pros (time flexibility) and cons (skimpy pay; frequently getting screened out) of taking surveys for money.
Some of those surveys are for academic research. (Many psychology papers, for example, rely on paid respondents recruited through platforms like Amazon Mechanical Turk.) Others help companies with market research — or feed the insatiable media market for polls.
Many people sign up for opt-in polls in good faith, according to Kennedy. “The problem is that this open-door approach also invites fraud,” she wrote in an email. “People create multiple accounts under multiple fake identities to complete as many surveys as possible just to make money.”
However, with the appropriate guardrails against fraud, YouGov chief scientist Rivers said, such methods offer a practical alternative to conventional probability sampling, where the costs are too high, and the response rates are too low. In some sense, he suggested, most polling is now nonprobability polling: When only 2 out of 100 people respond to a survey, it’s much harder to claim that those views are representative, said Rivers. “Sprinkling a little bit of randomness at the initial stage does not make it a probability sample.”
“Our approach has been: Let us assemble a sample systematically based on characteristics,” said Rivers. “It’s not comparable to what the census does in the current population survey, but it’s performed very well in election polling.” Rivers pointed to YouGov’s high ranking on the website FiveThirtyEight, which rates polling firms based on their track record in predicting election results and willingness to show their methods.
Gary Langer was not particularly impressed by high marks from FiveThirtyEight. (His own firm, Langer Research Associates, also gets a top grade for political polling they conduct on behalf of the partnership between ABC News and The Washington Post.) “Pre-election polls, while they get so much attention, are the flea on the elephant of the enterprise of public opinion research,” he said. The vast majority of surveys are concerned with other topics. They form the basis of federal data on jobs and housing, for example, and can reflect the public’s views on education, climate change, and other issues. “Survey data,” he said, “surrounds us, informs our lives, informs the choices we make.”
Given the stakes, Langer relies exclusively on probability polling. Research shows that opt-in polls just don’t produce the same kind of consistent, verifiable results, said Langer.
Hillygus agreed with Langer that, in general, probability polling is the more accurate method, although she cautioned that “there’s wide variation” in quality among pollsters using both methods.
Research suggests that widely used nonprobability methods, in particular online opt-in polls such as the Economist/YouGov survey, have inherent vulnerabilities.
The prospect of cash or rewards can incentivize some people to complete surveys quickly and with as little effort as possible. “They’re giving you data and answers that just can’t possibly be true,” said Kennedy.
For example, in one test of opt-in polling, 12 percent of U.S. adults younger than 30 claimed that they were licensed to operate a nuclear submarine. The true figure, of course, is approximately 0 percent.
Four to 7 percent of people taking opt-in polls are so-called bogus respondents, compared to only 1 percent of those in the probability-based surveys, according to a 2020 Pew study. Those respondents do things like take the survey multiple times, or approve of everything, regardless of what was asked. Others seem to be located outside of the U.S.
Bogus respondents are more likely to say that they are a young adult or Hispanic, but particularly with the Hispanic subgroup, data suggests that claim frequently isn’t true, said Kennedy. They may inattentively tick “yes” to the “Are-you-Hispanic” question, she said, or spoof their identity because they think it gives them a better shot at making it through the initial screening.
And then there are people who just like to mess around, said Hillygus. Multiple studies have shown that some teens provide disingenuous answers on national health surveys — exaggerating drug and alcohol use, for example, or falsely claiming to be blind, a gang member, or to have had multiple pregnancies.
Surveys asking about belief in conspiracy theories are ripe for trolling, said Hillygus. Polls showing that a large swath of Americans believed the Pizzagate theory, which falsely connected Hillary Clinton to a child trafficking operation in a run out of a Washington, D.C. pizzeria, are likely inaccurate, Hillygus and colleague Jesse Lopez concluded in research that, as yet, has not been published in a peer-reviewed journal. “More than 50 percent of the people who were claiming to believe in Pizzagate and other conspiracies were all people that we identified as survey trolls,” Hillygus said.
Randomly selecting people from the whole population protects against fraud. “The key virtue of probability-based polling these days is not about the probabilities of selection,” Kennedy wrote. “It is about ensuring that bad actors are not infiltrating the survey and giving garbage data.”
There are ways to screen out impostors and trolls. YouGov uses around 100 indicators to gauge whether respondents are providing reliable data, said Rivers. After the Pew study on the antisemitism poll, YouGov ran its own analysis and discovered that its survey team had relaxed standards for people under 30 to get a large enough sample size. About half of that group should have been excluded, said Rivers. “This whole thing should have been caught at the time,” said Rivers. “There were various red flags in the data.” Since then, YouGov has put additional restrictions in place, he said, with more planned.
But opt-in polling firms run the gamut. Some, like YouGov, publish their methodology and strive to ensure the integrity of their data. Others, experts say, do not.
Those pollsters may use questionable methods such as river sampling, where people respond to a website ad or pop-up and are immediately directed to a survey. On a news site, for example, visitors might encounter a pop-up prompting them to answer a survey question to get access to an article. The method proved “horribly inaccurate” in 2016 election polls, said Krosnick. “The reality is, there was no scientific sampling at all involved, it was just scooping up fish swimming by in the internet river,” he said.
None of these methodological challenges has dampened enthusiasm for survey research. The number of active pollsters has more than doubled since the beginning of the 21st century, according to the 2023 Pew study.
“The internet is like heroin,” said Krosnick. “It has addicted people to trying to think up creative ways to make money, in as automated a way as possible, to maximize their profit margins.”
There’s certainly a thriving media market for eye-catching poll results. In December 2023, when the Economist and YouGov published their poll on antisemitism, the story went viral. In contrast, Pew’s study questioning the result did not draw nearly as much attention.
In March, the Economist added an editor’s note to the top of the story, acknowledging the Pew study. But the article — which blames social media, and particularly TikTok, for indoctrinating young people with conspiracies and antisemitism — remains unchanged.
In an email, Tom Amos, a spokesperson for the Economist, told Undark that Pew’s analysis was well done. While the Economist doesn’t believe that opt-in surveys are systematically less accurate than other polling, Pew’s results tell a cautionary tale, he wrote: “Their experiment did present strong evidence that this survey method faces an above-average risk of bogus respondents from particular demographic groups saying untruthfully that they hold rare or extreme beliefs.”
Media consumers should be skeptical of implausible findings, said Krosnick. So should reporters, said Langer, who spent three decades as a journalist, and who said news outlets have a responsibility to vet the polls they report on: “Every newsroom in the country — in the world — should have someone on their team evaluate surveys and survey methodologies.”
In the end, people need to realize that survey research involves some degree of uncertainty, said Joshua Clinton, a political scientist at Vanderbilt University, who noted that polls leading up to the 2024 election are bound to get something wrong. “My concern is what that means about the larger inferences that people make about not only polling, but also science in general,” he said. People may just dismiss results as a predictable scientific failure: “‘Oh, the egghead screwed up again.’” Clinton said he wants people to recognize the difficulty of doing social science research, rather than to delegitimize the field outright.
Even Rivers, whose firm produced the Economist poll that made headlines, acknowledged that readers should be cautious with eye-catching headlines. “We’re in a challenging environment for conducting surveys,” he said. That means that people need to take survey results — especially those that are provocative — with a grain of salt.
“The tendency is to overreport polls,” said Rivers. “The polls that get reported are the ones that are outliers.”