Why You Shouldn’t Always Trust The Inside Scoop

This is the eighth article in a series that reviews news coverage of the 2016 general election, explores how Donald Trump won and why his chances were underrated by most of the American media.

So … this is a little awkward. Here we are, in the midst of a series of articles¹ about the media’s shortcomings in covering the 2016 general election. And President Trump, the winner of that election, has to go out and call the media the “enemy of the American people.” Suffice it to say that I’m not on board with Trump’s position. In fact, I’ve had a couple of friends who — while they’ve mostly agreed with the substance of these articles — have asked whether this is really the best time for media criticism. Doesn’t the industry need more solidarity at a time when it’s under attack?

Well, maybe. But I’m worried about that old adage about those who don’t learn from history being doomed to repeat it. There was never much of a learning process after the election, with media outlets finding various scapegoats for the surprising result² — the polls! fake news! Clinton’s strategy in Wisconsin! — instead of examining whether there was some deeper problem with their reporting methods. And now, Trump’s screaming, hyperbolic attacks on the press have the potential to drown out any constructive attempts at figuring out what went wrong.

Take, for example, Trump and the White House’s recent admonitions of reporters for using anonymous sources. “Because they have no sources, they just make them up when there are none,” Trump said at the Conservative Political Action Conference on Friday, to take one of many recent examples. By making what amounts to the worst possible version of the argument — ANONYMOUS SOURCES ARE LIES AND SHOULD NEVER BE USED!!! — Trump and the White House make themselves easy to dismiss. One can point out their rank hypocrisy: For instance, that White House chief of staff Reince Priebus recently used unnamed sources to criticize a New York Times article that used unnamed sources, or that White House press secretary Sean Spicer literally shut the doors on news organizations that were seeking comment from him on the record.

But given their exceptionally widespread use — I can’t remember a time when the political news cycle was more dependent on anonymously sourced stories than it is right now — a more nuanced, cool-headed critique of anonymous sourcing might be useful. For example, one could worry — at a time when trust in media is declining — that the “appearance of anonymous sources in routine government and political stories” could strain a news organization’s “trust with readers,” leading to questions about whether it was “carrying water for someone else’s agenda.” Or one could argue that while anonymous sources are sometimes necessary, they should be used only as a “last resort” and only to provide newsworthy information rather than “just spin or speculation.” At the very least, “a story that hangs entirely on anonymous sourcing should always get special scrutiny.”

Those phrases aren’t mine; they’re taken from The New York Times’s policy on anonymous sources. But like most other news organizations’ positions on the anonymous sourcing, the Times’s stated policy has relatively little to do with how these sources are used in practice. So let me tell a cautionary tale of a case where anonymously sourced information wasn’t very reliable, and may have misled readers and reporters. That was in the frequent references to the campaigns’ internal polls at the Times and other news outlets during last year’s election. On Oct. 31, for example, the Times cited private polling from anonymous strategists who claimed that FBI Director James B. Comey’s letter to Congress two days earlier had not seriously affected Clinton’s position in the race:

And while Mr. Trump has crowed about the 11th-hour twist to the race, the F.B.I. director’s letter about the emails has not yet produced a major shift in private polling, according to Republican and Democratic strategists with access to confidential data, who spoke on the condition of anonymity. Mrs. Clinton’s lead over Mr. Trump appears to have contracted modestly, but not enough to threaten her advantage over all or to make the electoral math less forbidding for Mr. Trump, Republicans and Democrats said.

In this instance, the inside scoop turned out to be totally useless. Public polls, like the ones used in the FiveThirtyEight or RealClearPolitics polling averages, were a lot closer to the mark, by contrast. They did show a significant shift after Comey’s letter, with Clinton’s national lead declining from 6 percentage points to 3 percentage points — the difference between a fairly safe lead and one that left her highly vulnerable in the Electoral College.

The thing is, this failure of internal polls wasn’t unusual. The Clinton campaign’s polls dangerously overestimated their candidate’s standing in the Midwest. Meanwhile, the Trump campaign put their candidate’s chances at 30 percent on the eve of the election — the same odds that FiveThirtyEight’s model gave Trump based on public polling — a comparatively bullish projection but one that still had their candidate as an underdog. In 2012, Mitt Romney’s internal polls were laughably off the mark in states including Colorado and New Hampshire, although by all accounts Barack Obama’s were pretty good. So among the past four presidential campaigns, we have one case where internal polls did well (Obama 2012), one where they were mediocre (Trump 2016) and two where they were utter disasters (Clinton 2016 and Romney 2012) — not a very good track record.

Of course, that’s a small sample size. So let’s open things up a bit. FiveThirtyEight’s pollster ratings database contains polls conducted in the final three weeks of presidential, congressional and gubernatorial campaigns since 1998. The database flags polls put out by campaigns or by Democratic or Republican interest groups. I’m going to focus on polls of U.S. House races since they’re the case when internal polls are released to the public most often.³

Polling individual House races is tough no matter what you do, but the partisan and internal polls have been especially bad. Polls conducted by Democratic candidates or groups have been biased toward the Democratic candidate by almost 5 percentage points on average while missing the final margin in the race by an average of 8 percentage points. Republican polls have had roughly a 4-point Republican bias and an average error of 7.5 percentage points. And these partisan polls have also misidentified the winner of the election almost a third of the time, about twice as often as nonpartisan polls have. The nonpartisan polls haven’t been great, but they’ve been much less biased and somewhat more accurate, especially if you exclude YouGov (which makes up a huge portion of the sample because it surveyed almost all House races in 2014).

There are certainly some ambiguities here. Our database only contains polls that were publicly available in advance of the election. These polls were usually released by the campaigns themselves, presumably to help the campaign receive more favorable press coverage. Perhaps the entire sample of internal polls, and not just those that made their way into the public domain, would show better results. Campaigns such as Clinton’s also have a lot of smart people working for them, and they possess voter-file and other proprietary data that’s potentially a lot more powerful than what public pollsters have. They usually also have larger polling budgets than news organizations and so can afford to take fewer methodological shortcuts. Under perfect conditions, therefore, internal polls would potentially be more accurate than public ones.

But circumstances are rarely perfect. In practice, chasing down internal polls isn’t a good use of a reporter’s time and energy, and readers should treat these polls with extreme prejudice when they encounter them in stories. Consider their potential shortcomings:

• Campaigns might not be honest with themselves. Think about any company or organization that you’ve worked for. No doubt it has access to all sorts of information that isn’t public. Would you expect it to provide a more honest and accurate self-assessment than a neutral third party?⁴ Well, maybe. It would depend a lot on the company’s culture and the incentives of the people doing the assessment. But there’s a good chance the self-assessment would be biased in various ways, probably toward a rosy view of the company’s performance.
• Campaigns might not be honest with reporters. Unlike a public pollster, a campaign’s objective when discussing polling is to influence coverage instead of providing the most accurate snapshot of the race. Campaigns are selective about what numbers they whisper to reporters and how they characterize their position. And they’re pretty good at spinning. While we don’t spend a lot of time talking to campaigns at FiveThirtyEight, we do so occasionally, and we’re sometimes surprised that the story we get from the campaigns doesn’t match what we’re reading in other news outlets that characterize the campaigns’ positions.⁵ Somebody is getting spun, and nobody thinks it’s them.
• Reporters might misinterpret the campaign’s information. Even if a campaign gave a reporter the straight scoop, the reporter might misinterpret the data or filter it through the lens of whatever prior beliefs they brought to the story. There’s a hint of this in the New York Times snippet that I clipped above, for instance: The story refers to how Clinton’s lead “appears to have contracted modestly” but then asserts that this wasn’t a big deal because of her strong position in the Electoral College. However, the Times’s view that the Electoral College favored Clinton was badly mistaken; just the opposite was true, which is why even a slight tightening of the national race was a big deal.
  Obviously, misinterpreting data can also be a big problem when it comes to publicly available information. But it’s especially problematic for proprietary information because the reporter usually won’t provide enough detail to allow for the reader to check the reporter’s work. The reporters who are most skilled at information gathering — at uncovering leads and developing sources — aren’t always the best at information analysis, especially when it comes to technical subjects such as polling, but if the reporter is the only person in the room and can’t reveal many specifics about their experience, you’re being asked to put a lot of faith in their interpretation of complicated evidence.
• Inside sources rarely provide a complete vantage point. Reporting based on anonymous sourcing often reflects some sides of the story but not others, and this isn’t clear to readers because of the opaque sourcing. It’s worth asking why was the media was so sure of Clinton’s chances, for instance, when it hadn’t been at all confident about Obama’s odds four years earlier. It may had something to do with an imbalance in reporters’ sources. For instance, who were the Republican strategists who told the Times that the Comey letter was no big deal? Were they Republicans working for the Trump campaign or were they #NeverTrump Republicans? In any event, many of the strategists working for Trump had frosty relationships with the press and may not have seen much benefit from spinning horse-race coverage in their favor anyway.⁶ Thus, the same reporting methods that had produced too much equivocation in coverage of the Obama-Romney campaign may have tilted analysis of the 2016 horse race in Clinton’s favor.

So suppose I’ve convinced you never to care about a campaign’s internal polls again, or at least not to do so in races where there’s a fair about of nonpartisan polling available. To what extent does this critique impugn reporting based on anonymous or inside sourcing overall, as opposed to being a one-off quirk specific to poll analysis?

I don’t know. The polling beat is unusual in that public and private information compete on a relatively level playing field. In most other domains, one or the other method is dominant. There’s no RealClearPolitics average for reporting on the intelligence agencies, for example, and there aren’t a lot of inside scoops to be had when reporting the weather forecast. Nevertheless, precisely because there aren’t a lot of opportunities to compare them, that access-driven horse-race reporting has somewhat miserably failed as compared with methods based on aggregating public information is interesting. It suggests that newsrooms might place too much of a premium on obtaining proprietary information. And to be clear, this is more access journalism’s failure than polling aggregation’s success; there have been cases where both methods missed the mark, but very few⁷ where insiders had the story right but the public pollsters didn’t.

So while I certainly wouldn’t want any news organization to stop aggressively reporting on Trump’s — or any other president’s — administration, I’d encourage reporters who are doing so to abide by their publications’ own standards. Use anonymous sources for matters of legitimate national importance and not for discussing horse-race strategy. Even if the sources are anonymous, provide more detail to readers rather than less; the greater amount of detail was key in The Wall Street Journal’s and The Washington Post’s reporting on Michael Flynn, which forced him to resign as national security adviser.

And even if you can’t name the sources, you can characterize their vantage points. If you have four anonymous CIA sources alleging something about Trump, do they reflect the consensus of top-level officers at the agency, or are they a dissident group? Those are very different stories, but the reader won’t be able to distinguish one group of “senior intelligence officials” from another without this guidance.

Perhaps most importantly, treat stories based on inside information as requiring an especially large dose of skepticism rather than holding them to privileged status. Oftentimes, reporters at powerful news organizations do exactly the opposite, assuming that information is valuable and accurate because it happens to be proprietary. Stories based on anonymous sources can have a romantic allure that conjures up Bob Woodward getting tips from Deep Throat in a parking garage. Now and then the stories indeed live up to that groundbreaking threshold, but other times they’re just glorified intraoffice gossip. Placing a higher burden of proof on these stories can improve the signal-to-noise ratio and help journalism to fulfill its highest aspirations.