Can we trust the polls this year?

It’s one of the most-asked questions of the 2024 election: Should we trust the polls? Right now, the polls give Vice President Kamala Harris a roughly 2 percentage-point advantage in the national popular vote; in the battleground states, polls show essentially even contests in Arizona, Georgia, Nevada, North Carolina, and Pennsylvania, and small Harris leads in Michigan and Wisconsin. This weekend’s much-hyped New York Times/Siena College poll showing a one-point Trump (within the 2.8-point margin of error) may have prompted a round of doomscrolling among Democrats, but it only affirmed something polls have been saying all along: This election is going to be close. But after back-to-back presidential contests in which state and national polls underestimated the level of support Donald Trump would eventually receive, it’s reasonable to wonder whether head-to-head polls are missing something again in 2024. Have pollsters learned their lessons from 2016 and 2020? Are there Trump voters that pollsters aren’t reaching? Are the polls biased against Harris? Or is polling and the way media covers it fundamentally broken? I posed those questions to nine pollsters across the political spectrum and came away with a few frustrating but helpful conclusions. Those critical of polling have a point. Predicting the kind of electorate that is going to turn out in any given election is hard. It has been getting harder because of Trump’s ability to turn out the kinds of voters many polls have trouble capturing. There are legitimate concerns about how polls are even conducted, as polling gets more expensive and the number of polls being run grows. At the same time, pollsters seem to have learned from the polling misses of the past. They are more conscientious about reaching the hardest-to-reach voters, have reconfigured the way they run their operations, and feel pretty good about capturing a snapshot of a political sentiment in time. They point out that the alternative to a poll-filled world is one where we’re all left trying to define “vibes.” Answering the question “Should we trust the polls?” requires addressing several other questions too: * What is polling is good for? * What is polling equipped to do? * Are we asking polling to capture something it just can’t capture? According to David Byler, a longtime data analyst and pollster for Noble Predictive Insights, trust is the crux of the issue. “It’s really important to define what you mean by ‘trust,’” he told me. “If by ‘trust’ what you mean is ‘Are the polls going to tell me with certainty who is going to win and are they going to eliminate this discomfort I’m feeling about the future?’ the answer is: It’s not perfectly predictive.” But if trusting the polls means trusting that the polls are telling us something useful, then every pollster I talked to agrees: These tools are measuring something real. “We are collecting real data that does capture real sentiments and asks people pertinent questions, and they’re the kind of questions that you can’t really ask with any other tool that we use for political analysis,” Byler said. The case for polling skepticism On the eve of the 2016 election, national polls agreed that Hillary Clinton would comfortably win the popular vote, while battleground polls showed she had small leads against Donald Trump in states like Michigan and Pennsylvania, but a big lead in Wisconsin. The national polls were right; the battleground picture was not. Something different happened in 2020: National polls agreed Joe Biden was way ahead of Trump, but he was also routinely posting large leads in the battlegrounds. Many of those swing states would end up being decided by very narrow margins. In response to those polling misses, pollsters have been doing a lot of work to avoid the same problems. As we head into November, though, those old worries persist. The phrase I heard most in my conversations was a worry about “solving for the last problem” or “fighting the last battle.” In other words, lessons have been learned, but will those lessons apply this time around? In 2016, for example, pollsters addressed some of the reasons they overestimated Mitt Romney’s performance in 2012 but missed that state-level surveys were overrepresenting college graduates. That miss ended up artificially boosting Hillary Clinton’s support, especially in the Midwest battleground states that proved decisive. Voters who made their minds up in the final days of the 2016 election also ended up breaking for Trump, meaning their support wasn’t registering in polling. In 2020, amid the pandemic, new problems popped up: At both the state and national levels, polls overestimated Biden and Democrats’ level of support, but that disparity could not be explained by the same reasons as in 2012 or 2016. Late-deciding respondents were as likely to support Biden as Trump, and neither inaccurate weighting by education nor inaccurate estimates of the electorate as a whole were the culprits, per an industry postmortem on polling published in 2022. So what happened? Worry No. 1: Nonresponse bias The likely culprit in 2020 was what pollsters call nonresponse bias — that Trump supporters in general may be less likely to participate in polls, resulting in their systematic underrepresentation in polling samples. --- How to navigate peak poll season As we navigate through the home stretch of the general election season, we’re likely going to be inundated with a bunch of polling at the district, state, and national levels. After talking to pollsters for the last year, here’s my best distillation of tips for reading the polls. - Remember that any poll is a snapshot in time; it’s capturing sentiment for the specific time it was out in the field. It is not predictive of what happens weeks from now - Don’t stress too much about any one poll; take them in aggregate, like through the polling averages at FiveThirtyEight, RealClearPolitics, Silver Bulletin, and Split Ticket. - If you’re reading a poll, check the sample sizes and methodologies. Is it large enough (usually at least 1,000 is a good national sample)? Is it of registered voters or likely voters (the latter is usually better closer to the election)? Was it conducted by telephone, online panel, text, or a combination of methods (a combination of methods can be better)? - Check the firm that conducted the poll. Is it partisan? Reputable? FiveThirtyEight has useful ratings for the quality of pollsters’ work. - Don’t forget the margin of error: Percentage-point leads that fall within a margin of error can’t be taken as gospel. And keep margins of error in mind when comparing polls over time: Shifts aren’t always real. --- The same bias may likely cause problems for 2024 polling, according to some pollsters I spoke with. “The idea that people who support Donald Trump may be less likely to take polls than people who support Democratic candidates does seem to have some validity,” Scott Keeter, a senior survey adviser at the Pew Research Center, told me. There are a number of reasons to think this bias still applies today, despite pollsters’ attempts to account for it. Generally speaking, Trump voters tend to be less trusting of institutional figures like pollsters and journalists, and so may be less willing to answer these surveys. More specifically, Trump himself routinely attacks polling and pollsters in public remarks, further ingraining that distrust in his supporters. Some experts, like the Republican pollster Patrick Ruffini, warn that there are specific states where this nonresponse bias may have a bigger effect because of the nature of the electorate and the way data is captured there. Places like Wisconsin and North Carolina, he told me, have a concentration of the kinds of voters that polls have tended to underestimate: rural and suburban white, working-class, and non-college-educated voters. And Wisconsin specifically doesn’t track partisan data in voter files, either through party registration or a record of which primary a voter might have participated in before — meaning it’s harder for pollsters to weight their samples. “What you might be getting in a lot of cases is you’re talking to a particularly Democratic-leaning type of white, working-class voter that does exist in Wisconsin, and you’re not capturing the Trump-leaning type,” Ruffini said. “And there are fewer tools in our toolbox to be able to correct for that.” Worry No. 2: Unlikely and late-deciding voters Another issue haunting polling in 2024 is that pollsters don’t necessarily know who the unlikely voters and late-deciding voters will be. Polls of likely voters factor in experts’ best estimates of what the electorate in November will look like, but the last few years have shown that turnout can vary significantly from cycle to cycle. Few observers expected a surge in rural and suburban white Trump voters in 2016, for example; and some independent voters decided to turn out relatively late in the 2022 midterms cycle, contributing to better-than-expected congressional results for Democrats. After Biden’s exit from the race and Harris’s ascension, voter registrations and energy from younger people, people of color, and women voters have all surged, likely in Democrats’ favor. Those shifts all suggest changes in momentum and enthusiasm, but according to Celinda Lake, a longtime Democratic pollster who co-led polling for Biden’s 2020 campaign, whether these people turn out in November is an open question. “Whether an abortion issue is on the state ballot really increases the turnout of young women,” Lake said. Worry No. 3: Hard-to-poll subgroups A fundamental part of the “trust the polls” question comes from how polls are covered in the political press. Big headlines about what’s happening to young voters, Hispanic voters, and Black voters get a lot of traction but are often based on analysis of data in the crosstabs of a poll, not necessarily of whole polls dedicated just to this subgroup. That’s often because polling specific segments of the electorate is expensive and impractical. You need a variety of methods to reach young people who won’t pick up phone calls from random numbers or to reach Hispanic Americans who may work odd hours and don’t necessarily speak English as their first language. At the same time, looking at those crosstab results can yield conclusions with margins of error much larger than those of a poll’s topline results. So when you see big conclusions being drawn from a small number of respondents in a national poll, it helps to be extra cautious. I’ve written before about the difficulty of polling young people and voters of color; though coverage of polling news has improved, the focus on the horse race can run roughshod over nuance and lead to sweeping — if unearned — conclusions about what the polls are saying. The case for trusting what the polls say With all these caveats, it can be tempting to just reject polling outright. But polling is still the most useful tool we have to gauge where the electorate is. Polling’s usefulness starts with understanding what it can tell us. In a polarized country, “you’re just not seeing as many large swings [in public opinion],” Mike Noble, an Arizona pollster and the founder of Noble Predictive Insights, told me. But polling is still well-equipped to take temperature checks. “[It is a good tool] to track trends,” he added. They may not be perfect, but there are still reasons to trust polling. Reason No. 1: Polling still gives a good snapshot of reality Though polling’s predictive capabilities aren’t as exact as consumers might want them to be, it still does a pretty good job at describing reality, pollsters told me. A good example is the 2022 midterms, when polling captured a surge in Democratic enthusiasm to vote after the Supreme Court’s Dobbs decision. Polls and polling aggregates captured this reversal in Democratic performance as it unfolded. As Keeter explains, predicting an election “where the public is divided almost 50-50” is challenging because the result will be decided by just a few points likely within the margin of error. However, issue polling, where public opinion might not be so closely divided, is easier to read. “If you’re trying to say, ‘Does a majority of the public support abortion rights?’ if the poll is 5 percent off from whatever the hypothetical truth is or maybe even a little more than that, you still get a pretty good idea that a majority of people do support abortion rights,” Keeter said. Even in 2016, when polling took a major reputational hit, national polls still captured a pretty accurate picture of how the national popular vote would break. Electoral College outcomes at a state level have been far tighter, though; consequently, Nick Gourevitch, who oversees polling at the top Democratic firm Global Strategy Group, told me, “You have to live with the fact that you’re not going to know the answer to who’s going to win the election, but [polling] is still a really valuable tool.” Like Keeter, Gourevitch points out that polling is potentially more useful when it comes to issues rather than presidential races, and that type of polling may be more helpful to campaigns and to candidates actually working on these issues, other pollsters told me. Reason No. 2: Pollsters are always refining their methods Everyone I talked to for this story started by telling me that the way the polling industry operates today looks very different from 2016, from 2020, and even from 2022. Pollsters now use more methods and tools to try to reach voters and are constantly watching changes in voter registration, taking into account how the likely electorate might be changing. “Pollsters we work with and talk to are trying to take all these things into account,” Margie Omero, a partner at the Democratic polling firm GBAO, told me. “We look at the people who don’t take surveys or who are harder to reach. We ask what might happen if you’re doing a poll conducted over a longer period of time. You’re trying to reach people multiple ways, so not just the easiest people to reach. You’re comparing how your poll demographics compare to the larger universe, not just folks who are super likely to vote.” Reason No. 3: Looking at polls in the aggregate, or as an average, is still very useful Since every poll is a snapshot in time, when taken together they can offer a lot more clarity about the trends in a presidential cycle than when taken individually. Polling aggregates and averages, like those run by FiveThirtyEight, RealClearPolitics, Silver Bulletin, and Split Ticket, offer consumers a filter through which they can see what all the high-quality public opinion research is saying. This can help to correct some of the quirks or troubles that individual polls might have. Polling aggregates are different from election forecasts — which try to go a step beyond aggregating polls and averaging them out to try to predict additional vibes-based factors — but can still provide a useful look at where an election stands. How to be a savvy poll-reader The good news for polling addicts is that the quality and frequency of polling will likely improve later in the election cycle. Post-Labor Day, polls tend to be a lot less noisy: The public is tuning back into the election; political news, ad spending, and campaigning are all ramping up; and after both national party conventions, the public will likely get to see at least one more presidential debate. Early voting will begin in many states by the end of the month, and these next nine weeks will see an explosion in polling. As November bears down, the pollsters I spoke with gave me the same set of tips for consuming polls. First, look at the sample size. Is it large enough (usually 1,000 people is good) to capture sentiments? Is it of likely voters (usually a better metric closer to the election) or registered voters? Are the conclusions being reported in write-ups that are overblown (making large demographic conclusions based on small samples) or presented with appropriate skepticism? Second, look at the methodology. Was a poll conducted by telephone, online panel, or text? Was it a combination of methods? Multi-modal polling tends to be more accurate than any single method. If it’s an online poll, was the sample drawn from an opt-in group (where people sign up to take a poll) or a probability-based panel (a sample drawn randomly from a large national database)? Probability-based samples tend to be better. Which firm conducted the poll? Referring to pollster rankings and ratings like those offered by FiveThirtyEight can be helpful here to discount junk polls or outliers. Check for partisan affiliations. Those can influence the decision to release a poll or not (some outlets have scrutinized specific Republican-aligned firms). Third, make sure not to miss the margin of error. Part of why polls can’t be too specific is because their topline results exist in a range of possibilities. That’s why movement within a margin of error can’t count as real change; if one poll has a margin of error of +/- 3 points and reports a 49-46 result, a second poll with the same margin of error reporting a 51-48 shift can’t immediately be taken as proof of a real improvement since the shift is covered by the potential error. Last of all, remain skeptical of polls as prophecy. According to Byler, “We’re in this era of incredibly close elections where no major-party presidential candidate is gonna go below 42 or 43 percent of the vote, so you end up in a situation where, because of polarization [and Electoral College bias for Republicans], both major-party candidates are always within striking distance.” “We’re just not in a world where polls provide that kind of 100 percent, 0 percent win probability,” he said. “You can get odds, you can get probabilities, you can get sort of levels of certainty. But we’re in an incredibly polarized moment, and [polling] is just not a precision tool that can provide certainty.”