(Originally posted on 11/4/2020 on Medium)
In a digital and data-driven world, political polls have become a national obsession. For many, nervous pre-election rituals involve checking the updated poll numbers every day, even as other decry that the polls cannot be trusted.
While the results of the 2020 election are not yet certain, especially with the specter of litigation looming, we do know pre-election polling was off. By historic margins. Again.
In fact, in many states, preliminary numbers indicate that the polling errors may have been larger than in 2016.
Tweet from Derek Thompson
“Too early to be sure about the *exact* numbers here, but seems like state polls missed by 5 points, on avg, in 2016 by understating noncollege support for the GOP. Then a bunch of pollsters studied the issue and changed their methodologies for 2020 … and whiffed by 7 points.”
For example, in Wisconsin, the NYTimes reported a polling average of +10 for Biden but with pretty much 100% reporting, Biden is less than .5% ahead. According to NYTimes analysis, if polls were off by as much as they were in 2016, Biden should have been 4% up.
There will be plenty of think pieces about where the pollsters made their errors and whether these offenses were fireable or forgiveable. Some will argue that Trump is the source of the anomaly. Others will say that there are two basic problems when it comes to polling on politics. The first is a numbers problem. Survey response rates are extremely low in the modern era, in the single digits. Since those who are willing to respond to a survey may not be representative of the entire population, this alone can create enormous errors. The second problem is a human problem: namely, people lie. Or, really, they tell untruths. The reasons are manifold: embarrassment, haste, confusion, or they simply haven’t decided but want to appear decisive. Everything, from how you ask questions, to who you ask, when you ask, who does the asking, can all affect the outcome.
Whatever the reasons for the errors, progressives in particular should use this moment to think more deeply about our reliance on polls, their centrality within modern news media, and what that says about who has the right to decide what is valid and true. That is to say, this new failure in polling should cause us to consider how understanding politics through polls limits our epistemological imagination.
In an era of uncertainty and with the proliferation of data sources, perhaps it is no surprise that there is a tendency to think that only hard numbers are scientific and reliable. This is problematic both because science is not actually about establishing certainty and absolute truth but also because polling is not actually a science.
The scientific method requires a researcher to hypothesize first, to come up with a coherent and testable theory about the world. Only then do they set out to gather evidence capable of refuting that hypothesis. Political polls however, often work precisely the opposite way. Evidence is gathered from the field and then theories are grafted onto the results, to explain them. In the process, polling quirks that are the result of random chance can become the basis for narrative explanations that make it difficult to see the forest for the trees. Because polls reduce complex human preferences and psychologies into binary (or at any rate a limited set) of choices, a poll can lend itself to any number of explanatory theories. This, in turn, has led to an entire industry of political commentators and data scientists who crunch numbers and publish interpretations.
For the most part, political polling and particularily the analysis of political polls is done by highly educated people, professionals in professionalized settings. The answers they offered are clinical and (theoretically) objective. Compare this to the people who are left out of the conversation, namely activists, organizers, and the very people who policies fall most heavily upon. Sometimes, these people are left out because they are not given access to data and the resources to analyze it. Just as often however, the analysis that these folx have to offer do not lend themselves to graphs and charts because the truth of their realities cannot and should not need to be validated through polling.
Viewing politics through polls therefore limits the kinds of knowledge we can access and sets a privileged few and their mathematical models up as objective arbiters of truth. It also limits our understanding of how political change happens and how power is built. Just as commentators view a poll and attempt to graph a theory onto the numbers, politicians and policy makers often work backward from polls to build a platform. That means they are attempting to piece together incomplete snapshots of what people supposedly believe rather than building a coherent narrative about what ails our country and what can lift our communities up.
An obvious question is what we should do instead. The current election cycle offers a clue. First, if Biden wins the election, it will not be because of big data but small, not because of pollsters analyzing national figures but because of a ground game based on localized community knowledge in places like Detroit and Milwaukee, which are providing votes for the razor thin margins by which Biden is carrying Michigan and Wisconsin. Second, while polling averages were off in many key states, it seems that the final electoral count will be within the expected range based on modeling. These models incorporate polls but they combine them with fundamentals, more sociological understandings the population being modeled.
Tweet from @benjaminwittes Models. The big analytical winners here are the modelers. If you look at Nate Silver’s model, this result is at the edge of the fat part of his distribution.
For those hoping for a repudiation of Trump and Trumpism, this election is a disappointment. Preventing the next isn’t a matter of obsessive polling and then attempting to craft policies which fit those polls but something more basic: figuring out what you believe in, who you’re fighting for, and how those things can contribute to a systematic platform grounded in the real needs of those communities.