The first political poll in the United States is believed to have been in 1824, when D.C. media took straw polls of the public’s preferences between Andrew Jackson, Henry Clay, John Quincy Adams and William Crawford. Jackson won the poll, and collected the most electoral votes and the largest share of the popular vote, but in the end JQ Adams claimed the White House. Not the first controversial election in the US, nor the last. But interest in polls has continued throughout the years, oddly despite little comprehension of how polls even work.
Lots of people think they know how polls work – you ask a bunch of people what they think and just total up the answers. The truth is a bit more complex. First and foremost, all polls depend primarily on who pays for them. The sponsor or patron of the poll decides the questions, the respondent pool, when and where to take the poll, and so on, so it becomes obvious that who pays for the poll drives the poll results.
From there, we usually see three types of political polls:
1. Polls designed to see who is leading in a given election – pretty basic
2. Polls which focus on a specific demographic, issue, or candidate
3. Polls designed to influence public opinion on a candidate or issue (i.e., push poll)
The first class is obvious, but the other two are important to understand. You may or may not have noticed that political candidates hire private polling firms to do internal polling, where polls are taken to learn how a specific group thinks, or which issues are the most significant to voters in a battleground state. This is why candidates keep their internal polls secret – it’s worth a lot to know what your opponent is worried about.
The third kind of poll is not often considered, but should be kept in mind. We see them all the time when a politician wants to create the impression that a position has public support or opposition. Obamacare was treated very differently in polls according to whether the people paying for the poll wanted to show it was popular, or to show it was unpopular. The way a question is phrased, who is asked and what questions precede and follow the key question all play into how respondents are likely to respond to a key question.
To put things in a kind of sense, imagine someone wants to run for President. You may have heard that when someone is thinking of running for office, they form an exploratory committee, which – among other things – will do some private polling to see if that person has a realistic chance if they run. The committee will pay for internal polling to see how well known the candidate is recognized, how voters in the party think of the candidate, and look for strengths and weaknesses to exploit or correct.
As the campaign progresses, the candidate’s team will use polling to find weaknesses in opponents, and also to sell the public on the candidate as the right leader for our time. And polls will be used to check for success in selling message and winning over key demographics.
A poll’s value lies in the accuracy of its weighting. The whole reason a poll works is because the respondent pool is adjusted to match the known demographics. If invalid assumptions are made about the people answering the poll, the results will not match actual conditions.
A poll’s accuracy proves its value, but a lot of polls mislead the public about how accurate they really are. A prominent example of this is the data nerd Nate Silver. Silver started with some good work in baseball statistics, but after 2004 started work in political poll analysis, and got the New York Times to sponsor his political poll website, fivethirtyeight.com; in 2010 the NYT paid for Silver’s blog to run under its own name. In 2008, Silver got a lot of praise, and money, for forecasting the winners of 49 out of 50 states in the Presidential Election between Barack Obama and John McCain. Silver got book deals, patronage for his website, guest appearances in TV shows … he basically became the favorite mascot of Democrats and the Left.
Silver gained even more praise and fame for correctly predicting all 50 states in 2012 when Mitt Romney challenged President Obama. Silver even appeared to do reasonably well in 2016, calling 45 out of 50 states correctly. And Silver was quick to remind everyone that ‘the polls were actually right, they were simply interpreted wrong’. I will address that claim in addressing Mr. Silver’s boast about his success in 2008 and 2012.
I took Silver at his word at first, but found trouble verifying some details on his claims. You see, it’s one thing to “call” the winner of a state. Frankly, most states are not hard to predict. Even months before the election, for example, no one would be surprised to see Joe Biden win California, New York, Illinois, and a number of predictably-left-leaning states, just as we all expect President Trump to claim Oklahoma, Texas, Wyoming and the heartland. In fact, only a handful of states are really up for change in a Presidential election, so instead of calling 50 states, an analyst is looking closely at 4 or 5 states. If we use five battleground states as the standard for deciding an election, in that context Silver got 4 out of 5 in 2008, 5 out of 5 in 2012, and 0 out of 5 in 2016, for an overall score of 9 out of 15, or just 60% (a D- grade).
To be fair to Silver, though, another, more reasonable way to grade an analyst’s work using all the states – compare the percentage of the popular vote predicted against the actual vote results in each state. I was able to work that out for 2016, but in 2012 and 2008 those details are hidden from the public at fivethirtyeight.com.
Strange, that something Silver takes such pride in, should hide the proof of his work.
Silver does not perform actual polls, of course. He does analysis of poll aggregation, meaning a meta-analysis. This practice is actually a point of debate among statisticians – some reject meta-analysis of polls as a real science, since different polls use differing methodologies, and a mix of uneven methods is just not acceptable as a scientific standard. But if your projection shows specific projected percentages for each candidate in each state, that at least provides one hundred two specific predictions which can be matched against actual results for accuracy grades. Yet when it comes to that standard, Mr. Silver seems to have ducked the challenge.
I tried to find the 2008 and 2012 specific percent projections for each state by Mr. Silver, but could only find two links which sort of addressed the question. For 2012, Nate’s blog projection was buried inside the New York Times website, which is protected by a paywall:
In that article, Silver compared 538’s state-by-state predictions of winner by margin, but did not offer the specific predictions of percentage of the vote in each state for Obama and Romney. But it does provide an interesting correction to Silver’s claim. Silver can say he correctly projected all of the fifty state winners, but if we say that 3.5% is the standard margin of error, Silver’s projected margin was wrong in eighteen states. Silver was only 64% correct in 2012 when margin accuracy is the grade. This detail is important to understanding how Silver blew his call in 2016.
As for 2008, it was even harder to find details of Silver’s prediction. I finally dug up this article, where Silver posted his predictions in a very vague matter – he gave a prediction that Barack Obama would win all of the states won by Kerry in 2004, plus “Iowa, New Mexico, Colorado, Ohio, Virginia, Nevada, Florida, and North Carolina, while narrowly missing Missouri and Indiana”. While that looks impressive (predicting 49 of 50 states), the lack of detail on the specific state results is a fumble that may hide much less impressive results. The article did post a table of selected polls which Silver seems to have used for his predictions – if so, Silver’s prediction by range was wrong (off by 3.6% or more) in twenty-seven states plus D.C., giving him a score of 45%, or an F.
In that context, small wonder Nate Silver hid the details of his predictions!
So why am I making such a big deal of the difference between guessing the winners of states and guessing the specific percentage of the vote each candidate will claim per state? Because that is the difference between guessing and real science. I noticed when I started looking at poll accuracy in 2001, that all poll analysts had bias, and those who favored Republicans tended to guess the outcome better when the Republican won, and those who favored Democrats tended to guess the outcome better when the Democrat won. And Nate Silver is a left-leaning political commentator. It’s very much consistent with the pattern that Nate was able to predicts the Presidential elections won by Democrats, but not the one won by a Republican. That means we should expect Silver to pick Biden to win in 2020, but there is not really much evidence that Silver’s prediction will be correct.
This brings me back to Silver’s comment that the polls in 2016 were correct – that is not really correct, except that the final polls – on average – predicted Hillary would beat Trump in the aggregate popular vote by about 3 percent. When you get into the details, which drove the actual results, the state polls were often wrong, and even the national polls often gave Hillary a much larger lead, cutting it down only at the end when the poll agencies had to make a call which would be compared to the final result.
Silver also made a point of blaming how polls were interpreted, which is just what he did all year long in his blog. And as for Silver’s ‘more considered’ prediction? Silver gave Clinton a 71% chance of winning the election in his final article, but most of the campaign he gave Clinton a much higher chance. But as for that 71% chance. Listen, if the weather guy tells you there is a 71% chance of a thunderstorm, you will be surprised indeed if it does not pour down cats and dogs. There’s simply no honest way to wave off Silver’s prediction, he just plain got it wrong, and frankly did so from the get-go. Anyone who understands how polls work, will understand Silver let his bias lead him to fundamental mistakes.
