Back in 2004, I determined a general measure of bias in the Presidential election by polls. I did the same thing for this year, which may be useful in examining the post-Convention stretch. I began by noting the 2-way poll results from Real Clear Politics from January 1 2008 through August 25.
I made a spreadsheet with the name of the group or agency which conducted the poll, the starting and ending dates of the polling, and the results for Obama and McCain. From this information, I drew up another spreadsheet which produced the daily average for the polls. I then compared each poll against the average for the days polled, to see where each poll stood against the average, noting whether Obama or McCain was getting more or less support than the average in a poll. I then was able to apply that information to determine the average bias, the aggregate variance, and essentially the stability of each poll.
I want to stop at this point, and make an observation about the fairness of these polls. As polls make statements which favor Obama or McCain, it is becoming common to see this poll or that dismissed as being unreliable or unbelievable. This is generally unwise, because in examining the polls presented, I was struck by how well, overall, most polls were conducted. While I have concerns about the demographics base used, and the weighting assumptions inherent in modern polling, the conclusions of each poll were consistent with other polls at the same time. Since twenty-six national agencies or groups have conducted national polls used for my research, such consistency indicates a high level of professional standards. However, within that acknowledgment, there were some clear trends and characteristics, which I think are worth mentioning.
As I said, there were twenty-six separate groups or agencies which conducted polls. Of these, six conducted only one or two polls this year (Battleground, McLaughlin, Franklin & Marshall, Ipsos,POS/GQR, and NPR). Accordingly, I have not considered these groups for the purposes of comparative bias.
That leaves another twenty groups, however, which have conducted a measurable pattern in their polls. For this report, I will note the five polls which most favor Obama, the five which are least favorable to Obama, the five which are most favorable to McCain, the five which are least favorable to McCain, and the five which are, as an aggregate, the least likely to be off the average measure for that date. I would like to emphasize that bias is used here not as an attack on the poll's integrity or veracity, but simply marks a tendency likely to continue in future polls between now and the election. It remains to be seen which of these polls will prove the most accurate in the final measure.
Here are the top five in each of the five categories, then:
Obama Favorable
1. ABC News/Washington Post (+3.33 pts)
2. CNN (+3.00 pts)
3. Quinnipiac (+1.97 pts)
4. Pew Research (+1.54 pts)
5. USA Today/Gallup (+0.92 pts)
Obama Unfavorable
1. Investor Business Daily/TP (-2.80 pts)
2. FOX News (-2.30 pts)
3. Cook/RT Strategy (-2.28 pts)
4. Hotline/FD (-1.62 pts)
5. LA Times/Bloomberg (1.26 pts)
McCain Favorable
1. Democracy Corps (+3.27 pts)
2. USA Today/Gallup (+2.83 pts)
3. Rasmussen (+1.89 pts)
4. CNN (+1.51 pts)
5. ABC News/Washington Post (+1.40 pts)
McCain Unfavorable
1. Investor Business Daily/TP (-4.55 pts)
2. CBS News (-2.80 pts)
3. Hotline/FD (-1.88)
4. Quinnipiac (-1.57 pts)
5. Time (-1.40 pts)
Aggregate Stability
1. NBC News/Wall Street Journal (1.81 avg total variance)
2. AP-Ipsos (2.06 avg total variance)
3. Gallup (2.17 avg total variance)
4. Pew Research (2.40 avg total variance)
5. CBS News/NY Times (2.53 avg total variance)
I will let the reader draw his/her own conclusions, after making three suggestions:
1. Notice which groups are favorable or unfavorable to both Obama and McCain
2. Aggregate Stability is a near-unknown quality, but represents a reason to have great confidence in a poll. If a poll does not miss the average by very much, it represents a reliable place for quick vitals.
3. Most of the variance shown is within the standard deviation for the population base used in the polling.



Comments (7)
I'm sure there's a good mat... (Below threshold)1. Posted by JW | August 27, 2008 6:17 PM | Score: 1 (1 votes cast)
I'm sure there's a good mathmatical explaination, but could you tell us (in layman's terms) how USA Today / Gallup is biased favorably towards both Obama (#5) and McCain (#2)?
1. Posted by JW | August 27, 2008 6:17 PM |
Score: 1 (1 votes cast)
Posted on August 27, 2008 18:17
2. Posted by hyperbolist | August 27, 2008 6:25 PM | Score: 0 (6 votes cast)
This is a good post, DJ. I'm going to share it with our statisticians and see if it doesn't tell us anything about our own company (we conduct polls for both major parties and for general consumption in the U.S.) and/or our competitors.
Just curious, but what do you conclude about IBD/TP's negative skew towards both candidates? That they're pessimistic about either candidate's prospects of steering the economy out of and away from a recession?
Is USA Today/Gallup really outside the margin of error re: Obama? Unless they've started reporting on bases above a thousand, I'd be surprised...
2. Posted by hyperbolist | August 27, 2008 6:25 PM |
Score: 0 (6 votes cast)
Posted on August 27, 2008 18:25
3. Posted by hyperbolist | August 27, 2008 6:29 PM | Score: 0 (6 votes cast)
It means, JW, that they're statistically more likely to present polling data that is favourable of Obama and favourable of McCain. Favourability ratings aren't based on preference but on overall opinion, typically--meaning, I could give John McCain a 7 out of 10 (or a "like very well" rating or whatever), but also give Obama an 8 out of 10 (or a "like extremely well"). Not exclusive of one another.
3. Posted by hyperbolist | August 27, 2008 6:29 PM |
Score: 0 (6 votes cast)
Posted on August 27, 2008 18:29
4. Posted by Knightbrigade | August 27, 2008 8:31 PM | Score: 1 (1 votes cast)
DJ, this is an EXCELLENT post. Great addition information for me to use when I start to deeply handicap this election.
4. Posted by Knightbrigade | August 27, 2008 8:31 PM |
Score: 1 (1 votes cast)
Posted on August 27, 2008 20:31
5. Posted by Just John | August 27, 2008 10:47 PM | Score: 2 (2 votes cast)
It's possible that the polls that are favorable to both are pressing the undecideds.
5. Posted by Just John | August 27, 2008 10:47 PM |
Score: 2 (2 votes cast)
Posted on August 27, 2008 22:47
6. Posted by JW | August 28, 2008 1:31 PM | Score: 0 (0 votes cast)
Thanks for the response, which now makes perfect sense of the results. Great post!
6. Posted by JW | August 28, 2008 1:31 PM |
Score: 0 (0 votes cast)
Posted on August 28, 2008 13:31
7. Posted by Larry | August 28, 2008 4:53 PM | Score: 0 (0 votes cast)
Of obvious interest is the number of people who LIE when they are polled. I have been trying to find someone who has actually been polled for many years. I was polled once several years ago. Failing to find anyone else who has been polled, I started asking a different question; if polled, would you lie.
The answers I got were interesting. A lie seems likely if a)the question was biased or stupid, or b)someone want to cover for a bias.
For example, I asked Hispanics if they were in favor of Obama. The answers I got were highly negative, yet polls show Hispanics favor Obama. Maybe it is a local deal. Hispanics and Blacks compete for the few good jobs around here.
7. Posted by Larry | August 28, 2008 4:53 PM |
Score: 0 (0 votes cast)
Posted on August 28, 2008 16:53