Thoughts on a Spin Attempt

The folks at Siena College have posted another ranking of the US presidents. And it looks great … unless of course you dig into how they graded the class, and then it proves itself utter garbage. Allahpundit at Hot Air pointed out the obvious bias, and the comments alone are a hoot. And Michael Avitablile posted the link here at Wizbang earlier today.

I mentioned that I disliked how Siena produced their lame attempt to boost Obama’s cred, ahem, ‘news release’. First off, it sounds great to hear that a school contacted a bunch of historians and asked for their opinion. Problem is, the serious historians would point out that you can’t grade presidents from different historical periods on the same criteria, and you really can’t tell for a generation whether a leader’s apparent effect was short-lived or truly changed history. That makes me wonder what sort of criteria was used to call the panelists “historians”.

But the real joke was the choice of categories. The poll allowed rankings in twenty categories, weighted them equally (another blunder I will address), and produced an aggregate grade. The categories were as follows:

Background (Family, Education, Experience)
Party Leadership
Communication Ability
Relationship with Congress
Court Appointments
Handling of US Economy
Luck
Ability to Compromise
Willing to Take Risks
Executive Appointments
Overall Ability
Imagination
Domestic Accomplishments
Integrity
Executive Ability
Foreign Policy Accomplishments
Leadership Ability
Intelligence
Avoid Crucial Mistakes
Your Present Overall View

– continued –

]]>< ![CDATA[

What every single one of those categories has in common, is that they are all completely subjective. Also, many of the categories chosen do not in fact represent clear presidential ability. For example, most objective historians I have read have described Richard Nixon, William Howard Taft and Thomas Jefferson as extremely intelligent, yet no one would regard them as equals in performance of their job. Some of the categories are self-contradictory, like “Ability to Compromise”: there is no doubt that the presidents in office between 1848 and 1859 did a lot of compromising on issues like Slavery, but this was a weakness, not a strength. And whether one is liberal or conservative, a president who abandons his stated moral values in order to get along with Congress would not be seen as a good or effective leader. The Siena poll is clearly one of those attempts to start with a desired result, then work back to build support for it, rather than honestly evaluate each president’s body of work.

Before moving on, I should also like to address the methodology of the poll. A poll is worthless unless you carefully consider both its limits, and its character. For example, a poll should reflect the overall population it purports to represent; it should be obvious that if I were to take an opinion poll only of delegates to the Democratic or Republican national convention, the results would be highly skewed because the demographics would not be true to the population. Similarly, pollsters are sensitive to any criteria which would cause the results to deviate from a valid representation of the whole. Consequently, simply saying you have consulted a group of historians does not necessarily mean that you have a valid sample of most historians’ opinion, and the criteria used must be chosen for objectivity and should be consistent in application. Most public opinion polls ask the same general questions and use the same methodology and weighting to demonstrate shifts in sentiment and opinion over time. Weighting is especially important. Every election, the public is asked what issues are most important to them. Candidates who lose, especially those who lose after starting with popular support, often make the mistake of assuming the voters share their same priorities. Terrorism, for example, was a much bigger issue to voters in 2002 and 2004 than in 2006 and 2008. It would be foolish to imagine it was the same value in every election. Similarly, issues of relatively minor interest in some years become more important in others. While voters this year would probably all agree that the Economy, Terrorism, Crime, Pollution, and Taxes are all valid issues, they would not count them as equally important. And so, when grading presidents, different categories should be counted to different degrees. Also, the Siena poll asks historians to rank presidents, and uses those rankings as metric values. The problem is, in a certain area the difference between, say, Numbers 10 and 11 in a ranking may be very small in value but the difference between numbers 11 and 12 may be huge, but the ranking system would not reflect this. So even on its self-described criteria, the Siena poll is both flawed in design and invalid in methodology.

If I tried to construct a presidential poll, and I have in the past, I would start by trying to establish objective metrics. The problems begin, of course, in trying to find a consistent metric which could be used for both, say 2008 and 1800. We don’t have reported numbers, after all, for inflation or unemployment during the Adams or Jackson administrations, after all. We don’t have recorded polls to reflect public support for the War of 1812 or the Spanish-American War. So I can accept that the poll will be subjective to some degree; the important thing is to recognize that bias at the start, rather than copy Siena’s dishonest approach. What Siena should have done, is to start by asking historians to describe the qualities of successful presidents and how to quantify those qualities. They should have recognized that presidents would all be prominent in certain categories, but some would excel in specific traits. For example, the education and intelligence of every president has been well above average; it’s only the arrogance and bias of the subjective judge who presumes that a given president is stupid, as evidenced by the habit of categorizing the presidents in intellect in rough approximation of your preference. That is, most conservatives would regard Reagan as brilliant, but few liberals would so, while most liberals consider Clinton to have been very intelligent as president but almost no conservative would agree in that appraisal of him. But considering the degrees earned, the career path leading to the White House, and the speeches made and campaigns conducted, it is absurd on its face to regard any president as inferior in intellect. People differ in opinion on the value of insights, depth of consideration and strategic contemplation, but the difference between the most brilliant president and the least of that fellowship, is far smaller than, say, the difference between brilliant people in general and the average intellect. It simply makes more sense to look for areas where presidents were notably distinct in their performance, establish criteria and quantified values for them ahead of assigning any specific president, then find as objective a fashion as can be done to review the actual men.

Where to go with this? This being the 4th of July weekend, we can discuss the presidents ourselves and what makes them great. I have a suspicion that we amateurs might do a better job of it than these self-impressed snots at Siena.