I don’t usually follow the U.S. Presidential primaries but this year I’ve made an exception. It’s a genuine contest this time, with both Republican and Democratic nominations up for grabs. I’ve been regularly browsing ABC’s The Note—the de facto newswire and tattler for Beltway insiders—looking for signs of how the candidates are fairing. As you’d expect, it’s an echo chamber: pundits riffing off of each other’s guesses and spin with reckless abandon … not to mention very thin substantive evidence about what’s really going on. Gossip thrives amid ambiguity.

It’s little wonder that the predictions have been so wrong thus far. Case in point: the prediction that Barack Obama would win the New Hampshire primary by a large (ten point) margin, which proved wrong as Hilary Clinton won by a comparable margin. As New York Public Radio’s On the Media surveyed brilliantly, the media flagellated themselves in exaggerated style over their premature coronation of Obama. These sweet and tender hooligans took turns promising to never ever do it again … at least not until the next time (as Morressey would put it). And there will be a next time because media figures made it clear that they didn’t really understand what exactly went wrong.

The best moment for me took place on The Daily Show when Jon Stewart interviewed celebrity pollster John Zogby of Zogby International. All of the pollsters got it wrong. Zogby has been the pollster with the most swagger because of his relatively good track record during the Bush Jr. era. He also raises eyebrows in the research community with his unconventional tweaks to polls (read: factoring in risky hunches dressed up as methodological choices, I suspect). When interviewed by Stewart, he was twisting in the wind, claiming he didn’t really know what went wrong. I suspect his armpits were getting quite damp. Stewart understandably asked about whether we should bother with opinion polling at all. I would sum up Zogby’s response as “Uhm …”

It’s worth pondering the real reason for the incorrect predictions: asking polling methodology to do something it can’t while not acknowledging the inherent limitations of the method. Don’t get me wrong. I’m a quantitative researcher who is fond of surveys. I have nothing against the method per se. But to be good at polling is to understand its limits. Plus, I don’t like it when professional research is made to look the fool. But the real villains in this case are the clients. The media’s blissful ignorance and lack of caution has become less tenable given that polls are getting harder and harder to perform well on a reasonable budget (hence pollsters like Zogby adding their special voodoo). As the years go by, quality polling results come with more and more footnotes, which is a non-starter when anything less than pat-answers is seen as a sign of incompetent analysis when in front of a camera.

So let’s peruse some of the footnotes.

If you were to conduct a poll, what would be a good-case scenario? Once you’ve answered that, how do the circumstances in the New Hampshire primary differ? I can’t speak for every pollster’s methods specifically but here are some things to consider.

First, there’s the issue of the time period in which the survey interviews were conducted. For most consumer surveys, people would be questioned by phone over the course of a week. That’s because you would want an equal opportunity for selection to create a purely random sample. People tend to have different patterns of phone availability throughout the week and this variability is getting larger as the diversity of lifestyles and work schedules grow. So if a questioner misses someone on Monday evening, that person might be able to catch the potential respondent on Thursday evening; whereas, another person is reachable in the evening on Monday and Tuesday, but maybe not the rest of the week. And good pollsters treat their sample of potential respondents like gold (or an even more valuable commodity) because low response rates increase the potential for error. New Hampshire was far from this scenario. Pollsters only had a few days to conduct the poll since the Iowa caucus, leaving many people in the sample unreachable. Pollsters burned through lists of at least six thousand contacts just to get about 800 responses.

(By the way, I started the discussion with consumer polling because all of the major polling firms conduct such polls as their bread-and-butter business. Electoral polls are their relatively infrequent loss-leader; that is, they are high profile but less lucrative polls that help drive commercial business through the door if things go right.)

Second, you’d ideally want that week to be rather uneventful with respect to the topic of the poll. If you ask people questions from Monday to Sunday and a major relevant event happens on Thursday, then part of your sample has information the other part does not have. The two parts of the sample are not strictly comparable. You’d want stability. Otherwise, the subject you’re studying is too much of a moving target. This is where shorter time horizons offer some advantage in situations where new information tends to pop up all the time. That’s certainly the case in New Hampshire, with competing camps bombarding the place with new information several times a day. However, even over a couple of days, your sample is not working from the same bits of information. Pollsters just can’t complete the poll fast enough in a hailstorm of news.

Then there is the issue of the “undecideds”. As a matter of convenience, pollsters reallocate undecided voters to the various candidates in the same proportions as the decided voters. The assumption is that undecideds will shake out in the same pattern as the decideds. This often holds true. But this shake-out didn’t happen in New Hampshire. Indeed, undecideds seem to be susceptible to last minute bandwagon effects, campaign blitzes, or well-seeded latent leanings. Good pollsters are more sophisticated about this in their analysis but that doesn’t get reported with the raw results. And the better media outlets will separate out the undecideds as a separate category but that’s unusual. News reports are reluctant to acknowledge that, say, 40-50 percent of voters are not expressing clear preferences. A pat answer is preferred.

Plus, the campaigns do matter in the sense that minds are changing throughout, especially in highly competitive races. So trying to get a snapshot that accurately captures everyones state of mind is nearly impossible in an affordable poll that asks questions over lengthy time spans.

Speaking of blanket coverage, opinion polls are now influencing voting behaviour among the politically aware. That’s point number four. Pollsters are reluctant to acknowledge this because it raises awkward questions about the usefulness of publicly reported electoral polling. In fairness, more research needs to be done in this area to fully understand the dynamic. But I would argue that polling affects were probably quite large in New Hampshire, the primary put under a particularly intense microscope. Obama’s campaign relies on getting usually reluctant voters to show up (notably, youth and voters of African ancestry). If the media are constantly talking about Obama’s giant lead in the polls, then many of the soft voters may be less inclined to show up for the vote. “He’s already got the race locked up,” some might say to themselves. In earlier times, that dynamic might not have taken place given the lag in news reporting and the lack of saturation coverage. But today, the media are fixated on the horse-race aspect of the campaign and it is difficult to argue that such a focus doesn’t have an effect on voting behaviour.

Fifth, some strategic constituencies are increasingly difficult to reach in phone polls. Obama is relying heavily on the youth vote whereas Clinton is pandering to middle-income women as a matter of strategic targeting. Obama seems unusually successful in mobilizing youth given that many skilled politicians have failed to do so in the past. Pollsters have to consider that young people are more difficult to reach. For example, many use only mobile phones as their telecommunications device, not telephones tied to land lines. Unless the pollster is using a random digit dialer (calling random phone numbers instead of randomly selecting off a list) then these people might not be reached, given that cell phone numbers aren’t found in published phone directories in most places. Youth are also more likely to be working during evenings and weekends, the times when pollsters tend to call. This makes it more difficult to track the youth vote, whether they show up in droves (as in the Iowa caucus) or don’t (as might have happened in New Hampshire). There are other complications of this kind.

Finally, there is something that empiricists call the social desirability effect. In the United States, this tends to be referred to as the “Bradley Effect”, named after Tom Bradley’s failed attempt at the California governorship whereby he had a polling lead that didn’t pan-out at the voting booth. The social desirability effect is the notion that a certain percentage of respondents will say to pollsters what they think is socially acceptable, even if their own views are otherwise. Yes, this is a form of lying. But unlike commonplace lying, which pollsters usually treat as random noise, this type of lying has an effect if one candidate has a characteristic that might be perceived as a problem to mainstream voters. American political observers spend a lot of time speculating about what the “Bradley Effect” is for that brand of candidate known as the “Minority Candidate.” Short story: none of them really know because it’s such a candidate-to-candidate leaning.

There are many more factors at play. For example, less urbanized states provide their own unique set of polling challenges. That’s where sober and informed analysis earns its keep. And there is the usual calculation of “likely” versus “unlikely” voters among survey respondents. Yet, given just those five considerations, it is clear that pundits shouldn’t have been so strident in their predictions given their reliance on highly qualified sources. There are always lots of asterisks. This is something to keep in mind as the other primaries take place.

Commentary by Peter Stoyko