Polls don’t tell the true story
by Kevin Kelton
You’ve heard of college basketball’s March Madness? This year’s presidential primary campaign should be called Polling Madness.
In the course of just three weeks, the media has reported that Democratic candidate Hillary Clinton is 11 points ahead of Bernie Sanders in Iowa, one point behind Sanders, 10 points behind Sanders, and 21 points ahead of the Vermont Senator.
On the Republican side, we’ve been told by pollsters that Donald Trump garnered 40 percent support in New Hampshire from September 3-10, but just 22 percent support from September 12-14, a mere four days later. Or we get news stories that claim Carly Fiorina has “rocketed” past Ben Carson in one national poll while simultaneously trailing him in state polls taken in Utah, Michigan and South Carolina.
It seems every day brings a new set of polls that casts either the national races or early primary state races in highly contradictory terms, and the raw data is reported without regard to the Margin Of Error. So a one point lead for a candidate is breathlessly reported as “ahead of” or “passing” his opponent, when in fact anything within 3.4 points in that poll was considered a statistical tie by the polling organization.
As pollster Nate Silver has noted, there are less quality polls these days and not enough hard analysis of the data they report. And then there are widely different methodologies that pollsters use to determine sample populations and question formatting. Some use live phone calls while others use robocalls or the internet. Which means comparing an IBD/TIPP polling outcome to a Rasmussen poll result is like comparing a baseball score to a football score.
Yet the news media uses these numbers to justify a dramatic narrative without any statistical or factual basis. And each day, they spin a brand new narrative based on a different set of numbers. Clinton is “cratering.” ‘Trump is “losing ground.” Sanders is “surging.” Fiorina is “taking away support.” All without any statistically significant justification.
So instead of reporting wild swings in races based on precise numbers, pollsters should use a less dramatic but far more accurate generalization of their results. Specifically, pollsters could report that a candidate:
- holds a solid lead
- holds a narrow lead
- a statistical tie
- trails by a narrow margin
- trails by a wide margin
Similarly, a race should only be quantified as:
- a virtual tie
After that, the newscaster can give specific numbers, but only in the context of the Margin Of Error and how this poll’s methodology may differ from other polls that have very different results.
If the news media would adopt a more responsible – and ultimately, more accurate – methodology for reporting poll results, news consumers wouldn’t be held hostage to the imperceptible but very real differences and gyrations between polls that come at them at machine gun pace.