Polling guru Nate Silver is not happy. The avalanche of criticism for polling failures has reduced him to expletive-laced crankiness. Silver is doubtful the so-called hidden Trump voter exists. He also claims that since the polls called the correct winner for most states, errors are excused, but that’s like saying it’s okay to drive drunk as long as you make it home in one piece. The fact is, the polling was terrible and the pollsters deserve all the criticism they get.
With few votes left to register, we have a good handle on the final results: approximately 50.6 percent for Joe Biden to 47.7 percent for President Trump, at the time of this writing. Investor’s Business Daily got closest with a 4-point Biden advantage, but the overall RealClearPolitics average for October was a 9-point Biden lead. Special mention goes to CNN, which judged Biden to have a 12-point advantage, and Quinnipiac, which predicted an 11-point edge.
Even worse were the state polls. Quinnipiac missed Ohio by 12 points, Florida by more than 8 points, and Georgia by 7 points. Morning Consult whiffed Pennsylvania by 8 points and Florida by more than 9 points. The New York Times missed Wisconsin by 10 points.
In the Senate races, Emerson was off a whopping 15 points in Maine. Quinnipiac had Lindsey Graham in a tie race in South Carolina (he won by 10 points), Theresa Greenfield beating Joni Ernst in Iowa by 2 points (Ernst won by 7 points), and Sara Gideon ahead of Susan Collins by 12 points in September, a mammoth 21-point miss. Even races where pollsters called the winner, they flopped badly on margin. CNN had Gary Peters beating John James in Michigan by 12 points, with the final margin being less than 2 points.
More than a ‘Margin of Error’
Some polling outfits claim the misses are just “margin of error,” but this is a misuse of the term. The problems for the public polls are more systemic than simply picking out the worst misses.
I have written about margin of error before, but let me describe it in the current context, not perfect statistical theory. Margin of error is kind of a measure of bad luck. Think of flipping a coin. If you flip a coin 10 times and get seven heads, that’s a lot — but not too crazy because it’s a small sample. Now, flip that coin 1,000 times, and the split will not be 70-30; it will be much closer to 50-50. If you repeated every day for a month, some days you would have more tails than heads and vice versa. It is very unlikely you will ever get exactly 50-50.
What if you repeat the experiment, and heads is always in the lead? Sometimes it’s only by a little, sometimes by a lot. If that is the case, then the “error” is all in one direction, which is bias. You have a bad coin on your hands.
Good polling would have yielded margins of victory that see-sawed around the true result, with some a little above and some a little below. Of the last 20 national polls, however, only one overestimated in Trump’s favor. Looking at the state polls, only in Georgia and Arizona did the polls behave nearly as they should: Of the last 20 polls, 10 overestimated Biden in Georgia and 14 in Arizona, six overestimated Trump in Georgia and five in Arizona, and four and one were on the nose in those states, respectively.
Elsewhere, the results were disastrous. Not a single poll overestimated Trump in Ohio or Iowa. In fact, 13 polls had Biden winning Ohio, although he lost by 8 points. In Wisconsin and Florida, only one poll in each slightly overestimated Trump (Susquehanna by 1 point in both states), and only Trafalgar got the exact margin in each.
These two polling outfits, generally considered Republican-friendly, performed well across all battleground states, along with Rasmussen, which is also largely derided as a GOP partisan. These three did err a bit toward Trump at times but mostly by 1 or 2 points, in contrast to the public polls, which were often far off the mark. ABC/Washington Post had Biden up 17 points in Wisconsin, when the real margin turned out to be 1 point.
Public Polling Flunked
In a conversation with Wes Anderson, a top GOP consultant who runs polling for OnMessage Inc., he linked the public poll’s big misses back to bad sampling. Public polls mostly just tap into each state’s voter file, start randomly dialing, and slowly fill out their quotas based on what each pollster thinks turnout will look like. Some then supplement with internet panels. After the results come in, they make statistical adjustments based on whether they think the results are off.
In contrast, top-flight political polling firms look at the voter file, first work to develop their modeled sample (which controls for likely bias in response rates), and then start dialing. Accounting for problems on the front end is more expensive because certain voters are more difficult to reach or refuse at a higher rate. Fixing problems upfront, however, means not having to monkey with the number on the back end.
The back-end fix that public pollsters use makes their misses even worse. They know any poll is susceptible to problems, such as non-responsive Trump voters, yet even after “fixing” the poll, their results are still way off.
FiveThirtyEight doles out letter grades to dozens of pollsters. Among the polls mentioned in this column, the New York Times and ABC/Washington Post have A+ grades, Emerson A-, Quinnipiac B+, CNN’s pollster SRSS and Morning Consult B/C. That’s some serious grade inflation.
Getting the call right should matter. There should be no A grades for these pollsters this year. Incredibly, those who did best got weaker grades: Rasmussen with a C+, Susquehanna C, and Trafalgar C-.
As for Nate Silver, clean up your language — and give yourself an F.
