Matthew Yglesias: A Word On The Polls

« The Sea Was Angry That Day... | Main | The Radicals »

A Word On The Polls

I did a post on Tapped the other day explaining that the tracking polls whose daily shifts have become an obsession for many people of my acquaintance aren't really shifting, and don't really show anything -- these are just fluctuations due to sampling error. Now that Zogby and others have started doing some daily tracking polls in the battleground states, a similar problem is afflicting the popular electoral vote poll aggregators where random fluctuations in samples are causing Ohio, Wisconsin, and Iowa to flip back and forth in a manner that doesn't necessarily mean anything.

October 28, 2004 | Permalink

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8345160fd69e200d8342489c053ef

Listed below are links to weblogs that reference A Word On The Polls:

» Gift Basket from Tom Jamme's Blog
Sweet Blessings, a new Christian-based online shop featuring cookie bouquets, candy bouquets and gift baskets, opens with a campaign to donate a portion of all profits to Habitat For Humanity. The devastation of hurricanes Katrina and Rita, while not a... [Read More]

Tracked on Oct 7, 2005 4:24:03 AM

Tracked on Oct 7, 2005 4:26:02 AM

Tracked on Oct 7, 2005 4:28:03 AM

Tracked on Oct 7, 2005 4:30:09 AM

Comments

I'd edit "doesn't necessarily mean anything" to "doesn't mean anything". A good statistics course will have a continuing "false pattern recognition of the day" prize-- and daily variations in local election tracking polls are a rich source of winning examples.

Posted by: A different Matt | Oct 28, 2004 10:56:31 AM

If pollsters, instead of reporting MOE, just rounded everything to (say) the nearest 5%, there'd be a lot less change, and people would realize that the daily polls are pretty useless. There would still be a misleading artifact, since 42% would be reported as 40% and 43% would be reported as 45%, but perhaps this could be eliminated by using the average od the two candidates as the base (instead of fixing the intervals as 5-10-15....). So in that case 42% and 43% would both be reported as the base 42.5%, bracketed by 37.5% below and 47.5% above.

With that method, if the the raw poll numbers moved to 46-39 a 5% lead would be reported. This is a considerable lead, but the bias of the deliberately coards system is that only leads of 5% or more would be reported at all. It still might be an over- or under-estimate of the size, but with 46-39 it's pretty reasonable to think that there's something there.

This would have the effect of making the polls seem more useless, which is the goal. Obviously the pollsters and the media have no interest in doing that, but it would be good for poll consumers, and would probably reduce poll consumption to once a week or so.

As several people said recently, though, MOA is a less important factor than poll design, including but not limited to sampling methods (e.g. Gallup's overrepresentation of Republicans).

It seems to me that someone (not me) could produce a weekly summary of polls on this principle. It would be much more boring than all the faux poll events everyone's following now, but when something showed up it would be more significant.

Of course, my method has its bias too and could be misleading in some circumstances. It seems that, for everyone not in the polling and media biz, it would be a vast improvement over the present system, though.

Posted by: Zizka | Oct 28, 2004 11:02:27 AM

"Deliberately coarse". At present pollsters are reporting more significant figures than they actually have, which is a kind of fraud.

Posted by: Zizka | Oct 28, 2004 11:05:19 AM

The question remains: Why is this even close???

Posted by: Grumpy | Oct 28, 2004 11:19:56 AM

Right. It is silly to induce heart palpitations over fluctuations that are the inevitable consequence of statistics and structural inability to sample certain voting groups. The polls are accurate to +-5%. Remember, as DonkeyRising points out, most of the polls had the Shrub up 3-4% on election day 2000, and he lost.

Posted by: Bob H | Oct 28, 2004 11:27:51 AM

Actually, what I think is the fraud is that they report their numbers as if the only error came from a limited sampling size. Whereas in fact the error is likely much larger because of problems with selective sampling, voter projections, and turnout models.

Posted by: Doug Turnbull | Oct 28, 2004 11:29:35 AM

Has anyone ever looked into the morality of polling? I think polls are increasingly used as a tool to sway voters instead of informing the public. What is the justification to constantly tell a public that a particular candidate "has established a 2% lead" when the MOE is 5% or another is "gaining momentum" because of a 3% fluctuation? The statisticians know that it is meaningless but the media jumps on it anyway.

Exit polls are used to tell voters that their state may already be decided before all of the votes are cast. What is the purpose of trying to find this information before the process is complete? Is the public's right to know more important than providing an unmolested vote?

I would have a problem with them if they were 100% accurate but considering the unexplained inability for multiple polls to correlate, the lack of standardized definitions (likely voter), and the MOE that swallows most of the fluctuations, I look at them as being pure propaganda or media fluff.

Posted by: CJ_n_PA | Oct 28, 2004 11:35:08 AM

Got that right! Heck, they often don't even report the sampling size error correctly, reporting it for the overall sample, and then talking about subgroups without warning that the sampling size error goes up accordingly.

Posted by: Brett Bellmore | Oct 28, 2004 11:38:38 AM

Good. Flip-floppers will vote for Kerry.

Posted by: praktike | Oct 28, 2004 11:42:01 AM

polls should be illegal

if you've got a (R) boss like that asshat in AL are you gonna tell some pollster you're voting (D)????

Posted by: Troy | Oct 28, 2004 11:42:44 AM

Left and right find an issue on which they agree! Congrats to Matt for being a uniter, not a divider!

Posted by: Anderson | Oct 28, 2004 11:42:48 AM

The first problem is that people place such importance in the polls, the one vote people should really be concerned with is their own. The second problem is symptomatic of the American public’s need to win, those in the middle who will throw their vote to a candidate based on the carefully manicured perception rather than on how that person represents them. I hate losing, but would rather go down fighting for the things I believe than roll over to be on the winning side. The need to be ahead, particularly in these close polls, is all part of the perception that the candidate is a winner. Just like all the celebrity endorsements, why would a person really care who Springsteen or Bon Jovi voted for. Showmanship and perception. People like Zizka and Windhorse, who I don’t always agree with but can respect their beliefs and follow their arguments, are far more likely to sway my vote than some rock star, actor, or model. This is surprisingly rare, unfortunately. It's like I tell my 12 year old about his coach yelling at him, listen to what he's saying, not how he's saying it.

Posted by: Kevin | Oct 28, 2004 11:47:35 AM

I'm surprised someone hasn't mentioned the article by the WaPo's polling director (though it's understandable if you missed it since the Post stuck it in the Style section):

As the director of polling for The Washington Post, I join my fellow pollsters in the hospitality rooms at professional meetings to drink cheap wine and listen as they talk nervously about the present and agonize about the future:

Two consecutive Election Day debacles have shaken public confidence in exit polls, once viewed as the crown jewel of political surveys.

Cell phones, Caller ID and increasingly elaborate call screening technologies make it harder than ever to reach a random sample of Americans. Prompted by the popularity of do-not-call lists, a few state legislatures are considering laws that would lump pollsters in with telemarketers and bar them from calling people at home.

Costs are soaring as cooperation rates remain at or near record lows. In some surveys, less than one in five calls produces a completed interview -- raising doubts whether such polls accurately reflect the views of the public or merely report the opinions of stay-at-home Americans who are too bored, too infirm or too lonely to hang up.

Survey says: The polls are crap!

Hurray for the CBC (near the end)!

Posted by: Bragan | Oct 28, 2004 12:13:44 PM

The question remains: Why is this even close???

Same here, I agree. Forget the +-5% -- the fact that the Bushies are above 20% in the polls is absolutely inexplicable - and this is what all the polls show. Is this a hoax of some kind?

Does anyone know a real person who will vote for Bush next week? What does he/she look like?

Posted by: abb1 | Oct 28, 2004 12:30:15 PM

Abb1

I know many people who will vote for Republican candidates as well as many who will vote for Democratic candidates. They look like my neighbors, and the people I see in the street. All feel their action is in the best interest of the nation. Those who discuss why they feel the way they do and listen to opposing views I respect and value. Those who close their minds and spout talking points they know nothing about, and are unwilling to discuss their points, because they can’t, I tolerate and ignore. You are one person who talks a whole lot but listens only to what they want to hear. It creates a very narrow view of the world when you only surround yourself with those who agree with you and discount those who don’t. Abb1 try taking a Republican to lunch, then discuss anything but politics. If it really is that hard for you to find a Republican, take out an ad in the paper.

Posted by: Kevin | Oct 28, 2004 1:11:36 PM

"I did a post on Tapped the other day explaining that the tracking polls whose daily shifts have become an obsession for many people of my acquaintance aren't really shifting, and don't really show anything -- these are just fluctuations due to sampling error."

This is false, if you understand how to read tracking polls. For example, Kerry had a mini-surge this past weekend.

There is information amongst the noise.

Posted by: Petey | Oct 28, 2004 1:18:24 PM

Well, I don't know about that, Petey. WaPo and Rasmussen showed a surge toward Kerry over the weekend, but TIPP wne tht other direction. I think the best thing to do is average the 4 tracking polls each day - if you do that, it's been consistently around Bush +2 since the final debate. If it's still around there come Monday, and there are enough undecideds to make the incumbent rule give Kerry 2 more points, we'll end up with a tie. (I don't think there'll be that many undecideds, but...)

Posted by: Al | Oct 28, 2004 1:33:17 PM

Hi Kevin,
thank you for your response. I live abroad and there are no Republican here, so that's not going to work. Back in Massachusetts, however, I did know some Republicans and even one nice family from Alabama (wife was a church minister) who always voted Republican.

Well, that's different: not all Republicans on a ballot are deranged homicidal messianic megalomaniacs. I am trying to imgine a real person who will actually vote for the Bushies next week. I'll then try to multiply this image by 50-60 million and see if the polls start making more sense to me, because now they sure don't.

Thanks.

Posted by: abb1 | Oct 28, 2004 1:38:15 PM

abb1

well, abroad is very general, I just finished 4 years in Germany and learned quite a bit, but only enough of the language to order dinner or get directions. I have already voted for Bush, have many people I know. I also voted for a few democrats on the state level who I know to be very capable people who share my views on things. As somebody who has been shot at in Iraq, I don't hold malice towards Bush for our presence there, I fully support it. Neither do I hold malice to the people of Iraq. Most of the people I met there were very decent people who were anxious to move forward. The insurgency has created a whole lot of problems for those people and our pulling out will only further screw them up. So, I don't see the administration as homicidal. We are engaged in a very hard job there, but success will benefit the people of Iraq, the US and the world as a whole. But I must say I do enjoy your very strong opinions, even though I often disagree with them. I will let you know if I am ever going abroad, maybe we can catch that lunch.

Posted by: Kevin | Oct 28, 2004 1:58:34 PM

"WaPo and Rasmussen showed a surge toward Kerry over the weekend, but TIPP wne tht other direction. I think the best thing to do is average the 4 tracking polls each day - if you do that, it's been consistently around Bush +2 since the final debate."

The use of tracking polls is less for figuring out who's ahead than it is for figuring out trends. That's the real information contained within all the noise.

And that's why I used the example of Kerry having a mini-surge over the past weekend. I'm not saying that means Kerry is ahead or going to win, but if you read the tracking polls, that's something you learned that you wouldn't have known without the tracking polls.

Posted by: Petey | Oct 28, 2004 2:26:10 PM

I have found an amusing run-up-to-election activity. Whenever someone brings up polotics, specifically asking my viewpoint on a candidate, I answer, "I prefer not to discuss it." I try judging their reaction to this reasonable response.

The interesting thing is that there are many people from both viewpoints that cannot let a conversation end with that answer. I have been called a "fag" by a Republican and a "ditto-head" by a Democrat simply because I didn't engage in their political discussion.

If and only if the other person accepts my answer in a courteous manner then I will start to talk about issues while still avoiding candidates. I find that these conversations can be very interesting and informative. There are great people all over this country with a wide variety of issues that they truly care about. The difficult thing is to weed out the jackasses who prefer to shout down opposing views instead of debating them.

Posted by: CJ_n_PA | Oct 28, 2004 2:32:38 PM

So, I don't see the administration as homicidal.

Well, I don't think this is a matter of opinion. I posted this link in another thread, but here it is again:

Scientists Estimate 100,000 Iraqi Deaths

LONDON (AP) - A survey of deaths in Iraqi households estimates that as many as 100,000 more people may have died throughout the country in the 18 months after the U.S. invasion than would be expected based on the death rate before the war.
[...]
The survey indicated violence accounted for most of the extra deaths seen since the invasion, and air strikes from coalition forces caused most of the violent deaths, the researchers wrote in the British-based journal.

Most individuals reportedly killed by coalition forces were women and children,'' they said.

I don't see how murdering 100,000 people, mostly women and children can be interpreted as anything but homicidal.

Compare to this:

PM admits graves claim 'untrue'

Peter Beaumont, foreign affairs editor
Sunday July 18, 2004
The Observer

Downing Street has admitted to The Observer that repeated claims by Tony Blair that '400,000 bodies had been found in Iraqi mass graves' is untrue, and only about 5,000 corpses have so far been uncovered.

Posted by: abb1 | Oct 28, 2004 2:38:02 PM

"If and only if the other person accepts my answer in a courteous manner then I will start to talk about issues while still avoiding candidates. I find that these conversations can be very interesting and informative."

You sound like a real douchebag to me.

Posted by: Petey | Oct 28, 2004 3:04:58 PM

abb1-

trust me on this point. If we were indiscriminately killing civilians and running over that country with wanton disregard for the people there, we would see a whole lot less of our outstanding soldiers being killed. Simply removing the entire Suni triangle from existence is possible, although not in anybodies best interest. There are plans that are being executed to help get the Iraqis up and taking care of their own business. A true “deranged homicidal messianic megalomaniac” would simply opt for the complete ethnic cleansing of the region and start from there. Again, I believe that both candidates have the best interest of the nation in mind, but they differ in their approach. My fear in Kerry being elected would be that he would change direction in Iraq too quickly and cause a tough situation to become worse for all concerned. He could be elected and I could be wrong, he may do a better job I don’t know, but given what I have heard from him to this point, I suspect he would try and do something very dramatic there, and that could be very dangerous at this point.

To say that the situation in Iraq is exactly the way plans called for it to be two years ago would be ridiculous, but as I have said before, the enemy has a vote. When you get insurgents mixed with peaceful civilians, it becomes difficult to execute a plan to remove those insurgents. What I never want to see is terrorists so entrenched in this country that we are faced with daily fighting of that type here. To say that we haven’t uncovered as many bodies left by Saddam Hussein as we expected to doesn’t mean he didn’t need to be removed from power. According to your article 55 mass graves have yet to be opened up. But those are only the ones we know of for sure. While 400,000 may be a high count, I suspect 5,000 is similarly a light count. WMD programs and plans existed, mass graves existed, and support for UBL existed. Arguing extent will run us all in circles, Iraq was a threat to us under Saddam.

Petey,

I appreciate the fact that there are people out there like CJ_n_PA who can avoid a shouting match before they engage in a conversation. I am bad, because I will allow myself to get drawn into a shouting match. Nobody is ever in a good receive mode at that point, and no matter how good your argument may be on a topic, nobody is hearing it over your voice.

Zizka,

“(e.g. Gallup's overrepresentation of Republicans).” I am not disputing this statement, because I don’t know otherwise, but what is you basis for this as a blanket statement?

Posted by: Kevin | Oct 28, 2004 4:03:39 PM

The random fluctuations should be, and are, reduced by looking at the average poll of the day. Polling has reached such a manic pace that there are often more than one poll completed in a given day for Florida and Ohio.

Averaging over available polls shows almost no evidence of true changes. This is extraordinarily true if one averages over state polls to get an estimate of nationwide opinion (see my blog rjwaldmann.blogspot.com).

One way to see this is to compare simulated elections by Andrea Moro http://www.econ.umn.edu/~amoro/Research/presprobs.html
around to simulated elections by Sam Wang
http://synapse.princeton.edu/~sam/pollcalc.html

as described (without naming Wang) by his colleague Paul Krugman.

Moro uses the latest poll and his prediction bounces around a lot. Wang averages recent polls and his prediction doesn't bounce around (Wang doesn't give a graph so you have to visit his site often).

Posted by: Robert Waldmann | Oct 28, 2004 6:15:42 PM

The comments to this entry are closed.