« Strategy Versus Truth | Main | Hadn't Realized That »

Margins Again

A little while back with reference to the "statistical tie" phenomenon, I suggested that instead of a spread and a margin of error it would be better "to publish some kind of table giving the level of confidence with which you can assert a variety of propositions about the race." Today, Kevin Drum (with a little help from some mathematicians) does the math. One issue here is that you run into some philosophical and metamathematical issues about the interpretation of probabilities. What one would like to conclude from Kevin's table is that there's a 75 percent chance that Kerry is in the lead. I think, though, that technically speaking that's only an accurate claim if you accept a Bayesian approach to statistics. Maybe that's wrong, though, and either way I can't quite explain what's at issue accurately and I know there are commenters here who understand this better than I do. I don't think, though, that there would be any harm in reporing polling results in this subjectivist way.

UPDATE: Zoe provides some explanation. As expected, I've got this kind of wrong.

August 19, 2004 | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8345160fd69e200d83421382a53ef

Listed below are links to weblogs that reference Margins Again:

» Poll Mayhem from T.J. Lang
Kevin Drum picks up on Matt Yglesias' suggestion prompted by my Margin of Ignorance piece and produces a chart to determine the probability that a candidate is ahead for any given poll. The idea is that by using this chart... [Read More]

Tracked on Aug 19, 2004 1:37:33 PM

Comments

I'm not sure what a "Bayesian approach" is, but I'm pretty sure that if you reject Bayes' law, you're not actually doing statistics anymore.

Posted by: Dave | Aug 19, 2004 10:26:40 AM

What it probably (no joke intended) means is that if you did the poll 100 times then for (about) 75 of those times, Kerry will be ahead. It could mean something a bit more technical (Kerry ahead by at least three points, etc...).

Posted by: matt | Aug 19, 2004 10:40:32 AM

No, it really isn't Baysian statistics at all.
What Kevin presents are the odds assuming that the errors in the poll are purely random. Systematic errors, by contrast, are not included. An example of systematic error would be adopting a "likely" voter model that was too heavily weighted towards one candidate.

A Baysian analysis is more difficult to quantify, but it would attempt to account for various forms of systematic bias. For example, you might try to compare two things in a data set and see if they are correlated. If you have lots of things in the data, and lots of possible ways of pairing them up, you are actually likely to find that two utterly unrelated things are correlated...e.g. people who eat mashed potatoes more than 2 times a week are twice as likely to use marijuana as people who never eat green beans.

A Baysian analysis would account for the number of times you could get results like that purely by chance; the more possible ways that you can compare things, the more likely it is that you'll see noise as a pattern.

cheers,

Marc

Posted by: Marc | Aug 19, 2004 10:46:04 AM

The accuracy of Kevin Drum's table does not require one to accept a "Bayesian approach", it is based on classical ("frequentist") statistics.

Basically, what the table says is that if I call a certain number of people RANDOMLY from the entire voting population, then there is a 75% probability that the results for the sample are the same as what one would get for the entire population. The problem is in the sampling. If I don't get a random, representative sample of the voting population, then the 75% probability is not valid.

Posted by: David | Aug 19, 2004 10:46:26 AM

Following up on my previous post.

You could consider a likely voter model to be a "Bayesian approach". A Bayesian statistical approach systematically incorporates any "prior information", including subjective opinions, into the calculation of a statistic. In a likely voter model, information about past voter turnout (by party affiliation, demographic characteristics, etc.) and opinions about potential 2004 voter turnout are incorporated into the calculation of the horserace percentages. The problem is that your prior information may be wrong (i.e., demographics have changed since 2000) and your opinions are almost certainly wrong. Furthermore, none of the polling companies actually use true Bayesian models. They take a classical model and tweak it based on prior information and opinions.

Posted by: David | Aug 19, 2004 11:32:02 AM

I think matt's trying to make the problem too abstract. look at it this way.

the math is based on a very simple sampling model, viz., a jar with a very large number (order of magnitude tens of millions) of black and white balls. you want to guess the proportion of each (actually, in this simple model the proportions are complementary, so you really only need one estimate).

one approach is to count all the balls, but that isn't practical to do frequently. when you do it's called an election (OK, OK, we don't really count all the balls - see FL 2000, but we're idealizing here). so you pull out some number of balls randomly and then compute the proportions. this is your estimate, it's statistical, it improves as you draw more balls, you apply the central limit theorem, blah, blah, and get kevin's table.

the reality, of course, is that the balls aren't only black and white and they aren't in a jar. the balls are in bins, some of which are big, new, shiny, and have individual resting pads (like an egg carton), some of which are small, old, dingy with the balls just piled in, etc. some balls are truly black and white (like maybe 40% each), some are grey, some are paired so that they're always the same color and should be drawn as a couple, etc. etc. thus, you have to decide how to pick your samples from the individual bins and decide how to deal with the grey balls, etc. that's why there are professional pollsters instead of just automated random telephoning machines.

BTW, in the simple model (black and white balls in a jar), to improve the accuracy, just pick more balls (ie, a larger sample size). in the complex model (ie, the real world), multiple independent polls may improve (if they tend to agree), or degrade (if they don't), your confidence, as matt observed in round one of this fun, but fruitless, exchange.

Posted by: Charles Wolverton | Aug 19, 2004 11:40:46 AM

At this level Bayesian and frequentist approaches usually end up being the same - it's a difference in philosophical approach rather than the answers you get at the end (with this sort of problem, anyway). I've written about Bayesians vs. frequentists some more over at Greenpass.

Charles Wolverton, your visual example of why stratified and cluster samples are useful is a wonderful one. I promise to give you full credit when I get to teach some stats sections next year :-)

Posted by: Zoe | Aug 19, 2004 11:53:55 AM

Actually I believe what the 75% means is that if the Bush and Kerry were actually tied then there is only a 25% chance that a poll (with only sampling error) would find Kerry leading by that much or more.

This does not mean that there is a 75% that Kerry is ahead. For example there is only a 25% chance that flipping a normal coin twice will produce 2 heads. This does not mean that if we take an unexamined coin out of our pocket, flip it twice and get two heads that we can conclude that there is a 75% chance that the coin has two heads. This is because (the Bayesian part) we start with a strong presumption that coins don't have two heads. Similarly our estimate of the chance that Kerry is ahead after we learn of the poll will depend on what we thought (and how strongly) about the race before. Obviously this is subjective so there is no objective way of generating a probability that Kerry is ahead from such a poll.

Posted by: James B. Shearer | Aug 19, 2004 12:19:55 PM

Flipping a coin two times does not provide a large enough sample (i.e., the MoE is very large). However, if I flip the coin one hundred times, then I will get a good estimate of the probability of heads if I flipped the coin an infinite number of times.

The magic of statistical induction is that I can poll 1,000 voters (assuming that I sample properly) and then make an inference about the results I would get for all voters in the U.S. The 75% number tells me the likelihood that my results for 1,000 random voters carry over to the entire population of voters.

Posted by: David | Aug 19, 2004 12:42:14 PM

I still think the biggest problem with reporting of horse race type polling, isn't the unsophisticated way they report the sample size error. It's the way they just ignore all the other, potentially larger, sources of error, just because they can't precisely quantify them. Anybody who thinks these polls are as reliable as they're being portrayed is being suckered.

Posted by: Brett Bellmore | Aug 19, 2004 12:45:22 PM

Brett, I don't think it is all that common to put a lot of faith in polls. On every thread about them someone is expressing your opinion. Indeed, it is 'cool' for media-types to mention how worthless polls are.

I think the problem is the opposite. Polling is indeed far from perfect. But it is the only measure we have. And they are better indicators than pundits consulting their magic 8-balls.

Posted by: Timothy Klein | Aug 19, 2004 1:52:20 PM

David that is not what the 75% means. Our estimate of the chance that Kerry is ahead after we learn the results of the poll depends on what our estimate of the chance that Kerry is ahead was before we learned the results of the poll. Since this will vary by individual there is no objective way to produce a single number.

It is true that the dependence on the prior estimate decreases as the sample size increases but it will remain significant for typical poll sample sizes.

Posted by: James B. Shearer | Aug 19, 2004 2:03:43 PM

I'm kinda late on this, but my chart is based on good 'ol frequentist statistics, although I don't think it really makes much difference.

Anyway, what that 75% number means is: there's a 75% likelihood that in the population at large Kerry really is ahead of Bush. There's a 25% chance that the result is due to sampling error and it's actually Bush who's ahead.

At least, that's what it means without getting into deep philosophical discussions of what "likelihood" really means....

Posted by: Kevin Drum | Aug 19, 2004 2:51:23 PM

Why is everyone so optimistic? Those who support Mr. Bush think that Mr. Bush will win. Those who support Mr. Kerry think that Mr. Kerry will win. The common trait is optimism. What explains that optimism?

I'm not so optimistic. I think that Mr. Kerry will win, and I may also vote for him, but I feel a little gloomy about the prospect of his likely victory.

Posted by: Arjun | Aug 19, 2004 3:10:48 PM

worth remembering in this debate, it doesn't actually matter what support for the candidates "really" is anyway, just what the result appears to be from a particular, in various ways biased, sampling conducted by local governments in early november.

Posted by: flip | Aug 19, 2004 3:54:05 PM

James, you are right, if one takes a Bayesian approach to statistics (i.e., that all statistics are informed, either explicitly or implicitly, by prior information). But a frequentist would say that any prior information is meaningless, since it is all contained in the sample anyway. Or, if you were extremely anti-Bayesian, you would assert that including prior information is a mistake because it is completely subjective.

Posted by: David | Aug 19, 2004 4:58:05 PM

"I'm not so optimistic. I think that Mr. Kerry will win, and I may also vote for him, but I feel a little gloomy about the prospect of his likely victory."

Me too. The fact that Bush is incredibly bad and immeasurably worse than Kerry does not in itself make a Kerry win a good thing, just a slightly better thing.

I am not optimistic either. In fact I am pessimistic, in that although things are really bad almost everywhere, I am certain they are going to get worse.

Posted by: bob mcmanus | Aug 19, 2004 5:53:42 PM

Man, you guys read like Rosencrantz & Guildenstern discussing what reality they're currently inhabiting in the Tom Stoppard play.

"There are three types of lies: Lies, Damn Lies, and Statistics." - Mark Twain

"Did you know that 47% of all statistics are false?" - My friend Dave Lowery

The most important 'poll' numbers are the favorability ratings within each candidate's base. Bush continues to have the highest favorability ratings since they were measured. This means that his base likes him, will not be swayed by negative advertising, and will turn out to vote.

Kerry continues to be the only candidate ever to run 'net negative' - more of his base hates Bush than likes him! These people can be swayed by logic, or by negative Kerry ads, or both. How many people will vote for a candidate they don't know or don't like (but don't dislike as much as Bush)? How many will stay home, vote for Nader, or write in Kucinich or Dean?

This dynamic is the reason why Bush will win. Love always conquers hate.

Posted by: Larry | Aug 19, 2004 8:17:36 PM

The comments to this entry are closed.