« Kerry Wins! | Main | When Lesbians Attack »
Margins
From Electoral Vote.com: "An American Research Group poll just released shows Kerry with a 48% to 45% lead in Ohio, with Nader at 2%. The MoE is 4%, so this is a statistical tie." Now as we've previously discussed this "statistical tie" concept is a bit dodgy. I can't do the requisite calculations, but I think the piece of stats knowledge people would like to have is an answer to the question, "with what level of confidence can we say that Kerry is ahead," or, equivalently, "if we conducted 100 methodologically identical polls during the same time interval, how many of them would show Kerry in the lead." The number would be less than the industry standard 95 percent, but more than the 50 percent implied by the "statistical tie" lingo.
Now as commenters noted last time, sampling error isn't the only -- or even the main -- source of erroneous polling. As a result, I think one should put less weight on margin of error considerations than of cross-poll consistency. If a whole bunch of polls, each of which use slightly different methods, keep showing a narrow lead for candidate X, then candidate X almost certainly has a narrow lead. When the polls disagree, then it's very hard to know what the truth of the matter is. In a tight state like Ohio, moreover, the outcome is more likely to depend on the quality of ground operations than on public opinion per se.
August 13, 2004 | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8345160fd69e200d83430b58353ef
Listed below are links to weblogs that reference Margins:
Comments
We'll be in Dayton this weekend helping out the Dems.
Posted by: John Isbell | Aug 13, 2004 11:43:24 AM
I was thinking along these same lines a few days ago. I could have started a blog and beat you to the punch but I would have a blog with one post on it after three days. People are correct to mention MOE as it relates to a single poll but you've managed to get more meaning to a poll by adding it to others.
Posted by: LowLife | Aug 13, 2004 11:47:05 AM
OT:
Deserter!
Posted by: MattB | Aug 13, 2004 12:00:43 PM
Yes Matt, you're exactly right. Systematic error is a lot more worrisome and difficult to deal with in these kinds of polls. For what it's worth, tho, here's the requisite calculation...
If 4 is the 95% confidence level, then the standard deviation of the underlying distribution is 2.43. If the measured spread is 3, the confidence that the true number is greater than zero is 89%.
Posted by: erg | Aug 13, 2004 12:24:35 PM
Pretty simple really. The margin of error for a 95% confidence interval is 2 standard deviations. Divide the margin of error in half, and you get one standard deviation or a 68% confidence interval. For this poll, the odds that Kerry is leading is somewhere between 68% and 95%.
Posted by: AF | Aug 13, 2004 12:26:41 PM
A statistical tie almost by definition is 50-50 with some MOE. Calling 48-45 with any MOE a statistical tie is just hackery.
Posted by: Tim H. | Aug 13, 2004 1:04:19 PM
Another big problem with some media polls is that newspapers too often tend to sit on their results in order to get them into the sunday or monday editions (higher visibility, higher circulation). The chicago tribune, for instance, often holds back its results for almost a week, which can render them meaningless.
One point you made about outlier numbers. Sometimes, the outlier is correct. We've had many such cases in Illinois. Carol Moseley-Braun was way behind in all but one poll leading up to her 1998 re-election. She just barely lost. Democrat Glenn Poshard was also far behind George Ryan that same year, according to all but one poll, and he barely lost the governor's race.
Things were better this year, but the persistent staleness of the newspaper results almost always proved fatal. They didn't catch Barack Obama's rise until long afterwards (a quickie TV-financed poll had it right away, however, and scooped everyone).
Posted by: rich | Aug 13, 2004 2:09:09 PM
It's August. The richer you are, the more likely you are to be on vacation and not answering your phone for poll questions. I wouldn't trust any poll conducted in August.
Posted by: Richard Bellamy | Aug 13, 2004 2:38:24 PM
richer.
not in NC, since school has already started. you're more likely to be home with your kids.
Posted by: cleek | Aug 13, 2004 4:05:02 PM
I have looked for about an hour for something that I read yesterday by a statistician on this very subject--that is, current electoral opinion polls and what we should take from them. I never found it that thus cannot link to it, but what HE said is that the probabilities become far smaller as you move to the limits of each error bar, so that (in the example he gave), if Bush-Kerry are 44%-47% with 4% MoE, then it is only a 2% probability that Bush is actually at 48% AND that Kerry is actually at 43% (I think he came up with about 2.3% or something close to that). In other words, extremely unlikely.
pollingreport.com quotes a Lou Harris researcher saying this about MoE, however:
http://www.pollingreport.com/sampling.htm
"...we (Harris) include a strong warning in all of the surveys that we publish. Typically, it goes as follows: In theory, with a sample of this size, one can say with 95 percent certainty that the results have a statistical precision of plus or minus __ percentage points of what they would be if the entire adult population had been polled with complete accuracy. Unfortunately, there are several other possible sources of error in all polls or surveys that are probably more serious than theoretical calculations of sampling error. They include refusals to be interviewed (non-response), question wording and question order, interviewer bias, weighting by demographic control data, and screening (e.g., for likely voters). It is difficult or impossible to quantify the errors that may result from these factors."
FWIW
Posted by: Curtis Love | Aug 13, 2004 4:29:22 PM
AF -
This makes some sense but I think that you are crediting Bush with a win whenever the polling was statistically poor for Kerry. If the difference were the MoE then Pbush would be 2.5%. Divide your numbers in two. Or more precisely:
Pbush = 1-(erf( delta / std /sqrt(2)) + 1)/2 =
delta = 3%
std = 2% (half of MoE)
Pbush = 6.68 %
This assumes a two man race. Of course the problem of systematic error is more significant (who has phones, who will answer them, who lies....)
Posted by: kmeson | Aug 13, 2004 4:50:58 PM
"with what level of confidence can we say that Kerry is ahead," or, equivalently, "if we conducted 100 methodologically identical polls during the same time interval, how many of them would show Kerry in the lead."
These two questions aren't equivalent. The first has a determinate answer on the basis of one poll; he second does not. The second assumes that we already know that the poll at hand is not an outlier. The fact that we don't know this is why you're right to be interested in cross-poll consistency.
Posted by: Jacob T. Levy | Aug 13, 2004 4:59:32 PM
The second assumes that we already know that the poll at hand is not an outlier.
I don't think the fact that you don't know the true mean makes it literally impossible to solve the second problem. I'm pretty sure you could still determine an answer to the question (provided you give a confidence interval for the answer.)
The mean first poll are used to estimate the true mean. The difference bewteen the poll result and the true mean would have some probability distribution (which you would know -- because poll results follow a binomial distribution.)
Then you can estimate how many likely other polls would be to have certain characteristics.
I'm not aware of a standard test to do this, but that also doesn't mean it can't be done. (Based on what I do know about probability and statistics, it should be do-able.)
My guess is that, if the standard method does not exist, solving the would involve drinking lots of Guiness-- as this student probably did.
Of course, computers would help.
Posted by: lucia | Aug 13, 2004 6:32:59 PM
The key in interpreting polls is to remember that the stated margin of error (MoE) refers only to the percentage of an individual candidate. When you're interested in the difference between two candidates' percentages, that MoE is actually twice the published MoE. Say Candidate A polls 50%, candidate B polls 47%, and the MoE is 4%. So A has a "three-point" lead, but his true percentage could be as low as 46 or as high as 54. B's percentage could be anywhere from 43% to 51%. The best scenario for A is an 11-point lead (54 - 43 = 11) and the worst is a five-point deficit (46 - 51 = -5). Plus-three falls in the center of this range, so the 95% confidence range for A's *lead* is three points plus or minus *eight* (twice four).
Michael Laviolette
Ph.D., Statistics
New Orleans, LA
Posted by: Michael Laviolette | Aug 13, 2004 7:30:54 PM
Part of the confusion stems from the whole concept of condifence intervals and the well know z-scores (or t-scores) in stats.
It was an attempt to bring together Fisher and Pearson's two methods of being able to reject the null hypothesis. The number selected are purely arbitrary and have stuck....the common alphas of .1, .05, .01 and .001 (a three star result!!! woohooo) were picked by Fisher and others over time and have become standard fare...
As they say in statistics, surely God love .051 as much as s/he loves .05....
What is commonly refered to as the MOE is the alpha level...and the use of plus or minus implies a two tailed test. One could ask if they were using a student-t model, which has fatter tails and increases the probability of failing to reject the null (and a Type II error...or is that Type I, I get them mixed up!!).
So a MOE is the confidence interval that surrounds the null number (in this case, the poll results) and depending on the alpha level, that is the probability that the true number is beyond the confidence level.
For example, a MOE of +/- 2.5 for a result of #1) 48% and #2) 46%, tells us that 95 times out of 100, the actual percentage of support the candidates has is within 1) 50.5 - 45.5 and 2) 48.5% - 43.5%.
The confusion that the reporters convey is that it is 50-50 chance of being wrong when in fact they mean that it is possible there is a 95% chance that the "real" support level is 1) 45.5% and 2) 48.5%, which would mean the second candidate is in the lead.
But it is just as possible that the real result is 1) 50.5% to 2)43.5%....
Or even better, the real support level is in that 5% outside the results we found...
This is why you should never allow social scientists to run things...are you listening Karl Rove!!! Put down the Straussians and step back very slowly...
Posted by: Nazgul35 | Aug 13, 2004 10:50:48 PM
I agree that this isn't really a statistical tie, but the probability that Kerry is leading - based only on this poll - isn't as great as the calculations above imply.
The problem is that the MOE is actually more than 4%, in terms of the Bush/Kerry margin. That's because over 90% of the sample in this poll, and most others, are either Bush supporters or Kerry supporters. So, if you have accidentally oversampled the Kerry support, it's wildly unlikely that you've correctly sampled the Bush support - very nearly certain that Bush support has been undersampled.
However, the fact that other polls show similar results or even better significantly reduces the likelihood that this poll oversamples Kerry backers.
Posted by: Alex | Aug 13, 2004 10:51:28 PM
I can't do the math either, but I can experiment.
A useful tool for this purpose is Crystal Ball - a Monte Carlo simulation add-in to Excel.
Set up the simplest simulation as follows. Assign Kerry a normal distribution with mean 48% and standard deviation 1.5%. Then a 95% confidence interval falls within +/- 2 standard deviations, or 3% (the figure should probably be 1.96 standard deviations, but I'm sure that all the initial figures have a ton of rounding.)
Similarly, assign Bush a normal distribution with a mean of 45%, and SD of 1.5%.
Note that these distributions are independently sampled. In some cases the totals may add up to >100%; I'll ignore this as a first approximation (since the lead is the issue we're concerned about, not voter turnout).
Here's the crucial step; having entered the "raw" numbers as Crystal Ball assumptions, create a simple formula which assigns Kerry a win whenever his vote percentage is greater than Bush's, and vice versa.
On a 10,000 trial simulation (takes about 30 seconds on my laptop), a 3% lead gives a Kerry win 93% of the time. Note that this is (pretty nearly) the result obtained analytically by kmeson.
You can also simulate other leads: a direct tie (0%) obviously is 50:50; a 1% lead is 68:32; 2% is 83:17; and 4% is 97%.
Yes, it's possible, as Dr Laviolette, writes, that Bush may lead (in the scenario he describes, 50%-47% Kerry, 4% MOE) by 5 points (or greater), but it's very unlikely: a 2.5%*2.5% probability, less than a tenth of a percent. By simulation, I estimate the 95% confidence interval for the scenario he describes as Kerry lead = 8.6%, Bush lead = 2.5%. So the MOE for the lead is actually about +/-5.5%, not +/- 8%.
I'd be a bit nervous about taking one of Dr Laviolette's classes...
Posted by: Andrew | Aug 13, 2004 11:51:29 PM
Re: Andrew's simulation
The flaw is in simulating the two candidates' distributions independently because the percentages are dependent. If 52% of the voters prefer Candidate A, then you know that the total for all other responses is 48%. So for each simulated trial in a two-candidate election, you need to (1) generate a percentage for A, (2) subtract from 100 to get the percentage for B, and (3) find the difference A - B to get A's lead.
To use Andrew's example, first let's scale his 48-45 scenario to a two-horse race. We get a "true" percentage for A of 48/(48+45) = 51.6, leaving 48.4 for B and a 3.2-point lead for A. So we generate Normally distributed random numbers with a mean of 51.6 and a standard deviation of 1.5. (By the way, that would require a sample size of about 1100.) Then compute the percentages for B and the A-B differences. When you run the descriptive statistics on the differences you'll get a mean of about 3.2 and a standard deviation of about 3, just as predicted. I ran a 10,000-trial simulation myself and got these results.
Bottom line: the margin of error for the candidate's lead is twice the margin of error for his percentage. You don't even have to take my word for it. Call your local university and ask a statistician for the standard deviation of the difference between the proportions of "successes" and "failures" in simple random sampling from a binomial population. It'll come out to twice the standard deviation of either individual proportion.
-M.L.
Posted by: Michael Laviolette | Aug 14, 2004 5:35:53 PM
Jeez, this is not such a hard problem
The only person in this thread who seems to make sense is Michael Laviolette, who says that this poll shows Kerry in the lead by 3 with MoE 8. Coming to this conclusion is the only tricky part: I guess a PhD in statistics really is worth something. I recently discussed the confidence interval for the lead on my blog:
http://ragout.blogspot.com/2004_08_01_ragout_archive.html#109150964383160017
and in several follow-up posts.
The estimated lead is normally distributed, with standard error 4. Hence, we can calculate the following confidence intervals:
Standard
Conf. lower upper Deviations
95% -5 11 1.96
90% -3.5 9.5 1.64
80% -2 8 1.28
55% 0 6 0.75
So, the 55% confidence interval says Kerry is ahead.
Equivalently, if Kerry and Bush were actually tied, there's a 45% chance that a poll with MoE 4 would find one of the candidates 3 points or more ahead. There is a 22.5% chance that a poll would show Kerry 3 points or more ahead.
Posted by: Chef Ragout | Aug 15, 2004 2:37:20 AM
Er...
Chef, Michael: neither of you seems to be answering the question as Matthew posed it. He's looking at a survey where 48% of responednts said "Kerry", 45% said "Bush", and 7% gave some other response (Nader, Buchanan, undecided, won't vote, who knows...).
To be fair, I don't think my first experiment was well constructed, so I've tried again. You're welcome to duplicate it, and (as always) point out any flaws in my reasoning.
Assume that we have a poll with results as shown above, and in Matthew's post (48:45:7, MOE is 4%). Construct a sample with N independent voters: I use Excel, with a RAND() function to assign "voters" to K, B or O. Tune the sample size so that the MOE reported for the candidates is 4% (or 3%, or any number). I found that 600 voters (like Chef in his blog, which I was not aware of last night) gave a 4% MOE for the two major candidates, while 1100 gave a MOE of 3%. Interestingly, the MOE for "Other" is about half the MOE for "K" or "B".
In this case, "Not-K" does not automatically mean "B". I think this makes a difference.
OK, now simulate 10 000 trials. Extract the data and measure the Kerry lead for hypothetical polls(46:45:9, 47:45:8, 48:45:7, 49:45:6, sample size of 600 or 1100).
I found that the number of times out of 100 a poll might find Kerry leading (which was the question Matthew posed in the first place) was
59:69:77:84 (600 sampled, 10 000 trials) and
63:75:84:91 (1100 sampled, 10 000 trials).
On the question of confidence interval of the lead, it turns out that this also has to do with the estimate of the proportion of "Other" voters. To see why this is so, try this thought experiment: imagine that we have a very apathetic population, and our intepid pollster finds that her poll gives Kerry 8%, Bush 5%, and "Other - undecided, apathetic etc" 87%.
I think intuitively you'd expect to have a lot lower confidence in any estimate of the lead with this poll than one where there aren't many "Others". What I found (by experiment, again - my math isn't good enough to prove this) is that a good estimate of the confidence interval of the lead is the sum of the two major candidate's CIs multiplied by 1 - pOther/2.
Note that when pOther = zero (which I think is the problem analyzed by Chef and Michael) this gives the answer they calculate. But I don't think that you can just throw out the "Other" responses, at least in looking at the question originally posed.
Posted by: Andrew | Aug 15, 2004 10:05:27 AM
Andrew,
You're right that the presence of undecideds makes a difference: it slightly narrows the confidence interval for the lead. But I simulated this too at one point, and 5-10% undecideds don't make enough difference to matter, especially when you're working an MoE that's been rounded to a single digit!
Your simulation results actually show that the undecideds don't matter. Your finding that when Kerry's in the lead by 3, a N=600 poll will find him in the lead 77% of the time, is just the flip side of my calculation that when Kerry & Bush are tied, an N=600 poll will find Kerry ahead by 3 or more 22.5% of the time. Hence, if you simulated my case, you would get almost the same answer as I did analytically.
Finally, the case I discussed is: assume that Kerry & Bush are tied, what is the chance that an MoE=4 poll would show Kerry 3 or more points in the lead. If the poll outcome is unlikely, we can reject the hypothesis that they're tied. That's the conventional question to ask in (classical) statistics.
Your simulation seems to be mathematically correct, but I'm not sure if it tells you anything about how much stock we should put in the poll numbers. It does answer one of Matt's two questions, but Matt was asking the wrong question.
Posted by: Chef Ragout | Aug 15, 2004 7:33:16 PM
See Kevin Drum's post "Margin of Error" of August 19 at www.washingtonmonthly.com. A couple of statisticians at Cal State-Chico worked up a table that provides the probability that a candidate is truly ahead, given the candidate's lead and the margin of error (that is, the sample size). I haven't tried to duplicate this, but the numbers certainly make sense. I'm sure they use the same principles I've mentioned, such as the difference being more variable than the individual percentages.
Bear in mind that those calculations, like mine, refer to a single poll. Matthew also addressed the question of combining polls, which is much dicier. The practice of combining different studies to reach a single conclusion is called "meta-analysis" and is used often in medical research, but it's also very controversial. The main problem lies in combining studies (or polls) with differing methodologies, questions, etc. Proceed with caution.
--M.L.
Posted by: Michael Laviolette | Aug 19, 2004 11:58:50 AM
Hi I have been given the task of getting links for our websites thathave good page rank on the links directories.In addition we have many categories so your site will be place on an appropriate page. If you would like to trade links please send me your website details.Best Regards,[email protected]
http://kadera.bravehost.com
Posted by: casino | Sep 25, 2005 7:36:20 AM
The comments to this entry are closed.
