« Seriously | Main | Precisely So »

Civilian Deaths

I was emailing back-and-forth with David Adesnik earlier today about the Lancet's civilian casualties study, and I said their 100,000 estimate didn't strike me as being nearly as implausible as he suggested, but I'd be happy to back down when faced with a legitimate debunking of the methodology. Fred Kaplan, who I'm constantly citing when he smacks Bush down, steps up to the plate noting that the survey comes with an extraordinarily large margin of error and that the authors' estimate of Iraq's pre-war death rate is considerably lower than that calculated by, among others, the UN. Color me convinced. The other, much lower, estimates out there are well within the Lancet's error band, so there's every reason to think the lower estimates are in the neighborhood of the truth.

October 30, 2004 | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8345160fd69e200d83459f12969e2

Listed below are links to weblogs that reference Civilian Deaths:

» Gift Basket from Tom Jamme's Blog
Sweet Blessings, a new Christian-based online shop featuring cookie bouquets, candy bouquets and gift baskets, opens with a campaign to donate a portion of all profits to Habitat For Humanity. The devastation of hurricanes Katrina and Rita, while not a... [Read More]

Tracked on Oct 7, 2005 5:24:02 AM

» Gift Basket from Tom Jamme's Blog
Sweet Blessings, a new Christian-based online shop featuring cookie bouquets, candy bouquets and gift baskets, opens with a campaign to donate a portion of all profits to Habitat For Humanity. The devastation of hurricanes Katrina and Rita, while not a... [Read More]

Tracked on Oct 7, 2005 5:26:01 AM

Comments

No doubt about it; OBL certainly wants to see 4 more years of Bush/Cheney. The current administration is the best thing that's ever happened to him.

Posted by: Prince Roy | Oct 30, 2004 4:28:15 AM

It's hard to see how the mortality rate in Iraq now could fail to be worse than it was under Hussein. They have less electricity, worse water, and various neighborhoods of different cities are subjected to aerial attack. Civil order is generally absent.

Iraqi locals as well as visiting Americans are afraid of going out or driving to other towns, by most accounts.

The country's a war zone. What would you expect?

Posted by: bad Jim | Oct 30, 2004 4:51:16 AM

Matt, you should also cut to the chase and mention Kaplan's estimate, which is a floor of 14,000 and probably thousands more than that (due to the media not being around to see them). And this does not include deaths from "indirect causes" or those who are wounded or disfigured.

Posted by: ploeg | Oct 30, 2004 6:34:43 AM

IBC uses press reports of deaths but we don't know how representative these are. Do they report the eventual deaths of people injured in a particular incident? Given the poor facilities in Iraq the survival rate for those seriously injured must be much worse than in the west. Kaplan himself considers the issue of under-reporting and plumps for a figure if about 1in 2, but why this figure rather than, say, 1 in 3/4/or5?
Kaplan says the post 1991 mortality rates are *murky* but he doesn't (and cannot) justify his eventual figure of 7.9 per 1,000.
It would be interesting to see figures from other conflicts. My understanding of Kosovo was that the *precision* bombing wasn't very precise at all, but I cannot find the article with the figures. And how representative were press reports of deaths in previous conflcts? For the record, I think the numbers appear high but if nothing else this has moved opinion closer to the 30,000 figure rather than the 10,000 or so that was constantly being quoted. At least these guys went to Iraq and made an attempt to find out what was happening on the ground which is more than the US government has bothered to do. *We'll bring you freedom but let's not bother counting the cost!*

Posted by: MisterP | Oct 30, 2004 7:37:52 AM

At Least 650 Billion Civilians Dead In Iraq

Maybe more.

Proud member of the "Due Check" AND "Intept Based" Community.

Posted by: Modern Crusader | Oct 30, 2004 7:43:33 AM

Kaplan says:

It means that the authors are 95 percent confident that the war-caused deaths totaled some number between 8,000 and 194,000. (The number cited in plain language—98,000—is roughly at the halfway point in this absurdly vast range.)

This isn't an estimate. It's a dart board.

Sure it's an estimate. It's a usual bell-curve-shape distribution of probablility with 98,000 being the most likely result.

The range is wide, true, but I don't see how this is a reason to question their conclusion.

Kaplan thinks 30,000 is a better estimate - this is, apparently, based on his gut feeling, and his gut feeling is produced by his consuming US media and Pentagon war-time propaganda. Fine, he has a right to an opinion.

Acording to a real research, though, 98,000 number is much more likely than Kaplan's 30,000; with 30,000 being about as likely as 130,000.

Posted by: abb1 | Oct 30, 2004 8:49:37 AM

It's important to know what the CIs are. Clearly more research is needed, but it is clear that the risk of death is higher in post-invasion Iraq than pre-invasion Iraq, and the risk of death from violence greatly increased.

But those of you who are jumping to the 8,000 extra deaths number, the lower bound of the CI, must admit that 196,000 extra deaths is just as likely. Neither is as likely as 98,000 extra deaths.

I don't know what Kaplan means when he says it's not clear what happens when you include the Fallujah sample. The paper clearly says that the increase they measured indicates 200,000 additional extra deaths in the area represented by the Fallujah sample alone, but that the sample is such an outlier that the uncertainty is huge.

The UN estimate of the pre-invasion death rate is irrelevant. Either the Lancet study gets the death rate right, in which case the UN is plain wrong, or the UN is right, in which case the Lancet study underestimates death rates due to some systematic bias. Then you have to ask whether there's any reason the Lancet study would underestimate only the pre-invasion death rate. I can't think of any reason why that would be the case, except maybe that it might be easier to remember who died in the last 18 months than in the 14 months before that. What you DON'T do is compare the Lancet post-invasion rate to the UN pre-invasion rate. That's apples and oranges, and just plain dumb.

Interestingly, Osborne Daponte is the demographer infamous for estimating that 200,000 Iraqis died in the 1991 Gulf War. Contradicting none other than...wait for it...Dick Cheney.

Posted by: Melissa O | Oct 30, 2004 8:55:45 AM

No one has yet mentioned the most startling fact about the Confidence Intervals in the Lancet study - that they just barely prove any civilian deaths at all. The bottom tail of the 95% CI is at a mere 8000. If they had used a tighter measure of confidence, say 1%, the study would have found no statistical evidence for civilian deaths in Iraq due to the war effort!

Clearly, that does not mean that no civilians died during the past year and a half, just that the study is so imprecise that it can barely prove that any civilians died at all. Plenty of reason to be skeptical about the precision of the mean.

Posted by: Andy | Oct 30, 2004 9:47:37 AM

Andy, the study says: We estimate there were 98,000 extra deaths (95% CI 8000-194 000) during the post-war period.

This is not the same as civilian deaths in Iraq due to the war effort.

You can think of this 98,000 number as the difference between the number of atrocities committed by the 'coalition of the willing' and the number of atrocities that would've been committed by the Saddam government and the sanctions if the invasion had not happened.

Therefore 0 or a negative number is not impossible in this study.

Thanks.

Posted by: abb1 | Oct 30, 2004 10:28:18 AM

If the study had argued that 100,000 had died from a variety of causes, war, increased crime, increased disease, I'd be willing to argue about it. Instead their claim is that 80-90% were caused by war, and that the large majority of those were caused by coalition aircraft and artillery. To get figures like that, you have to assume carpet bombing/destruction of Dresden-like results during the war, or you have to assume that since the war we've been killing an average of 75 civilians a day, every day. Assume a normal ratio of at leat 3 wounded to 1 killed, and you get 300 casualties from artillery and aircraft a
day, every day. Moreover, the study's cluster sample found violent deaths in only half of their clusters, so that's 300 casualties a day concentrated in half the country. Is it likely that 75-80% of these would simply escape notice? For one thing where are the bodies? If the argument was more multi-causal, I could see there being a large increase in cholera or something, but not as it stands.

For a further methodological critique:
http://www.chicagoboyz.net/archives/002543.html

Posted by: rd | Oct 30, 2004 10:45:45 AM

So, what if it is really 50K or 25K? Does that change the fact that it is mass, criminal murder?
And what does it say about us as a people that the only people who seem to care are our liberal bloggers? Not a word on this on any TV channel I watch.

Posted by: Bob H | Oct 30, 2004 10:48:34 AM

I have not yet read the actual Lancet study, but i read spencer ackerman's interview with one of the authors and i read kaplan's piece. One point that the author made is score-settling as a factor. I also happened to re-read the original NY Times article on Al Qa Qaa yesterday, and i had failed to note my first time through the reference to "headless bodies" being dumped there.

I personally don't have enough grounding in statistics or in medical studies of this sort to weigh in an opinion about the likelihood of one level of death as opposed to another, but i do suspect that kaplan (who is one of my absolute favorites) doesn't take into account the "score-settling" contributions to mortality....

Posted by: howard | Oct 30, 2004 10:54:55 AM

Matt -

Remember casualties (injured or dead) are not the same as fatalities (dead). 100K CASUALTIES is totally possible. 100K FATALITIES is not.

Posted by: bob | Oct 30, 2004 11:01:15 AM

The title of the article is clearly tendentious:

It says "100,000 deaths, or 8,000?" Well, based on the argument made, if it is unbiased, it should read "194,000 deaths -- or 8,000?" The author inexplicably and inexcusably deletes the whole upper half of the confidence interval in his estimate. Why? Because he is trying to establish a frame that debunks the Lancet's number as too high. No other reason. The poster upthread, who said that a higher CI would indicate NO deaths is similarly tendentious. Iraq Body Count's ducumented tally represents an absolute floor for the death total, and they ONLY document violent deaths. There is probably some clever statistical way to combine the survey data with documentation efforts like IBC that would normalize the survey to regions where IBC feels like it had pretty good coverage. I'll bet that that would yield a number closer to the 100,000 estimate than 8,000, and I'm equally sure that the author of this death-denial article knows it too!

Posted by: humberto | Oct 30, 2004 11:50:43 AM

Kaplan's article title should have read something like: "100,000 deaths? Or 198,000? Or 8,000?". He does not sufficiently emphasize the upper bound of the confidence interval. The confidence interval is the problem of sampling errors (just happening to pick clusters which had high or low deaths).

More important, it seems to me, are the non-sampling errors. It's just a lousy way to set up a study, from a statistical/scientific viewpoint. Kaplan's critiques on this seem more on target.

It also seems morally/politically questionable to lump together
A) violent combatant deaths caused by the ruling authorities, (whether Saddam or coalition, whether the people killed by either were carrying arms against Saddam or the coalition)
B) violent non-combatant deaths caused by the ruling authorities
C) violent civilian deaths caused by those resisting that authority (whether Kurds killing Arab civilians before the invasion, jihadists setting off carbombs after, or violent crime)
D) extra deaths from non-violent causes (whether Iraq's flailing economy before the invasion, due to UN sanctions or Iraq's flailing economy after the invasion, due to chaos and infrastructure destruction)

In particular, it's morally/politically questionable to conflate A, B, and C.

If you're the legally legitimate occupying authority, as the UN said the coalition was, in which case you're responsible for A, B, and D. Resistance to your legal and legitimate authority is illegal and illegitimate, thus the fault of the resisters who cause C.

More-ever, since the de jure (if not de facto, but we're talking de jure here) handover of sovereignty, the provisional Iraqi government bears legal responsibility for D. Who bears moral responsibility would seem to hinge on how much you think close the de jure is to de facto.

Posted by: Dubious | Oct 30, 2004 12:26:39 PM

I think you're all misreading the true intent of the survey. It was not meant to be a definitive assessment of how many casualties there have been as a result of the war. In fact, they claim right in the report that it's not very accurate. The true intent of the survey was to prove that a more definitive survey could easily be undertaken to determine the war's toll.

Posted by: Robert McClelland | Oct 30, 2004 1:03:59 PM

Thank you Robert McClelland. Sheesh.

Kaplan is in so far over his head with the (fairly simple) stats in that article that it's embarassing. Considering how much of the policy relevant information in today's world comes in the form of statistical data, I really don't see how we can have a well informed electorate if people (especially journalists) continue to misunderstand the goddamn things.

People have already pointed out the fallacy of assuming that all possibilities within the confidence intervals are equally likely, another major howler is when he talks about how they destroyed the randomness by using similar sites to replace ones they couldn't get to. Remember - random sampling isn't done for the sake of randomness itself, it's done because the randomness is really a proxy for "representativeness" when you don't know the underlying distrubtion of the response variable in the sample a priori. Therefore if you can't get to a site, and you are dealing with a relatively small number of sampling units, it's much better to try to go to a site that is as similar to the one you couldn't get to as possible than to select a new one randomly. The second method will lead to a bias towards more common sites.

Posted by: Jeff K. | Oct 30, 2004 2:39:40 PM

abb1 wrote "Sure it's an estimate. It's a usual bell-curve-shape distribution of probablility with 98,000 being the most likely result."

you fail statistics 101, it is hardly a bell curve.

to jeff k above, kaplan understands statistics better than you do.
extrapolating fallujah to the rest of the country (it provides 2/3 of the estimated deaths) is hardly a proxy for "respresentativeness."

Posted by: kyle | Oct 30, 2004 2:54:49 PM

Well, Kyle, why don't you explain it to me? Please enlighten. I'm waiting with bells on.

BTW, I don't know about your (or Kaplan's) expertise in statistics, but what about your reading comprehension: The estimate of 98,000 deaths is the extrapolation from the set that does not include Fallujah.

Thanks.

Posted by: abb1 | Oct 30, 2004 3:08:38 PM

It seems to be entirely too hard to admit that everyone is not omniscient these days. When I do see that, I am thankful. Thanks for taking the time to mention publicly that a published 'fact' might not be God's Own Truth.

I'll try to sustain that willingness to question things I'd like to believe over on my side of the fence, too.

Posted by: chap | Oct 30, 2004 3:08:58 PM

Jeff K.,

Kaplan might be in over is head...but he is several steps in front of you. From Kaplan's piece:

"The survey team simply could not visit some of the randomly chosen clusters; the roads were blocked off, in some cases by coalition checkpoints. So the team picked other, more accessible areas that had received similar amounts of damage. But it's unclear how they made this calculation."

As someone who does this sort of thing for a living, I know that replacing unobservable observations with substitutes can be an incredibly dicey proposition. It requires a lot of assumptions and is highly prone to error. Good researchers spell those assumptions out clearly and admit it is a major problem.

Posted by: MnZ | Oct 30, 2004 3:11:40 PM

They started a war against iran.
They rampaged and pillaged kuwait.
That forced us to have bases in saudi arabia.
having bases in saudi arabia forces bin laden attack the us.

screw the iraqis.. im sick of hearing of the plight of them and the palestinians..
screw em.

Posted by: gijoe | Oct 30, 2004 3:49:31 PM

OK MnZ - the dartboard thing - he is pretty clearly stating that he thinks that all possible values within the 95% CI are equally likely - is that not in over his head? Is that not pretty seriously in over his head? How is that several steps ahead of someone who understands that?

Second - I agree. It's always dicey to substitute sites. It's messy. But what's the better solution here? To drop the samples? To resample randomly? Considering not being able to get to a site probably covaries with probability of fatalities occuring in that site, and that fatalities are spatially autocorrelated, aren't those objectively worse responses? If it's true that they didn't spell out how they selected the other sites, that's definitely not great. But the substance of Kaplan's critique, that substition of sites is totally invalid is just wrong.

Anyway, for those and some of the other reasons cited above (i.e. pretending the confidence interval only extends down) I think the Kaplan piece is pretty lousy, but I usually really like his stuff. I maintain that news organizations should really start to think about keeping a statistician around.

I don't know about the numbers. Although I do think it is useful to have a different analysis to remind us that the 10,000 we keep hearing about is based on a lot of not very good assumptions as well. It's a human tragedy either way. Maybe not the best thing to be arguing about at this point.

abb1, thanks.

Posted by: Jeff K. | Oct 30, 2004 3:52:39 PM

Kaplan overstates what a 95% confidence interval means. It's not that the researchers are 95% sure the true number is within the C.I.

A 95% confidence interval means that if all the assumptions are met (e.g. normal distribution, random selection) and we did this same sampling approach 1,000 times, 95% of the times our C.I. would capture the true number and 5% of the time it wouldn't.

And because deaths are most certainly not randomly distributed we must question how robust their statistics are to violations of the assumptions.

Posted by: Brian | Oct 30, 2004 4:16:17 PM

I'm impressed. JeffK is aware of such terms as "covariance" and "auto-correlation." I guess that's supposed to make us believe that he has any idea of what he's talking about.

Posted by: raj | Oct 30, 2004 4:45:30 PM

The comments to this entry are closed.