Let’s take a moment to explain why this is, excuse my French, merde.  One Les Roberts of Johns Hopkins comes in again with a statistical fairy tale that WaPo and NYT shoot full of holes in their coverage.  Think about it.  How shoddy does your reasearch have to be in order to have both the WaPo and NYT highlight the flaws in a piece which essentially makes the Bush Administation look bad?

Pretty shoddy I’d say.

The study in question places the current “body count” in Iraq at 650,000.  Now that would tend to leave one with the impression that there are 650,000 combat dead.  But that’s not what the study says.  The study tries to link the war to other causes of death.  So it’s not only about violent death, it’s about the second and third order effects of the invasion. 

Talk about partisan, Roberts’ figure even exceeds the number (by a factor of 10) provided by Iraq Body Count, which is a notoriously partisan group and which has been debunked in many other places before, including the Weekly Standard. 

Yes, you truly have to be on thin ice on an issue when the NYT feels compelled to point out in the 5th paragraph how lousy your track record is.

The findings of the previous study, published in The Lancet, a British medical journal, in 2004, had been criticized as high, in part because of its relatively narrow sampling of about 1,000 families, and because it carried a large margin of error.

Because you see, it wasn’t enough to get it completely wrong in 2004.  Mr. Roberts and crew needed an encore.

The study relies on a stratified, multi-stage cluster sample design. Clustering of households in the sample results in variances that are smaller than you would get with a simple and random sample design.  (And if you’ve ever taken so much as an introductory statistics course you know the word “random” is a mantra to be repeated over and over…) 

So a failure to take account of this “design effect” when using your basic statistical tests in the analysis will make accurate testing difficult, because standard tests assume independence among observations and a simple, random sample design.

In other words, you can’t get there from here using the method they are using.  Well, you can, you just have to take the results with a mountain of salt instead of the usual grain.  Go ahead and roam the net for the countless articles noting the flaws in using cluster samples.  Go ahead, I’ll wait.  You should get something like 10,000 hits.

Let me now lovingly quote from WaPo and NYT:

But it is an estimate and not a precise count, and researchers acknowledged a margin of error that ranged from 426,369 to 793,663 deaths. (NYT)

Umm, I’d say a swing in raw numbers that large makes your entire methodology suspect.  No? 

And this…

This viewed was echoed by Sarah Leah Whitson, an official of Human Rights Watch in New York, who said, “We have no reason to question the findings or the accuracy” of the survey.

“I expect that people will be surprised by these figures,” she said. “I think it is very important that, rather than questioning them, people realize there is very, very little reliable data coming out of Iraq.” (WaPo)

Let me just say that when Human Rights Watch points out “there is very, very little reliable data coming out of Iraq,” it doesn’t mean that data coming out of Johns Hopkins coming out of Iraq is better.  It’s still the same unreliable data.

Violent deaths have soared since the American invasion, but the rise is in part a matter of spotty statistical history. Under Saddam Hussein, the state had a monopoly on killing, and the deaths of thousands of Iraqi Shiites and Kurds that it caused were never counted. (NYT)

So the data was lousy before, but it’s better now.  Right?  Wrong.

[Burnham] thinks further evidence of the survey’s robustness is that the steepness of the upward trend it found in excess deaths in the last two years is roughly the same tendency found by other groups — even though the actual numbers differ greatly. (WaPo)

Translation: because everyone else shows the same trend, my raw numbers must be right.  Nice try, Cupcake.

I can tell you from personal experience that polling and sampling in Iraq are notoriously difficult.  From late 2003 to early 2004 I worked on (among other things) Information Operations in Iraq.  I had a fair amount of experience working with Gallop, Oxford International and some minor polling contractors hired by USAID.  I can tell you that surveys in Iraq are notoriously unreliable for a few reasons. 

(Save the bile, I understand economic and political polling data is not a direct and unbroken parallel to public health data, there are, however, parallels that apply here.  And if the authors of that study can be fast and loose with “science” then by the rules of logic they can be refuted with equally fast and loose assertions.)

First, because of the security situation.  Your average data gathering drone doesn’t like venturing out too much to get a truly random sample because things like, oh, science and rational thought are viewed as Western “lies” to which no self-respecting Arab would subject himself. 

OK, I’m being a little facetious here (but only a little).  But it is true we had problems getting truly random samples and it is true that many Iraqis were suspicious of being asked even innocuous questions for two reasons.  One, they didn’t believe you (the pollster) really cared about what they thought and they would often say anything, quickly, to make you go away.  Which brings me to my second reason.  Being seen talking to a Westerner, or a local working for a Western firm, was often prima facie proof to the jihadists that you were a “collaborator.”

The second reason is because Arab culture is in general – oh, how to say this gently – fabulistic.  When we would get reports from locals there were not 5 jihadis around the corner, there were “dozens.”  There weren’t 5 or 6 IEDs on the road to the airport, there were “hundreds.”  You get the picture.  This deception even included on some occasions generating bogus “proof” to prove erroneous or outright false assertions.  We quickly learned not to trust one-on-one reports from any Arab unless they were backed up be multiple sources and hopefully with independent (read that “Westerner” or technological) means.

So I am pretty pessimistic, to put it mildly, on the accuracy of this study.  The WaPo and NYT note its gaping flaws, the authors have a lousy track record, and my personal experience warns me off of this.  Hey, a trifecta!