Conservatives and liberals have long disagreed about whether a “good guy with a gun” is the solution to stopping “bad” people who are packing heat. Settling this argument would be easier if we had ample, objective data on how often people use guns for self-defense, but we don’t have much to go on, and some of the research out there on this topic just sucks.
Reason and Breitbart have recently both pushed a study claiming people use guns for self-defense way more often than criminals use guns. The problem is that the researcher’s use of data is so questionable that he pulled his original paper while he “(rethinks) the data and his conclusions.”
The researcher is criminologist Gary Kleck, whose new study attempts to back up already-contested research he and Marc Gertz did about 20 years ago. Based on a survey, they estimated in 1995 that Americans use guns in defense between 2.1 and 2.5 million times each year.
At least that’s what I found in their study. To give you an idea of how misunderstood their original paper was: I read six articles on this topic in both partisan and nonpartisan news sources, where I found six different numbers that supposedly were Kleck and Gertz’s annual estimate of the number of times guns were used defensively each year. Even in their own paper, Kleck and Gertz begin by saying their estimate is 2.2 to 2.5 million before later changing it. It’s confusing, even for people who write about data for a living.
Sometimes when things are confusing, it’s because they don’t make good sense. Even before his flawed attempt to mirror his findings from 20 years ago, Kleck’s original research was much cited by gun rights advocates, but equally criticized by others who study this kind of stuff.
For instance, this analysis from 1998 found that surveys like Kleck and Gertz’s overstated the number of times people use guns in self-defense. And the data also doesn’t “provide sufficient information to distinguish between virtuous and objectionable uses,” researchers Philip J. Cook and Jens Ludwig wrote. “Hence these estimates contribute little to evaluating the benefits of widespread gun ownership and carrying.”
Just last month, Harvard Injury Control Research Center Director David Hemenway told NPR: “The researchers who look at (Kleck’s study) say this is just bad science. It’s a well-known problem in epidemiology that if something’s a rare event, and you just try to ask how many people have done this, you will get incredible overestimates.”
Kleck’s recent study compares his own findings from 1995 to a Centers for Disease Control and Prevention survey that asked participants during the 1990s if they had used a firearm in self-defense.
The very first issue with Kleck’s research – and Breitbart and Reason’s coverage of it – is that all three claim that the CDC had studied this topic without making the data public. Kleck goes so far as to say CDC “suppressed” the information.
However, the CDC publicly posted the data Kleck used here, here and here, sharply contrasting his suggestion that it was a recent discovery and that the CDC had kept its findings secret.
CDC spokeswoman Courtney Lenard said in an email that the survey about using guns in self-defense was an optional section for the Behavioral Risk Factor Surveillance System, a national health survey. States had an option to ask questions about firearms, leading to the surveys Kleck found from the 1990s.
“As such, the data are state-specific and not nationally representative,” she wrote.
This is why Kleck withdrew his study. His recent work passed the data off as though it applied to the whole country, but it didn’t. Editor Brian Doherty put a disclaimer at the bottom of the piece he wrote for Reason, explaining that he’d learned that the CDC data only covered 15 states.
“Informed of this, Kleck says he will recalculate the degree to which CDC’s survey work indeed matches or corroborates his, and we will publish a discussion of those fresh results when they come in,” Doherty wrote. “But for now Kleck has pulled the original paper from the web pending his rethinking the data and his conclusions.”
It doesn’t appear that Reason otherwise changed its piece to reflect this new information, and neither did Breitbart.
The other issues I saw with the study are pretty in the weeds. To summarize as best I can, Kleck did math on the CDC figures to “adjust” it based on assumptions from his work 20 years ago, with the goal of making them comparable (even though they’re already not, since they both aren’t based on national data).
As far as I know, it’s not a great idea to do that. It’s like taking apples and dividing them by oranges. You can’t do it, and why the hell would you want to do that with fruit in the first place?
Furthermore, the magic tricks he performed referred to numbers and situations I couldn’t find in his original study. And the incredibly important number of people who claimed they defended themselves with a gun in the 1990s? Somehow Kleck has a different number for that now.
Although Kleck removed the study from an online library for scholarly research, his admirers have uploaded it elsewhere, which was why I was able to find it on an incredibly racist website that was entirely in Polish. The genie is already out of the bottle.
Contact Mollie Bryant at 405-990-0988 or bryant@bigiftrue.org. Follow her on Facebook and Twitter.
Follow Big If True on Facebook, Twitter and Instagram.