Explore The NRA Universe Of Websites

APPEARS IN News

Questionable Metrics in New Study

Friday, March 29, 2019

Questionable Metrics in New Study

A Columbia University doctoral student in epidemiology and professors from the NYU School of Public Health, the BU School of Public Health, and the Penn School of Medicine published a study last week in The BMJ (formerly known as the British Medical Journal) that purports to have found that “states with more permissive gun laws and greater gun ownership had higher rates of mass shootings, and a growing divide appears to be emerging between restrictive and permissive states.”

Studies like this are often, unfortunately, publicized without much critical thought. An article in The Houston Chronicle claims that this study “pushes back against” the argument that one’s personal safety is increased by owning a firearm. This study does no such thing, but why let a detail like this derail some anti-gun media bias at its worst.

The study’s researchers used The Traveler’s Guide to the Firearm Laws of the Fifty States to give each state an annual rating between 0 (completely restrictive) and 100 (completely permissive). This is a central component of their analysis but the Traveler’s Guide was not designed for this use. The ratings in the Guide are arbitrary and seemingly give each law the same weight when some laws are more onerous to gun owners than others. Even Daniel Webster, the Bloomberg Professor of American Health at the Bloomberg School of Public Health and the Director of the Johns Hopkins Center for Gun Policy and Research, raised this issue with Vox, telling them that indices like this make “it hard to draw concrete policy lessons from findings attached to such indices.” The Vox article spreads more inaccuracies about gun laws and related research than we can address here, but including even a bit of criticism is a welcome change from how other media outlets regurgitate the flawed findings of anti-gun researchers.

This is not to disparage The Traveler’s Guide. It is both interesting and useful but it was designed as a quick reference for traveling gun owners. Using it in an attempt to quantify the differences in gun laws between states is ill-conceived at best. The study’s researchers also failed to reach out to the author of the Guide, so it’s clear they also had no additional insight into the rankings.

We couldn’t find an explanation for the scoring in the Guide, but more recent versions have detailed the reasons for a change to a state’s score for a given year. In 2018, Arizona’s score was stable from the prior year and this was the explanation: “wide open desert & Constitutional carry make it one of our best.” Arkansas saw a score increase of 3 points for enacting an enhanced concealed carry permit, while Iowa saw stand your ground, a preemption upgrade, and State Capitol carry enacted and only gained five points. Even if there were a formula behind the score, it would still be arbitrary. There are also some odd categorizations in the Guide. For example, California is listed in the 2010 edition as having “unrestricted, no permit or license required” to own a firearm but the state began requiring residents to obtain a handgun safety certificate before they could acquire a handgun in 2001.

The full data set was not available at the time of this article, but the charts included in the study show that Massachusetts, Connecticut, Illinois, and Maryland are all more restrictive than California during the study’s time period (1998-2015). Does that sound right to you? It doesn’t sound right to us.

But – again – the scores are based on a guide written for travelers and there is no described system for assigning the scores. It is arbitrary and for informational purposes only. It was not designed for use in an analytical model.

The annual score is not the only issue with this Columbia study. The researchers used a proxy for gun ownership rate that is the ratio of suicides involving firearms to total suicides in a state. This proxy is widely accepted but is not typically used as an independent variable, the central variable of interest in an analysis. Some reasonable control variables were included, but violent crime rate and age cohorts were not. There was no mechanism to control for enforcement of laws, which is important when comparing a range of policies across states. The outcome variable presents the next considerable issue with this analysis. The outcome variable is the number of mass shootings per million people in a state, drawn from the FBI’s Uniform Crime Report Supplementary Homicide Reports, which does not include all homicides from all states and do not include the state of Florida at all. Excluding one of the most populous states is odd.

Let’s look at the data the researchers did use. They “found” 344 mass shootings from 1998 to 2015. The Mother Jones website lists 51 for the same time period. Mother Jones excludes incidents that occur as part of another crime (like a robbery, a home invasion, or gang activity) and focused on incidents in public places. The Columbia study apparently used a broader definition which yields a considerably higher number. It is important to consider the definition used. The researchers defined a mass shooting as “one event in which four or more individuals were killed by a perpetrator using a firearm and the perpetrator themselves did not count toward the total number of victims.” This sounds reasonable, but it includes targeted attacks, domestic incidents, and other criminal activity. The phrase “mass shooting” invokes the sort of random, public rampage as defined by Mother Jones.

Vermont had the highest rate of mass shooting deaths in this time period, at just below 0.3 per million people. The Gun Violence Archive also uses a broader definition of mass shootings and reports a single incident in Vermont in the time period: a horrific event in which a woman killed three relatives and a social worker after losing custody of her child. While this is undeniably a terrible crime, it’s not typical of the type of crime most Americans think about when discussing mass shootings. This crime also shows how even a single incident in a low population state can substantially distort a dataset when dealing with rare events. 

Readers may notice that the rate of mass shootings is presented in terms of “per million people” instead of the customary “per 100,000 people.” This is because mass shootings are fortunately rare. Public mass shootings, the sort of incidents that the public consciousness associates with the term, are even more rare.

That is part of the reason why mass shootings are so difficult to study.

These events are even harder to study when the variables used in the analysis do not measure what they are purported to measure.

 

 

IN THIS ARTICLE
Research Bias
TRENDING NOW
Baltimore Gets Serious on Crime Control, and the Results Speak for Themselves

News  

Monday, July 14, 2025

Baltimore Gets Serious on Crime Control, and the Results Speak for Themselves

As the mid-year mark of 2025 hits, a promising report on crime trends has come out of the City of Baltimore. Surprising news at first glance until you dig deeper into the policy direction the ...

U.K. Moves to Legally De-suppress Suppressors

News  

Monday, July 14, 2025

U.K. Moves to Legally De-suppress Suppressors

On July 4th, President Donald Trump signed into law his “One Big Beautiful Bill,” which included a provision that eliminated the tax stamp fee of $200, but did not deregulate suppressors under the National Firearms ...

President Trump Supports Hunting and Resource Protection with Executive Actions

News  

Monday, July 14, 2025

President Trump Supports Hunting and Resource Protection with Executive Actions

Just as the United States was preparing to celebrate 249 beautiful years, President Donald Trump signed an Executive Order on July 3rd establishing the “Make America Beautiful Again" Commission supporting hunters, outdoorsmen, and outdoor recreationists by prioritizing the ...

Legacy Media Finally Acknowledges Politization of Public Health

News  

Monday, July 14, 2025

Legacy Media Finally Acknowledges Politization of Public Health

It appears the editors of The Atlantic are finally willing to entertain an idea that has long been obvious to gun rights supporters.

House Annual Appropriations Process Update

News  

Monday, July 14, 2025

House Annual Appropriations Process Update

As the House Appropriations Committee is putting together legislation to fund the government, NRA-ILA has worked closely with policy makers to ensure several long-standing priorities for gun owners were included in the underlying bills.

U.S. House Passes Reconciliation Bill, Removing Suppressors from the National Firearms Act

News  

Second Amendment  

Thursday, May 22, 2025

U.S. House Passes Reconciliation Bill, Removing Suppressors from the National Firearms Act

Earlier today, the U.S. House of Representatives passed H.R.1 the One Big Beautiful Bill Act, which included Section 2 of the Hearing Protection Act, completely removing suppressors from the National Firearms Act (NFA).

DOJ Declines to Seek Supreme Court Review of Decision Striking Down Federal Laws Prohibiting FFLs From Selling Handguns to 18-to-20-Year-Olds

Thursday, July 10, 2025

DOJ Declines to Seek Supreme Court Review of Decision Striking Down Federal Laws Prohibiting FFLs From Selling Handguns to 18-to-20-Year-Olds

In Reese v. ATF, the Fifth Circuit held that 18 U.S.C. §§ 922(b)(1) and (c)(1)—which together forbid Federal Firearms Licensees from selling handguns to 18-to-20-year-olds—violate the Second Amendment.

NRA-ILA July 2025 Litigation Update

Thursday, July 10, 2025

NRA-ILA July 2025 Litigation Update

In the second quarter of 2025, the National Rifle Association filed two cert petitions in the U.S. Supreme Court and five amicus briefs, while continuing to litigate dozens of ongoing lawsuits across the country.

Florida: Second Amendment Sales Tax Holiday Signed by Governor

Monday, July 7, 2025

Florida: Second Amendment Sales Tax Holiday Signed by Governor

Governor Ron DeSantis recently signed the Florida Budget for Fiscal Year 2025–2026, which includes a Second Amendment sales tax holiday from September 8 through December 31, 2025. The NRA is thankful for Governor DeSantis’ strong ...

Maine: Lawmakers Call for Anti-2A Progressive Professor to Be Fired

Tuesday, July 8, 2025

Maine: Lawmakers Call for Anti-2A Progressive Professor to Be Fired

In case you missed the media firestorm last week, a progressive professor at Eastern Maine Community College in Bangor, Maine, has come under fire for her emails belittling a student for her religious beliefs and views ...

MORE TRENDING +
LESS TRENDING -

More Like This From Around The NRA

NRA ILA

Established in 1975, the Institute for Legislative Action (ILA) is the "lobbying" arm of the National Rifle Association of America. ILA is responsible for preserving the right of all law-abiding individuals in the legislative, political, and legal arenas, to purchase, possess and use firearms for legitimate purposes as guaranteed by the Second Amendment to the U.S. Constitution.