Gun rights supporters know that civilian disarmament advocates have long employed dubious social “science”/public “health” research in their mission to strip Americans of their Second Amendment rights. Worse, these political actors insist that taxpayers fund these attacks.
Last week, NRA-ILA pointed to new research showing that the social “sciences” exhibit woeful, and increasing, political bias. Compounding this problem, this week, a large team of researchers published another study showing that roughly half of social “science” research cannot be replicated. Replication is vital to determining whether a study’s conclusions are in fact valid.
The item is titled Investigating the replicability of the social and behavioural sciences, and was published April 1 in the journal Nature. An article in the journal Science, titled Across the social sciences, half of research doesn’t replicate, summarized the findings.
Describing the replication research project, Science explained.
Called Systematizing Confidence in Open Research and Evidence (SCORE), the effort investigated more than 100 papers published in dozens of leading journals in business, economics, education, political science, psychology, and sociology. The replication success rate—49% for the 164 papers evaluated, reported today in Nature—is consistent with findings from previous studies in individual fields such as psychology, suggesting the problem is pervasive in the social sciences.
Citing a scientist interviewed for the piece, the Science item noted,
Improving repeatability requires reforms in professional evaluations and funding practices to incentivize researchers to prioritize rigor and quality over the quantity of papers they publish, Cobey says. “Answering the ongoing questions of the credibility of research requires a cultural change in how we conduct research.”
The problems highlighted by this research project will be familiar to those aware of a 2015 study published in Science that detailed the results of a four-year effort to improve the accuracy of psychological science. A team of 270 scientists led by University of Virginia Professor Brian Nosek attempted to replicate 98 studies published in some of psychology’s most prestigious journals by conducting 100 attempts at replication. In the end, according to a Science article accompanying the study, “only 39% of the studies could be replicated unambiguously.” The episode, in part, led to what has become known as the replication crisis.
These problems with social “science” research won’t come as a surprise to loyal readers of the NRA-ILA Grassroots Alert.
The dubious incentives are obvious. Today, much of the social “science”/public “health” field is concerned with empowering government to dictate an ever-expanding array of individual behavior, including what people can eat, drink, read, think, drive, and how to protect yourself and your family. The bulk of the project is fundamentally opposed to individual liberty. All the while, Americans are assured that it’s necessary for an ever-growing amount of taxpayer resources to be devoted to public health “experts” studying and implementing these efforts.
It doesn’t take a public choice theorist to understand how these incentives work to create a feedback loop. Researchers have an incentive to magnify a supposed problem in hopes of receiving more funding. Politicians seeking further power are eager to fund “research” aimed at justifying that accumulation of authority, creating a symbiotic relationship between political actors and so-called “scientists.”
In the gun control context, the problem may be particularly acute.
In 2022, Reason magazine did an excellent job of exposing almost all “gun violence” social “science” for the junk science it is by producing an accessible video explainer on the topic.
Drawing on the expertise of statistician and New York University and University of California at San Diego instructor Aaron Brown and a 2020 analysis by the RAND Corporation, the video explained that the vast majority of gun violence research is not conducted in a manner sufficient to offer meaningful conclusions. An article accompanying the video, written by Brown and Reason Producer Justin Monticello, noted,
A 2020 analysis by the RAND Corporation, a nonprofit research organization, parsed the results of 27,900 research publications on the effectiveness of gun control laws. From this vast body of work, the RAND authors found only 123 studies, or 0.4 percent, that tested the effects rigorously.
Reason and Brown examined the remaining 123 studies from the RAND analysis and offered the following,
We took a look at the significance of the 123 rigorous empirical studies and what they actually say about the efficacy of gun control laws.
The answer: nothing. The 123 studies that met RAND's criteria may have been the best of the 27,900 that were analyzed, but they still had serious statistical defects, such as a lack of controls, too many parameters or hypotheses for the data, undisclosed data, erroneous data, misspecified models, and other problems.
Then there is the sheer difficulty of conducting meaningful social “science” research.
A 2022 article published in the Proceedings of the National Academy of Sciences, titled Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty, showed that researchers given the exact same data and hypothesis come to wildly different conclusions as a result of the researchers’ idiosyncratic decisions.
Explaining the results of the experiment, the authors reported,
Results from our controlled research design in a large-scale crowdsourced research effort involving 73 teams demonstrate that analyzing the same hypothesis with the same data can lead to substantial differences in statistical estimates and substantive conclusions. In fact, no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.
…
Even highly skilled scientists motivated to come to accurate results varied tremendously in what they found when provided with the same data and hypothesis to test.
...
Our findings suggest reliability across researchers may remain low even when their accuracy motivation is high and biasing incentives are removed.
In other words: Much of social science is of dubious value, even when its practitioners aren’t politically or financially-biased (which, in the real world, is rarely ever the case).
In attempting to explain the wide variation of results, the authors state,
Researchers must make analytical decisions so minute that they often do not even register as decisions. Instead, they go unnoticed as nondeliberate actions following ostensibly standard operating procedures. Our study shows that, when taken as a whole, these hundreds of decisions combine to be far from trivial.
This concept is sometimes presented as the “garden of forking paths.” Each minute decision a researcher makes in working with data or constructing a statistical model can lead to different sets of decisions down the road. These different decisions compound, resulting in extreme variations in results among even well-meaning researchers using the same data.
Whether primarily the result of political bias or the immense flaws inherent to the field, the sorry state of social “science”/public “health” research makes clear that it should have no bearing on Americans’ Second Amendment rights. Moreover, taxpayers shouldn’t be forced to foot the bill for what often amounts to overproduced hokum no more reliable than a coin flip.
More Like This From Around The NRA
