Highly Touted Study Tests Non-Existent Policies, Uses Deceptive Data
From NRA-ILA, July, 2019
A study published in the journal of the American Academy of Pediatrics this week generated considerable media attention that focused on the purported finding and not the underlying research design:
Those are some pretty strong headlines. The study behind these headlines is bogus; it relies on very loose variable definitions and interpretations, seemingly prioritizes convenience over substance, and was likely only published because it found that gun-control is good.
Only half of these articles note that “children” in this study actually includes adults. This is a well-tread deception frequently used by anti-gun organizations, and we are always surprised when respectable researchers indulge in this dishonesty. For several years, we thought researchers had finally acknowledged that 18- and 19-year-old people were not children but this study adds on everyone through the age of 21.
Twenty-one-year-old people are not children. Children cannot buy alcoholic drinks or gamble in a casino. These are adults. You won’t find children enrolled in a medical school but you may find a 21-year-old adult. We suspect that these students would prefer not to be referred to as children, given that they have agency over their decisions and almost certainly have for some time.
But this study includes these adults in their analysis of “pediatric” firearm deaths. In their (only) five-year study period – despite decades of previous data and more recent data all readily available – the researchers identified 21,241 total firearms-related fatalities. The vast majority of these fatalities (69%) were of adults aged 18 to 21. This does not specify by intent; 62% of all fatalities were homicides. More than 43% of all the included fatalities were homicides of 18 to 21-year-old adults. In other words, more than two in five cases included in this analysis were adults.
A better definition of “child” would limit the included ages to those aged 14 and under. These are middle schoolers, kids just about to enter adolescence, and younger.
They’re not adults.
Focusing on actual children would have decreased the number of cases in this study to under ten percent of the number of cases actually used.
The paper begins with the claim that “Of note, ~7 US children die of firearm-related injuries daily.” The citation for this stat is the CDC WISQARS database but the daily average doesn’t line up with the definition of children used in this study. The lowest number of firearms-related fatalities since 1999 when including legal adults through age 21 is much too high to produce that number. Limiting it to children through 18-year-old-adults is close in 2017 – the most recent year of data available, which wasn’t used in the study itself – but not in any other year since 1999. If you look at actual children aged 14 and under, the number is vastly lower than what the authors claim. This begs the question – what definition of child and what year of data did they use to come up with the 7 per day average? If they did use 2017 data for children through 18-year-old-adults, why change the definition for the analysis?
And – perhaps more importantly - why didn’t they use all of the available data in their analysis?
As we noted, the outcome variable for this study of pediatric firearms-related mortality included adults. The rest of the equation – the test and control variables – were just as perplexing. Let’s start with the simpler issue: the control variables.
Research on firearms policy often includes a control variable for violent crime and age cohorts, both of which have been found to be associated with murder rates. Given that 69% of the fatalities here were homicides and the primary exposure variable was a rating of gun laws (designed to reduce crime), incorporating these would make sense. The researchers did control for race, ethnicity, educational attainment, and poverty. They also included a control for gun ownership.
The gun ownership control is based on a 2013 YouGov survey that found Hawaii to have the 10th highest gun ownership rate in the country – higher than Mississippi, Louisiana, Texas, and well, most other states. There’s a proxy for gun ownership that has been validated by researchers, and that puts the states in a vastly different order than the YouGov survey – which featured self-reported data on gun ownership in February 2013 survey. President Barack Obama and anti-gun politicians were seizing on the tragedy at Sandy Hook to push gun control. That’s the metric this “study” used.
Using the validated proxy based on 2011-2015 data, Hawaii’s gun ownership rate is more than cut in half. New Hampshire’s triples. Many states have double-digit differences between the two metrics. The pediatricians who authored this latest study split the variable in two – high and low gun ownership states. The metric used for the split is important; fourteen states fall on opposite sides of the binary categorization under each metric. This group includes low-crime, high gun ownership states like Vermont and South Dakota.
The primary test variable is the numeric grade assigned to each state by the Brady Campaign, which gave states scores between -39 to 81. The reviewers at the journal Pediatrics couldn’t be bothered to check citations, so the Giffords Law Center is cited even though the Brady Campaign to Prevent Gun Violence was the actual source of the grades. Supposedly, the Giffords Law Center (an amalgamation of the Legal Community Against Violence and Americans for Responsible Solutions) is now the sole provider of state gun law grades, but they use letter grades. The Brady Campaign issued numerical grades, and didn’t hesitate to give some states negative grades.
But the researchers here explain the Brady Campaign’s grading process as gathering “an expert panel to objectively assess and rate state firearm legislation on the basis of a series of 33 different gun policies.” If you can’t trust the organization borne out of a desire to ban all handguns to be objective about gun policy, who can you trust?
There are also more substantive methodological concerns about using a grade for a range of policies rather than focusing on a single policy. Using the grade does not allow for the identification of an effective policy. All policies are seemingly treated the same – the Brady Campaign did not make it clear if certain policies were more heavily weighted than others when determining the grades. Of course, with 31 states receiving negative grades in 2015(after nearly a decade of declining violent crime and murder rates nationwide) how serious can these grades be?
Of course, the outcome variables matter, too, so back to the children and adults being disguised as children for the purposes of gun control. The mortality data for ten states was not available, so the researchers used the mean annual mortality rate over the five-year period as these states’ mortality rates for sensitivity analysis. The data was not available because the number of fatalities fell below the reporting threshold, so the researchers assigned them the mean – the average – of the available data. That artificially inflates the number of fatalities because the mean is calculated without the lowest counts – which were suppressed for the very states for which the mean is substituting. That was just for the sensitivity analysis; in their first model, the researchers found that a 10-point increase in a state’s arbitrary “gun law” score the firearm-related mortality rate among children and teenagers and adults masquerading as children decreased by 8 points. When controlling for race, ethnicity, educational attainment, and the supposed gun ownership metric, they found that a ten-point increase in the anti-gun score is associated with a four-point decrease in the rate. We would wager that controlling for violent crime, age cohorts, alcohol consumption, and other variables commonly used in firearms policy research the effect would further decrease.
These findings are based on the more than twenty-one thousand child, teenage, and adult fatalities the authors studied. What would the result have been if they limited it to actual children?
Let’s take a look at the data for 2011-2015. There were 11,698 firearms-related homicide victims aged 15 to 21 in this limited period. The three counties with the most victims in this age range accounted for more than 13% of all such victims. These counties were Cook County, Illinois (graded 40.5 statewide in 2015 by the Brady Campaign); Los Angeles County, California (76); and Wayne County, Michigan (3, the 18th highest grade). These three counties accounted for just over 5% of the population aged 15-21 in this period. All this shows is that other factors – beyond the sheer number of gun control laws enacted – should be considered when analyzing policy.
Of course, it is also important to consider the specific policies themselves. This study only looks at three policies outside of the nonsensical grades: universal background checks for firearm purchases, universal background checks for ammunition purchases, and microstamping/ballistic fingerprinting. The pediatricians only found a statistically significant association between universal background checks and firearms-related fatalities among children, teenagers, and adults. This conflicts with another study led by well-known anti-gun researchers at UC Davis and the Bloomberg School of Public Health that found universal background checks had no effect. It also contradicts the acknowledgements of Bloomberg School professors that so-called universal background checks just aren’t effective.
But what about the other two policies – universal background checks on ammunition purchases and firearm identification (ballistic fingerprint – databases of shell casings – or microstamping). Well, the authors contend that three states had universal background checks on ammunition – Massachusetts, Illinois, and Connecticut. In each of these three states, you must have a firearms license or permit to purchase ammunition. No background check is done at the point of sale. New York and Maryland both abandoned their ballistic identification databases because they were extraordinarily expensive and predictably ineffective. California’s microstamping law is really a ban on new handguns, as the technology as required by the law does not actually exist.
Neither of these non-existent-in-practice policies was found to have a significant association with the firearms mortality rate of children, adolescents, teenagers, and fully-grown men and women through age twenty-one.
The authors acknowledge that their study did not establish causality, and that they had no metric to measure enforcement of any policy. This is a common weakness in firearms policy research, as measuring enforcement is difficult. However, the mere presence of a law absent of any enforcement is unlikely to have a chilling effect on criminals. We already know that criminals don’t get their guns legally. The Rand Corporation noted in their 2018 review of relevant research that background checks that private-seller background checks (the only sales that do not currently require a background check under federal law) have an uncertain effect on firearm homicides.
This study would not have met Rand’s criteria for inclusion in that review. The research design would not have met the qualifications as it was not designed to identify a causal effect and it utilized an aggregate state score instead of testing a specific policy. The test of so-called universal background checks, while not a test of an aggregate score, would still not have met the other requirements. We’re not sure how Rand would have handled the tests of non-existent policies, but we can make a pretty good guess.
Contrary to their claims, this study does not add anything to the body of research on firearms policy. The sloppy citation, seemingly absent peer review, intentionally limited data, and poor choice of variables should make readers wary of trusting “published” research, but the anti-gun media sees what the anti-gun media wants to see. Questions, critiques, and giant flashing neon signs calling a study unreliable at best are all ignored in the name of the cause.
No death – especially a child’s death – should be trivialized. But this study took too many liberties to be reliable and yet it is being held up as an indication that more gun control laws reduce deaths.
In the worst-case scenario, studies like this will be used to pass laws that will do nothing to stop criminals, help those facing their darkest hours, or prevent accidents. Instead, studies like this will be used to inch closer to what the Brady Campaign, Giffords, and other anti-gun organizations really want: overarching gun control.