A recent article published in The Lancet alleges that the number of police killings of unarmed black men is causing the mental health of black Americans to decline. Police killings of unarmed black Americans started to gain a lot of media traction in the past decade - with the rise of social media, perceived racial tensions, and the Black Lives Matter movement, news of these types of events travels far and fast.
While it makes sense that violent killings of people by law enforcement may contribute to increased public anxiety and subsequent poor mental health, we still want to make sure that the conclusion is based on an objective look at the data and not just an appeal to popular belief.
So let’s break down their methodology a bit.
This study used the US CDC's Behavioral Risk Factor Surveillance System (BRFSS) survey and the Mapping Police Violence (MPV) database to draw a connection between the number of police killings of unarmed blacks to the number of poor mental health days among black Americans. The BRFSS is cross-sectional (an observational study that uses population data from a specific point in time) and nationally representative telephone survey conducted by state health departments, using a standardized questionnaire and technical and methodological assistance from the CDC. MPV is a collaboration of people collecting data on police killings nationwide.
The authors looked at the answers to the question “for how many days during the past 30 days was your mental health not good?” among black American respondents from 2013 to 2015. They compared those answers to data on police killings from MPV database, which started tracking police killings in 2013. They defined the primary exposure as number of police killings of unarmed black Americans occurring in the 3 months prior to the date of the BRFSS interview in the same state, and the primary outcome was the number of self-reported poor mental health days.
There are a few problems with this study, and here is why you should be skeptical about its claims:
The authors claim that the number of police killings of unarmed black Americans causes an increase in poor mental health days among black Americans.
This is a classic conflation of correlation and causation. By nature, cross-sectional studies are descriptive, providing information about a given population at a specific time point. Because of that, causality is difficult to establish. This study uses both cross-sectional and ecologic study designs to try to create a causal relationship between number of police killings and personal mental health of black Americans. That is just bad epidemiology, as neither of these study designs are meant to establish causality or to provide information about individuals, rather about the population as a whole.
The study results were all found to be significant (p-value <0.05), but the survey was conducted among an extremely large sample size.
The magic number of 0.05 is used to determine whether study results are significant or not, and often contributes to whether or not a study gets published. For this study, all of the p-values were <0.05, which would lead to the conclusion that all of the results were statistically significant. However, when making judgments about p-values, you have to consider sample size. As sample size increases, p-values typically tend to become smaller, unless the null hypothesis is true (in this case the null would be that number of police killings do not have an effect on number of poor mental health days on black Americans).
The sample size for this study was 38,993, and the researchers only looked at one question in the BRFSS. It's practically guaranteed that the p-values for this study would be small. In their own analysis, the authors only saw a significant change in the number of reported poor mental health days in one time point out of 12 (in a 12 month period), about 1 month after exposure to a police killing.
The authors tried to consider the effect of the media on reportings of poor mental health days, but it is likely that media coverage, social media, and cultural change play a larger role in the effect on mental health than the killings themselves.
It is almost impossible to determine the effect of media coverage, social media, and rising social tensions surrounding police killings on the mental health of black Americans in the context of this study. The researchers did not ask further questions about the subjects’ exposure to these types of news stories or social media postings. Additionally, following the killings of Michael Brown (2014) and Freddie Gray (2015), riots broke out in Ferguson and Baltimore. It would make sense that these riots could be a contributing factor to an increase in poor mental health days among black Americans. Google search results show that there were almost 2 million media reports of Michael Brown’s death. It is very likely that the media environment surrounding these events heavily contributed to this association and added to public perception that black Americans are police targets, increasing anxiety and fear among this population.
This study posits that police killings of unarmed black Americans contribute to 1.7 additional poor mental health days per person per year, or 55 million excess poor mental health days among black American adults in the USA.
This estimation is striking, but given the methodological and statistical issues with this study, you should be skeptical of this number. Social and health disparities are a real problem in America, and action needs to be taken to address these issues, but making a claim with numbers made from faulty data can be dangerous and misleading.
As I have pointed out, there are glaring issues with this study from design to lack of consideration of confounders. Future research may be warranted to examine the causal effect of police killings on mental health of black Americans, but this particular study does more to create a narrative than to scientifically prove that relationship.
