When Experts Make Mistakes: 3 Ways to Spot Junk in Science Journals

The popular press regularly botches its science reporting. Not a week goes by without an exaggerated headline about the so-called “vaping epidemic” or a news story erroneously warning that an innocuous pesticide causes cancer.

Fact checking the media's sloppy science journalism is one of my favorite pastimes, but it also prompts an important question: why does the press bungle its science coverage so frequently? Sometimes it's because reporters are gullible, ideological, or have no particular interest in or knowledge of the topic they're covering. But another serious problem is that universities, science journals, and individual experts often get things wrong. The scientific community knows this all too well, as cancer epidemiologist Geoffrey Kabat recently explained:

[T]he public and journalists – the consumers of information about health – need to be aware of something that researchers know well – there is no [study] that is so dreadful that it cannot be published somewhere.

If you're a consumer of science news, or just a curious person looking for information on nutrition or medicine, you have to learn how to spot junk science in sources that are typically reliable and evaluate claims made by people who are usually trustworthy. So here are a few guidelines that have helped me separate sound research from misinformation, as both a science writer and a consumer.

Learn some science

This is unsexy advice. But grasping fundamental scientific concepts, like "the dose makes the poison,” is probably the best thing you can do to hone your twaddle detection skills. For example, some animal and cell culture research has suggested that using an e-cigarette (“vaping”) can increase your risk for some cancers, because nicotine and other ingredients are thought to damage DNA.

This is certainly possible but unlikely for several reasons. Nicotine in the small doses found in e-cigarettes and absent the carcinogenic chemicals in tobacco smoke is just a mild stimulant. In fact, nicotine's toxicity has been greatly overestimated based on dubious 100-year-old experiments. Additionally, animal and in vitro studies often do not translate to humans; accurately studying our behavior in a Petri dish or mouse is quite challenging, it turns out. This is clearly the case for vaping, since epidemiological and clinical studies with actual people show that switching to an e-cigarette from tobacco yields a variety of health benefits.

So know the facts; don't take it on faith that any one expert or study is telling you the whole story, which leads to another important point.

Question the experts

Americans generally put a lot of trust in scientists. This is a good quality for society to have on balance since science improves our lives in a variety of ways. But as Kabat's comment indicates, academics publish bad research sometimes. They make faulty assumptions motivated by ideological or professional considerations that skew their results and even fabricate data on occasion. In short, they're imperfect people, just like the rest of us. Once you realize that bona fide experts make mistakes, you can consider their claims critically to make sure they conform to the evidence.

For example, T. Colin Campbell's book The China Study, which extols the virtues of plant-based diets, was pulled apart by a self-described “random internet blogger” named Denise Minger. Minger dug through the peer-reviewed literature and dissected Campbell's thesis claim by claim, and her analysis matched similarly critical reviews published by qualified physicians.

Don't fool yourself

There's a crucial qualifier in order, however. Healthy skepticism and confirmation bias are not the same thing. Brushing up on your science fundamentals and questioning the experts shouldn't morph into ignoring inconvenient evidence. Sometimes scientists publish low-quality, contrarian research and people buy it simply because the results confirm their preconceptions. I could bash vaccine skeptics here, but I'll use myself as an example.

After growing up in a fundamentalist Christian church, I was skeptical of evolution as a young adult. Any scientist who challenged Darwin's theory was correct because their arguments gave me an intellectual justification for my religious views. It wasn't until I took a college biology course (taught by a Christian professor oddly enough) that I began to realize I was fooling myself while pretending to make science-based arguments.