Which Health Studies Are Reliable? ACSH Explains Epidemiology

Every student in America should be required to take a class called, "What Do We Know, and How Do We Know It?" Perhaps if we learned from an early age how we know the things we claim to know, fewer Americans would fall for ridiculous conspiracy theories.

Public health is a field that is widely misunderstood, even by science journalists. That is because epidemiology is an inexact science that is complicated by a large variability in the quality of the data it produces, as well as by its reliance on advanced statistical methods. Let's leave the latter aside and focus on the former. Which epidemiological studies are most reliable and why?

From weakest to strongest, here are the most common epidemiological study designs:

Case report. A case report is the medical equivalent of "show-and-tell." Doctors find a patient with an unusual disease, and then they write a report about it. According to The Lancet, a case report "should not be a rarity but one that a general physician might encounter, in which there was some difficulty in reaching a diagnosis, and that provides a teaching point." 

Cross-sectional study. A cross-sectional study is a snapshot in time. It answers the question, "How many people have a particular characteristic (such as Disease X) at this very moment?" This study design does not provide any information on risk or cause.

Ecological study. An ecological study examines disease at a population level. A good example is a study that uses a scatter plot to correlate average alcohol consumption and the rate of coronary heart disease deaths in various countries. While such a study design provides clues for further research, it must be remembered that there is no data on individuals. All of the data is collected at the level of the population, which means that a person's individual exposure and disease outcome are not measured. Obviously, such a study design could be easily confounded by outside variables.

Case-control study. A case-control study compares people who have a particular disease ("cases") with a similar group of healthy people ("controls"). This kind of study is most useful for rare diseases. However, case-control studies are notorious for producing false positives. The reason is because people who have a rare disease are much likelier to remember their lives differently than healthy people. (This is a well-known phenomenon called "recall bias.") Consider people with brain cancer. Many of them will report using a cell phone on the side of their head where the cancer developed, regardless if it's true or not. 

Cohort study. A cohort study is the best possible observational study. In this study design, people are compared based on their level of exposure to a particular risk factor. For example, comparing smokers with non-smokers to determine which group is more likely to develop cancer 10 or 20 years later would be a prime example of a cohort study. Even though confounding variables can make a definite determination of cause difficult, the cohort study is the only kind of observational study that gives a reliable measure of risk. 

Randomized controlled trial. The randomized controlled trial (RCT) is the gold standard in epidemiology. It can determine both cause and risk. However, it is generally only ethical to perform this kind of study for medical interventions to determine if they are safe and effective. It would be unethical to force-feed a large group of people Twinkies every day to see if they develop diabetes. 

Systematic review/Meta-analysis. A systematic review attempts to examine all the relevant evidence published on a given topic, and a meta-analysis combines the data sets of smaller studies into a single, large data set. Because smaller studies often produce contradictory results, the point of reviews and meta-analyses is to arrive at a "consensus." Comprehensive reviews and meta-analyses may include data from dozens or even hundreds of papers.

Source: "Health Literacy: Evidence Pyramid." University of New Hampshire. Accessed: 4-Apr-2017.