“Nutritional epidemiology has long been criticized for producing sensational and conflicting findings, which has eroded confidence in the discipline. Nevertheless, nutritional epidemiology studies continue to play a crucial role in shaping dietary recommendations and policies, making it imperative to draw credible inferences from these studies.”
Those are among the closing thoughts of a recent study looking at a new methodology for drawing “credible inferences.” Dr. Stanley Young, a member of the ACSH Board of Scientific Advisors, has been a leader in decrying the use of p-hacking and, more specifically, the impact of an analysis of red meat. Nutritional studies using more credible randomized controlled trials are difficult to perform and often lack real-world relevance. As a result, observational studies are frequently the coin of the realm. However, as Dr. Young and many others have noted, there are often many ways to select and analyze a given dataset, “each of which may produce results that vary in direction, magnitude, and statistical significance.”
Because our understanding of something as complex as nutrition’s impact on our biology is limited, and there is a lack of consensus on the best analytic approach, there is significant flexibility in choosing analytic models and pertinent data. Observational studies are highly dependent on the researchers' choices. While this is not necessarily true in all cases, both sides of nutritional issues suggest that the investigators' beliefs and expectations influence their work. And even sensitivity analysis to separate the wheat from the chaff is subjective.
An inconsistency in findings, especially in nutritional studies, erodes trust. How might we reduce the noise and amplify the signal? The researchers offer specification curve analysis.
Specification curve analysis - multiverse analysis
It is a daunting name for a relatively simple idea. In specification curve analysis (SCA), you begin by identifying the plausible choices for the analysis. This includes the analytic statistical models, the criteria for selecting participants, the variables, and how they are described. You eliminate improbable combinations and are left with, as Dr. Young has pointed out, a large group of “analytic specifications.” [1] You then test the data set against all the combinatorial possibilities, generating a specification curve plot that visualizes the distribution of results for each combinatorial possibility and the magnitude and direction of their impact.
To demonstrate the SCA technique, the researchers considered an unsettled nutritional controversy,
“The effect of unprocessed red meat on all-cause mortality—a question that has yielded inconsistent results in the literature and produced conflicting dietary recommendations.”
Making use of a previously performed meta-analysis of “15 publications reporting on 24 cohort studies,” they identified numerous factors, including:
- Nutrition Model Types adjusting for total energy or considering nutrient density
- Characterization of Red Meat consumption as a continuous variable, i.e., 30 grams, or as categorizable, i.e., 30 – 60 grams. :
- Subgroups of Interest categorized by gender or age
- Core Covariates (included in all models): age, sex, smoking, total energy intake, year, menopausal status, hormone therapy, parity, and oral contraceptives. Secondary Covariates (adjusted in some models) include race/ethnicity, education, marital status, alcohol consumption, physical activity, BMI, socioeconomic status, comorbidities, and dietary variables.
When considering all these variations, it yields 10 trillion unique combinations, a number that is difficult to calculate even with today’s computing power. The researchers restricted their analysis to covariates that included all core variables and a random set of secondary covariates – 1440 unique, plausible specifications. The models used data from our old friend, the National Health and Nutrition Examination Survey (NHANES) from 2007-20014, linked to mortality data from the National Death Index and nutritional information from the Food Patterns Equivalents Database. After excluding missing data, there were 10,661 “participants.” NHANES uses dietary recalls at one point in time, so the conclusions about any dietary component based on this data are fuzzy at best. Still, this paper is a proof of concept to a SCA, not the definitive analysis of the role of red meat and our health.
SCA found that among those 1440 variable ways to analyze the data
- 36% showed a hazard ratio (HR) of greater than 1, meaning red meat increased all-cause mortality; 64% showed a HR of less than one, meaning red meat did not enhance all-cause mortality.
- Only 48 of those 1440 variations were statistically significant, “40 had indicated red meat to reduce all-cause mortality and eight indicated red meat to increase all-cause mortality.”
- 45% of the variations yielded a HR between 0.9 and 1.1, meaning that red meat had little impact, good or bad, on all-cause mortality.
While, as the researchers have cautioned, this is not the final word on red meat and our health, it certainly makes a strong case “that findings in nutritional epidemiology studies may be contingent on analytic methods.”
The presence of analytic flexibility, whether we posit that to ill-intentioned conflicts of interest or not, makes nutritional epidemiologic studies less certain. SCA is another tool to clarify our uncertainty, but in a fractal way, it too can be subjective. The researchers determined what methodologies and variables they considered justifiable. They also selected a specific dataset among many, and the data available within the dataset further limited the possible variations.
Specification Curve Analysis offers a promising way to cut through the noise. By rigorously testing all plausible analytical choices, SCA highlights just how contingent results can be on the chosen methods. While this approach doesn’t eliminate subjectivity or the potential for bias, it provides a more transparent and comprehensive view of the data. The study on red meat and all-cause mortality is a prime example, showing that our conclusions can significantly vary depending on the analytic path taken.
[1] As the researchers point out as an example, “three unique choices for five aspects of the analysis yield 243 unique analytic specifications.”
Source: Grilling the data: application of specification curve analysis to red meat and all-cause mortality Journal of Clinical Epidemiology DOI: 10.1016/j.jclinepi.2024.111278
Chuck Dinerstein, MD, MBA
Director of Medicine
Dr. Charles Dinerstein, M.D., MBA, FACS is Director of Medicine at the American Council on Science and Health. He has over 25 years of experience as a vascular surgeon.
