The National Association of Scholars (NAS) project Shifting Sands: Unsound Science and Unsafe Regulation examines how irreproducible science negatively affects select areas of government policy and regulation governed by different federal agencies.
The academic scientific professionals are rewarded for exciting research with new positive (significant association) claims—but not reproducible research. This encourages researchers, wittingly or negligently, to use different flawed statistical practices to produce optimistic but likely false claims. Our report applies Multiple Testing and Multiple Modeling (MTMM) to assess whether such faulty practices have affected a body of research.
We found persuasive circumstantial evidence that the scientific literature, in general, and statistical practices of nutritional epidemiology regarding red and processed meats’ adverse health outcomes and the cardiovascular disease risk reduction of soy are untrustworthy. These flawed statistical practices center around using the semi-quantitative Food Frequency Questionnaire (FFQ) – a self-administered dietary assessment instrument.
We offer 12 recommendations that are intended to bring FDA methodologies up to the level of the best available science, as per the mandate of The Information Quality Act (sometimes called The Data Quality Act):
While we believe the FDA should consider our methods in evaluating food health claims, we care even more about reforming the procedures the FDA uses to assess nutritional science.
The government should use the very best science—whatever the regulatory consequences. Scientists should use the very best research procedures—whatever result they find. Those principles are the twin keynotes of this report. The very best science and research procedures involve building evidence on the solid rock of transparent, reproducible, and reproduced scientific inquiry, not on shifting sands.
You can find the full executive summary here
The entire white paper can be downloaded here
You can find a video of the Shifting Sands conference on the National Association of Scholars website.
Dr. Young is an adjunct professor of statistics at North Carolina State University, the University of Waterloo and the University of British Columbia where he co-directs thesis work. He is also a member of the ACSH Board of Scientific Advisors.
