Research published in Science Advances first examined reproducibility in psychology, economics, and articles in Nature/Science. Many would argue that psychology is too soft a science, and economics is too dismal (?), but can we agree that the last category should routinely report reproducible research? I will focus on those findings when possible. Papers in Nature and Science were the most frequently reproducible; 62% of 21 studies, although the effect size was reduced by about 25%, the results were still in the “right direction.”
To determine the fate of those papers, the current researchers looked at how often and where these papers were cited – how often were these studies forming the foundation of thought for other exploration? Were they built on solid or shaky ground?
Reproducibility was defined as a p-value of less than 0.05. As the chart shows, research that was found to be not reproducible was cited quite a bit more than work reproduced. How much more? Double. As the researcher suggests, that is quite a “citation gap” for work that is inaccurate.
The next logical question how widely was this “fake” science disseminated and by whom. Papers subsequently citing the non-reproducible were no more likely than papers citing reproducible research; nor was the quality [1] of the journal better or worse. Misinformation was an equal opportunity problem.
In the eight Nature/Science papers that were not reproducible and generated 798 additional citations, only 15% acknowledge that the outcomes reported in their discussions were, in fact, not reproducible. I would have to conclude that some of the research in these “harder” sciences had shaky foundations.
Why?
There is no apparent reason, and I do not believe that most cases are due to scientific misconduct. I will leave aspersions of intent to those who seem to have special clairvoyance. The authors note that retracted papers are cited less frequently, suggesting that irreproducibility is not as widely disseminated – that researchers are not fully up-to-date on what is or is not still considered valid. That happens; attention to detail is more difficult than it seems. We could be more supportive in this area by maintaining a registry of non-reproducible studies, but we don’t even do this for retracted articles unless you count Retraction Watch as a registry.
The careful reader will note that the study period ran through 2019, before the tsunami of publications on COVID-19. It will take quite a long time to sort out the good from the bad in that category. If the current researchers are correct, there is a lot of misinformation that will be cited and used as the foundation for “further research.” If the experts find it challenging to separate the wheat from the chaff, we should not be so distrustful as wary, even when the science “finds” what we want to believe is true.
Just for giggles: what paper, identified by Retraction Watch, as being retracted has the most citations? No, not the Wakefield autism linkage that came in second. Number one was a paper from the New England Journal of Medicine showing that adherence to a Mediterranean Diet reduces cardiac “events.”
[1] Quality was based upon a journal’s impact factor, which measures how often articles published by the journal are cited.
Source: Nonreplicable publications are cited more than replicable ones Science Advances DOI: 10.1126/sciadv.abd1705
Chuck Dinerstein, MD, MBA
Director of Medicine
Dr. Charles Dinerstein, M.D., MBA, FACS is Director of Medicine at the American Council on Science and Health. He has over 25 years of experience as a vascular surgeon.
