This week, Drs. Makary and Prasad laid out the Priorities for a New FDA in JAMA Network Open. I guess they did not get the memo from Secretary Kennedy, who stated on a podcast,
“We’re probably going to stop publishing in the Lancet, New England Journal of Medicine, JAMA, and those other journals because they’re all corrupt.”
Given that their article was an opinion piece, I suspect that it did not undergo pesky peer review. As with many of the current administration documents, while containing kernels of truth, how they are applied is, at least for me, troublesome. As a public service, let me peer review two of their points: “Unleashing AI” and “Harnessing Big Data.”
Unleashing AI
“The advent of generative artificial intelligence (AI) holds several promises to modernize the FDA and radically increase efficiency in the review process. From day one, we have made it a top priority, and on May 8, 2025, the agency implemented the first AI-assisted scientific review pilot using the latest generative AI technology.”
– Drs. Makary and Prasad
The doctors state in a separate YouTube broadcast that the scientific reviewer loved the AI assistance and that it reduced one aspect of the review process from “3 days to six minutes” – a notable time efficiency. Unfortunately, while a reduction in review time is helpful, the veracity of safety and efficacy information in the FDA’s scientific review remains paramount. There is no information regarding the accuracy of AI’s work for the FDA unless, of course, we consider the MAHA Commission report with its “formatting errors” in citing fictitious studies.
While a move towards AI is to be applauded, some questions remain. As STAT asks, what is being automated in scientific reviews, what safeguards are in place to ensure accuracy, and how are FDA reviewers being trained to use their new tool? This new and perhaps overdue disruption of the FDA’s scientific review process belies a bias towards action and then consideration, or as one of the two co-authors wrote,
“We're subtly taught a bias toward treatment rather than restraint.”
― Marty Makary, Unaccountable: What Hospitals Won't Tell You and How Transparency Can Revolutionize Health Care [1]
Harnessing Big Data
“The mass availability of health data and cloud computing has enabled 2 new opportunities: (1) researching root causes of chronic diseases and (2) post-approval monitoring of new products. …
Advances in causal inference in nonrandomized data, including the use of target trials, which attempt to balance confounding and time zero, have potential to yield actionable causal conclusions, in many cases at lower cost. Moreover, post-approval monitoring in Big Data will allow the FDA and researchers to see safety signals in real-time and evaluate effectiveness in the real world.”
– Drs. Makary and Prasad
Again, there is a kernel of truth here. For those advances in causal inference, the Drs. Makary and Prasad offer a citation to support their claim. As with any journal article I review, I took the time to read a bit more from their citation, published in the supposedly corrupt journal JAMA.
The study noted that real-world evidence (RWE) studies are a valuable complement to randomized controlled trials (RCTs) in informing regulatory decisions. However, their reliability hinges significantly on the ability to closely emulate the design and measurements of an RCT. They emulated 32 “highly selected” RCTs using administrative claims data, the coding information used for billing, and the same type of information that Medicare Advantage programs have been found to have “upcoded” for profitability. The term “highly selected” refers to the fact that for the emulations to be done, administrative claims data had to include measurable data on treatment, comparators, outcomes, and key inclusion and exclusion criteria.
Emulations had, in the researchers' words, an overall “modest” 72% statistical agreement with their highly matched RCT, rising to 94% when the match was stronger and 50% when the match was weaker. The researchers identify several difficulties and limitations associated with emulation. Among these, treatment adherence, patient characteristics, the use of loading doses, and titration patterns all differ in real-world settings versus clinical trials. This may be an advantage because of the common finding that real-world results are attenuated compared to those obtained during a clinical trial. However, controlling patient characteristics, a shorter study interval (often due to lack of adherence), and a lack of a true placebo arm greatly reduce the value of emulations.
The researchers end on this note,
“In conclusion, we observed similar findings between highly selected, nonrepresentative RCTs and nonrandomized database studies. In the absence of RCT evidence, database studies can complement RCT evidence to enhance understanding of how medications work in clinical practice.”
While Drs. Makary and Prasad tout emulations’ “potential to yield actionable causal conclusions,” their citation is more constrained. And, of course, there is this,
“Folks who think COVID-19 vaccines should continue to roll out without randomized trials are anti-vaccine, anti-science, and pro-corporate.”
“The FDA is a failure. It rubber stamps too many useless products. It needs to either remove itself from the picture, or demand randomized trials measuring appropriate endpoints.”
Post-marketing surveillance, or Phase IV safety detection, will also be a beneficiary under Drs. Makary and Prasad’s New FDA.
“Moreover, post approval monitoring in Big Data will allow the FDA and researchers to see safety signals in real time and evaluate effectiveness in the real world.”
– Drs. Makary and Prasad
Oh, but if this were true. Search as I might, I could not find the source of that Big Data.
“I don’t think VAERS [the Vaccines Adverse Event Reporting System] is a good database because it is self-reported ...” – Dr. Marty Makary
Again, a kernel of truth, self-reports are not overly helpful, and the CDC itself finds that a vanishingly small number of reports are made to VAERS. However, Secretary Kennedy’s suggestion to “implement an automated system as well as seek to create data sharing arrangements globally on vaccine use and health” comes with its own problems.
Electronic health records (EHRs), our repository of clinical information, are prone to errors. One study of the accuracy of cancer data found that information gathered in an outpatient location was 98% less likely to be accurate and 458% more likely to be “suboptimal” compared to data collected on the “main campus.”
In addition to errors within the dataset itself, further processing, the “automated” systems that Secretary Kennedy touts come with their own programming errors. Finally, a nationwide system of safety surveillance would require an interchangeability between EHRs. After an expenditure of $39 billion, less than half of US hospitals “engaged in all domains of interoperable exchange (send, find, receive, and integrate)” on a routine basis. Now, after 15 years of implementation and interoperability, a physician states, “It takes gymnastics to get an Epic system [the market leader by a lot] to talk to a non-Epic system.”
Let’s not unleash AI or Big Data
Makary and Prasad’s vision of a new, data-savvy, AI-enhanced FDA is peppered with familiar buzzwords and flickers of truth—but it is ultimately hamstrung by contradictions, unvetted assumptions, and the convenient omission of practical hurdles. If this is the peer-review-free future of regulatory science, we might want to keep our skepticism sharper than ever.
[1] You can see more of Dr. Makary’s enthusiasm to “just do it” when it comes to AI in that YouTube video at roughly 4:50.
Chuck Dinerstein, MD, MBA
Director of Medicine
Dr. Charles Dinerstein, M.D., MBA, FACS is Director of Medicine at the American Council on Science and Health. He has over 25 years of experience as a vascular surgeon.
