Point: Meta-analysis of randomized controlled trials shows it does work when given early.
Dr. Nigel Bark, Chairman American Council on Science and Health
This meta-analysis is reported in a preprint (meaning it "has not been certified by peer-review and should not be used to guide clinical practice") posted on-line on September 30th.
Clinical trials testing the effectiveness of therapies for COVID-19 have mainly focused on hospitalized patients where HCQ has not been found effective. This analysis aimed to see if HCQ in randomized controlled trials reduces hospitalization, mortality, or the development of COVID-19 illness in outpatients. The authors followed the guidelines of the Preferred Reporting Items for Systemic Reviews and Meta-Analysis (PRISMA). They searched the literature database, found 305 possible articles that were then manually searched, identifying five well-controlled trials. These trials, from the United States, Canada, and Spain, included 5,577 subjects. Three of the trials were administratively stopped early by the investigators (presumably because of statements from the FDA and WHO), which contributed to their lack of statistical significance. The relative risks and 95% confidence limits of the outcomes were calculated. The variation between studies was minimal (heterogeneity was zero).
The outcome was a statistically significant (p=.025) 24% reduced risk of COVID-19 infection, hospitalization, and death. The reduced risk was even more significant when only those in one study who started HCQ within two days of exposure were included. The average age of subjects was about 40. Such subjects have a lower risk anyway. But one Spanish study included 293 nursing home residents. In these, HCQ for post-exposure prophylaxis reduced the risk by half.
They also looked at side-effects. No serious cardiac adverse events were reported. No arrhythmias were reported in three of the four studies that assessed arrhythmia. A fourth study had one arrhythmia in 936 patients on HCQ and one in 469 controls. There were no serious cardiac effects, and no trial was stopped because of safety concerns. The most common side-effects were gastrointestinal.
As they note, the trend in all the trials was towards reduced risk. The point of meta-analyses is to combine studies so that there are enough subjects to show the statistical significance of the findings.
The authors mention six controlled but not randomized trials that showed similar results, and another six in hospitalized patients when HCQ was started within a day or two of admission. This early treatment effect is similar to that of Tamiflu in influenza, acyclovir in herpes encephalitis, and for prophylaxis, zanamivir for influenza, and HIV antivirals.
The paper states that HCQ impairs the endosomal transfer of virions within human cells and conveys zinc intracellularly to block the SARS-COV-2 RNA-dependent RNA polymerase, which is central to the virus's ability to replicate.
This is a good meta-analysis of good randomized trials written by experienced clinicians and epidemiologists from Yale, UCLA, Baylor University Medical Center, and Henry Ford Hospital Detroit. It is important.
Counterpoint: Meta-Analysis or Alchemy?
Dr. Charles Dinerstein, Director of Medicine at the American Council on Science and Health
Let me begin by pointing to our agreement that meta-analysis can combine properly selected studies to reach a statistical significance that the individual studies failed to achieve or where the significance was small. Therefore, much of this debate centers on the selection and interpretation of those five studies.
The most robust objective evidence that hydroxychloroquine has no statistical significance lies in this chart. In each instance, the confidence intervals for each study cross, and often by a large margin, 1, the value where there is no reduction in risk. As a clinician, my statistics training informs me that irrespective of where the math places the aggregate, there can be little confidence that the treatment, hydroxychloroquine, has any clinical value.
The first study [1] looked at 2300 direct contacts of PCR+ COVID-19 index cases treated with hydroxychloroquine as prophylaxis. Treatment was initiated 3 to 4 days after contact with the index case. The incidence of PCR+ cases in the control arm, 6.2%, was not statistically significant compared to that of the treatment arm, 5.7%. "Postexposure prophylaxis with HCQ did not prevent Covid-19 disease or SARS-CoV-2 infection in asymptomatic contacts exposed to a PCR-positive index case."
The second study [2] is by the same authors looked at the early treatment of PCR+ cases with hydroxychloroquine. 293 PCR+ patients were randomized within three days of their symptoms. Again, there was no benefit. "Our results indicate no impact on viral burden up to 7 days nor symptoms resolution or hospitalization rate up to 28 days following diagnosis."
The third study [3] looks at a similar patient group, 423 early cases of COVID-19 randomized to hydroxychloroquine or placebo – about half being started on prophylaxis within 24 hours of symptoms. There are two significant differences in the methodology of this study. First, only 58% of these patients had a confirming PCR test. Second, the outcome was changed from hospitalization and deaths to symptomatic relief when it became apparent to the investigators that the recruitment goals were inadequate to make a statistical conclusion about those graver outcomes. The trial was halted when a statistically significant endpoint was reached for the lesser outcome, symptomatic relief. As in the second study, the researchers demonstrated no statistically significant improvement in symptoms with hydroxychloroquine. This study is much different from the prior one because of the changing outcomes, and in 42% of cases, the presence or absence of COVID-19 was "inferred," not objectively identified – introducing bias and uncertainty.
In the fourth study [4], 1483 high-risk healthcare workers were treated with prophylactic hydroxychloroquine once or twice weekly – a dosage regimen much different than the three prior studies where medication was given daily for about a week. Like the third study, outcomes were based on either PCR+ cases or symptoms, introducing bias and uncertainty. The study was stopped when the researchers determined they would not meet their recruitment goals. The participants were at higher risk, but in 80% of cases, they wore appropriate medical grade PPE. Is this really a population comparable to the first three groups?
The fifth study [5] involving the same research group turned their attention to individuals exposed for 10 minutes or longer, at home or work, to a PCR+ or symptomatic individual and subsequently treated within four days with a daily regimen of hydroxychloroquine or placebo. The lack of laboratory testing again introduced bias and uncertainty as subjective symptoms were equated with objective findings. As in the third and fourth studies, this research was terminated because they could not recruit enough participants to have a measurable statistical sample for their outcomes of interest. Two-thirds of the participants, exposed at work, were the same high-risk healthcare workers described in the fourth study. In only 10% of the outcomes related as COVID-19 positive was there a confirming PCR test; nearly all outcome positivity was inferred.
The studies chosen for the meta-analysis were to involve low-risk ambulatory patients and were designed to identify moderate-risk clinical events, e.g., infection, not hospitalization, or deaths – outcomes that the meta-analysis considered when available. Do you believe the studies meet that criteria? Which are the apples, and which, the oranges?
- Can we remove the fourth study? It made use of a completely different treatment regimen?
- What about the fifth study? Do 66% of the participants being high-risk healthcare workers qualify as low-risk ambulatory patients?
- Should we keep the third study, where 42% had presumptive outcomes?
- Should we retain those last three studies, which were stopped because they could not provide any statistical significance for hospitalization and deaths?
When you remove these three studies, you are left with just the first two – with similar treatments initiated within three to four days of contact or symptoms. They found no statistical difference. At best, all we can conclude is that hydroxychloroquine may be "Tamiflu light." But to actually show that this is the case, you would need to run a large study and treat everyone within 1-2 days of contact or symptoms. If you believe hydroxychloroquine has a role, do that study; then, we will have something meaningful to discuss.
Rebuttal to Dr. Dinerstein
The great value and strength of meta-analysis are combining statistically underpowered studies to reach a more statistically precise estimate of the effect. It may be counter-intuitive as a clinician, but it is the standard statistical way to obtain clinically meaningful information from a group of underpowered studies.
The conclusion of each of the studies was that HCQ did not prevent illness, hospitalization, or death because the result was not significant. But in each of them, those on HCQ did better than the controls, but not significantly. Combining them showed HCQ was better.
The following are responses to the comments on specific studies.
Study 2 - the “bias” from including those without positive COVID testing is against a positive effect of HCQ because those patients may have other conditions. Their symptoms were evaluated by an infectious disease specialist to ensure their compatibility with COVID-19. Only 42% had a positive COVID test, but another 39% had been in high-risk contact with someone who did have a positive test.
Study 4 - these were health care workers, as were 63% of those in study 5 by the same group. Still, the criteria were different: healthcare workers working with possible or definite COVID-19 instead of people knowingly exposed, without a mask, to someone with COVID-19 in study 5. The recruitment period was about the same as study 5, but the treatment regime was quite different. So, the subjects were not the same in the two studies.
Study 5 - while most participants were not tested, the diagnosis depended on the consensus of four infectious disease (ID) specialists reviewing the symptoms. This would reduce the chance of a positive result: a bias against a positive effect because non-COVID-19 patients could be included.
You could say this was a study of fruit rather than apples or oranges. There were different populations and different treatment regimes. Still, they were all outpatients, all randomly assigned to HCQ treatment, and followed to see if they got COVID-19 diagnosed by various methods (because testing was so unavailable) or were hospitalized or died. When death is a rare result, it is essential to use other end-points. So, all the studies are appropriately included.
The aim was to study outpatients, not necessarily low risk. (Hopefully, healthcare workers treating COVID patients with appropriate personal protective equipment are not high risk.)
The fact that studies were ended prematurely is an additional reason to include them because that early ending prevented possible statistical significance. The reason for not completing recruitment was the combination of negative studies in hospitals, false reports of serious cardiac side-effects, and the resulting pronouncements of the FDA and WHO. Who will volunteer for a study of HCQ after hearing those things?
I continue to believe that this is a good meta-analysis of well-controlled, randomized trials, several placebo controlled, which shows that HCQ is reasonably effective, with a statistically significant 25% reduction in illness or hospitalization or death.
Rebuttal To Dr. Bark
I do not doubt the value of meta-analysis in combining studies to strengthen the aggregate's statistical significance. Our disagreement remains over whether these studies should be combined. Dr. Bark notes the dissimilarities in the studies that, in my mind, reduce the value in the comparisons. I would also argue that the differences in treatment regimens, ranging from daily to once a week, further reduce the ability to combine these studies.
I mention the reported cause of the studies' termination to dispel any belief that an anti-hydroxychloroquine cabal of scientists was involved in that decision. Dr. Bark may indeed be correct in his assessment of why recruiting failed, but alternative explanations can be made. In any case, the underlying cause of recruitment failure has no impact on our disagreement.
There remains one additional significant problem with the meta-analysis; the statistical sleight of hand involved in combining all three endpoints, infection, hospitalization, and death. These are three related but quite different outcomes – hydroxychloroquine is a preventative measure in the case of infection and, in the remaining two, a mitigating factor. Combining the three, especially without indicating their individual impacts, is opaque science. From a clinician's viewpoint, I still have no evidence that hydroxychloroquine effectively interrupts COVID-19's chain of transmission.
Dr. Bark's rebuttal, notwithstanding, I maintain my earlier conclusion. If you believe hydroxychloroquine has a role, do that study; then, we will have something meaningful to discuss.
A Final Thought
On behalf of the American Council on Science and Health, I would like to thank Dr. Bark for spending the time to discuss and fashion his argument over this study. I would like to suggest that this is the type of discussion we should aspire to, free of ad hominem attacks, based on facts, and where individuals can differ in their conclusions without requiring political labels.
[1] A Cluster-Randomized Trial of Hydroxychloroquine as Prevention of Covid-19 Transmission and Disease
[2] Hydroxychloroquine for Early Treatment of Adults with Mild Covid-19: A Randomized- Controlled Trial
[3] Hydroxychloroquine in Non-hospitalized Adults With Early COVID-19
[5] A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19
Source: Randomized Controlled Trials of Early Ambulatory Hydroxychloroquine in the Prevention of COVID-19 Infection, Hospitalization, and Death: Meta-Analysis medRxiv DOI: 10.1101/2020.09.30.20204693
