Have mRNA vaccines killed more people than they have saved?
That’s what American entrepreneur Steve Kirsch claims in his list of Covid-19 challenges. Today, Rootclaim has officially accepted his challenge in the amount of $500,000.
After reviewing all challenges we decided to accept challenge no. 6: “The Pfizer and Moderna mRNA vaccines have killed more people than they have saved from dying from COVID“. This addresses two of the most pressing and hotly debated issues of the pandemic: vaccine efficacy and vaccine safety. Advancing public discourse on these issues will likely save lives, and improve preparedness for future pandemics.
After analyzing the available evidence, we conclude that despite several shortcomings, mRNA vaccines have saved many more lives than they cost.
While we challenge Kirsch on this specific item, we actually agree with a number of his other claims, including some that run counter to mainstream opinion. As Kirsch pointed out in his post, we agree with item 9 (“Lab origin is more likely”) and even offer our own challenge on the subject. Before examining vaccines, we studied the benefit of masks (items 7 and 10 in Kirsch’s list) and were surprised to find it is far from clear they are indeed effective, given the many factors involved in their practical use, such as most people wearing them poorly, virus transmission through the eyes, virus adaptation, and considerations of herd immunity. We are also generally in agreement on the importance of drug repurposing in COVID (related to challenge no. 8).
We have great admiration for Kirsch’s willingness to take a personal risk on his public claims. This is in sharp contrast to the many public figures constantly making overconfident statements on matters of great importance, without taking any risk. This is something we repeatedly encounter in our work. Some examples:
Scientists who confidently claim a zoonotic origin of Covid-19, while our analysis found it much more likely that the virus was developed during gain-of-function research and released by accident.
The common claim of widespread fraud in the 2020 US election, while we found this election to be no different from previous ones, with minor incidents of fraud that did not change the outcome.
These examples demonstrate the low value of claims made when nothing is at risk: public discourse is awash with baseless, overconfident claims that carry no repercussions for their claimants if they turn out to be false. We believe that adding ‘skin in the game’ can dramatically reduce this problem, and therefore offer our own public debate challenge, which coincidentally happened to be very similar to Kirsch’s. So far no one has applied.
We therefore greatly appreciate Kirsch’s courage and leadership here. We see it as our responsibility to accept a challenge when we think the claim is wrong, and of course, take the loss if we fail.
It should be emphasized that regardless of who wins in this particular case, this is a victory for public discourse. First, by offering a reliable resolution to the important question of vaccine efficacy and safety, and more importantly, by setting a standard for settling controversies: an impartial, judged debate where both sides take a significant risk on the outcome. Hopefully, in the future, people making confident assertions on issues of importance without taking a risk will be ignored as background noise.
Update: As we were applying, we noticed Kirsch has recently added a note to his challenge page, terminating the bets due to no one applying. Since we were already in private discussions with Kirsch on the terms before this update, we would be very surprised to find this would apply to us.
Update #2: We and Kirsch are making good progress on setting the parameters of the $500,000 challenge and we’re in the process of finalizing our agreed picks for two judges. Our preference will be for the most experienced, well-respected, and unbiased experts.
As the pandemic seemingly becomes endemic, two new, purpose-built drugs have just been FDA-approved. Pharmaceutical companies have been busy perfecting these new treatments, paxlovid and molnupiravir, almost since the pandemic began. The approval and roll-out of vaccines similarly took the better part of a year, on an expedited process no less. But the COVID-19 vaccines were an outlier, usually developing these purpose-built treatments takes years and can cost billions.
But waiting two years is too late. In a pandemic, a rapid response is paramount. Waiting for an effective, low-risk, purpose-built treatment that may only come in years will lead to innumerable deaths in the meantime, as happened during COVID-19.
So, under the constraints of a pandemic, the best way to save lives is to save time. Repurposing mobilizes already existing drugs with a known safety profile into the fight against the pandemic. Limiting ourselves to new drugs when a wealth of potential candidates already exists is extremely harmful. But repurposing does more than just provide a more efficient use of resources: deploying multiple drugs simultaneously limits opportunities for the virus to adapt as opposed to introducing each drug gradually.
The immediate question, then, is how to identify promising candidates and how to get them widely adopted. There are currently around 20 candidates for repurposing to combat COVID-19. Working out which are the most effective, and recommending them to the public, could end the COVID-19 pandemic.
Those who have read our ivermectin blog post will be familiar with the major stumbling block: no one in industry currently has the incentive to conduct repurposing trials and no one in government has the right combination of access to data and decision-making capacity to pursue a drug repurposing policy.
Despite the lack of incentives, some researchers did manage to conduct trials into drug repurposing for COVID-19. They had to run on shoe-string budgets, resulting in imperfect, flawed results. However, the resulting data is not useless, and careful analysis of all trial data can signal which repurposed drugs are likely enough to work.
Of these, we find four to be the most promising as they have multiple independent sources signalling efficacy. This reduces the possibility of being wrong by only relying on one source, which may turn out to be false, either as a result of fraud, study flaws, or confounding factors. We estimate each treatment has over an 80% likelihood of being very effective, and they all carry near-zero risks. A protocol combining all of them is near certain to be highly effective, and may even end the pandemic.
Examples of signals of efficacy are: randomized controlled trials (RCT), a plausible mechanism of action, and a correlation between the use of the drug and COVID-19 severity. Especially interesting is a study on electronic health records (EHR) from Clalit health services (the largest health provider in Israel), which examined which purchases made in pharmacies were later associated with reduced severity. Only 41 drugs were found to be significantly protective, out of over 1000 examined, including all treatments below.
Plausible mechanism of action:
Fluvoxamine is a selective serotonin reuptake inhibitor (SSRI) antidepressant, hypothesized to have multiple mechanisms of action including an anti-inflammatory effect as an agonist for the sigma-1 receptor, reducing platelet aggregation, decreasing mast cell degranulation, and more.
In the Clalit study, patients purchasing escitalopram (another SSRI) were 17% less likely to later be hospitalized.
Randomized control trials
An RCT found fluvoxamine to reduce hospital admissions by 35% and possibly cut deaths dramatically (1 versus 12 among those adhering to protocol). Unlike other repurposing trials, this was a large-scale high-quality study, which received unanimous support from experts.
As fluvoxamine was already considered a promising candidate prior to this trial, this success validates the whole model, indicating there are many more effective treatments that are currently overlooked.
Plausible mechanism of action:
Vitamin D is an essential hormone of the immune system, and our modern indoor lifestyle causes many to be deficient in it. It is reasonable to assume that these deficiencies result in a suboptimal immune response to COVID-19.
Additionally, it can promote autophagy, the process of replacing older cells with newer ones which can help fight infections better.
Prior to COVID, a meta-analysis of 25 RCTs found vitamin D supplementation to be protective against respiratory tract infections, especially among those deficient in it.
In the Clalit study, patients buying vitamin D were 9-11% less likely to later be hospitalized.
Randomized control trials
An RCT of 76 patients in Spain found that calcifediol (a more active form of vitamin D) dramatically reduced ICU admission and mortality. Of the 50 patients treated with calcifediol, 1 required an ICU compared with 13 of the 26 patients in the control group (a risk ratio of x25!). The trial had significant flaws but they are likely not enough to explain such a strong result.
Oral, ocular, and nasal disinfection
Plausible mechanism of action
Virus replication is believed to start in these locations, and since they are regularly exposed to the outside world, they are able to survive disinfectants that would be too damaging in internal use. Regular disinfection of these areas is hypothesized to neutralize the virus before it expands into other organs and becomes uncontrollable.
Impressively, 7 out of the 41 drugs identified in the Clalit study are related to eye hygiene.
Randomized control trials
10 RCTs have so far been published testing the disinfectant povidone-iodine. Assessed together, they show an overall improvement in symptoms at all stages of infection but are most effective when administered early (81% improvement).
One RCT found that using iota-carrageenan nasal spray as prophylaxis reduced symptomatic cases by 80%.
We propose the establishment of an independent commission composed of medical and scientific experts with the remit to:
Collect all research on repurposed drugs and contact researchers for clarifications.
Determine the weaknesses and reliability of each study.
Estimate the probability of efficacy of various repurposed treatments, considering different indicators of efficacy such as their mechanism of action, observational studies, and controlled trials.
Assess the risks and interactions of each, relying on decades of real-world experience.
Build protocols of repurposed treatments that best balance risks and benefits. Different protocols should be built, based on patient profiles and disease stages.
These recommendations should form the backbone for a two-front lobbying campaign to convince decision-makers and encourage public adoption.
Lobby for adoption by public health authorities.
Lobby decision-makers and politicians to support the initiative publicly.
For public and doctors:
Familiarize doctors and nurses with the late-stage disease protocol.
Familiarize GPs and family doctors with the early stage and preventative protocols.
Public information campaigns to promote the initiative and reduce scepticism.
At surface level, our proposal seems fairly trivial: use the drugs we already have at our disposal, instead of focusing on developing new ones months or years down the line. Unfortunately, and with catastrophic consequences, implementing this obvious option is extremely difficult within the current incentive structure. But make no mistake – this is the most promising path out of the pandemic.
Do you think ivermectin cures COVID-19? Think carefully: in the age of culture wars, your answer defines your politics. If you think ivermectin is useful, you are an anti-vaxxer, Trump-supporting libertarian; If you think it isn’t, you are a godless, big-government cosmopolitan. Sorry, we don’t make the rules.
Fine, but still, who is right? The truth is it’s still too early to say. The sorts of bold claims both sides are making are simply premature. The results of the TOGETHER Trial and the flaws recently detected in other trials weaken ivermectin’s case. On the other hand, one trial, that seems to have been well-conducted, points to it being mildly effective if taken at the onset of infection. Overall judgement? it may be useless, or it may be effective under certain circumstances.
But fixating on who is right misses the forest for the trees. Each side blames the other for Covid-19 deaths because of their supposedly wrong answer to this question. But, as it turns out, both sides are promoting deadly policies.
The damage ivermectin’s die-hard supporters cause is plain to see. Dubbing it a “miracle cure” provides false confidence to the masses to disregard other effective measures. The damage caused by the other side is more insidious, but likely just as lethal. Let us explain why: when dealing with safe, cheap drugs even a moderate probability that they work is reason enough to recommend them. Sure, you may later discover some of the treatments weren’t effective, as may have happened with ivermectin, but that’s negligible compared to the damage of waiting and then discovering some treatments were effective all along.
When experts focus on a study’s flaws, discarding any merit it may otherwise have, what the public hears is: ignore this, vaccines are your only possible hope.
The best demonstration of this is in the FLCCC treatment protocol, which recommends a range of low-risk, potentially-useful repurposed drugs. While the TOGETHER trial showed that their ivermectin recommendation may have been ineffective, it also showed that Fluvoxamine, another drug in their protocol, reduced hospitalizations by 35%, and possibly cuts deaths dramatically (1 death vs 12 among those adhering to the protocol). Similarly, the FLCCC protocol recommended corticosteroids months before those became the standard of care. It is fair to assume that in the future more treatments in their protocol will be discovered to be effective.
So, regardless of the harmful, overconfident statements made by the FLCCC, doctors following their protocol saved lives, and the conservative, supposedly responsible, approach of their detractors has caused millions of deaths.
So what went wrong?
In a word: incentives. The market is not incentivized to conduct the kind of large, costly clinical trials needed for high confidence results, when the treatment is a repurposed drug that costs a few dollars per patient.
When the free market fails, the government should step in; after all, it has a massive public health incentive to find cheap, safe, and effective treatments. The trouble is that there is no single person within the government who has the authority, knowledge, and incentive to make the sorts of cost-benefit or probabilistic analyses needed in a pandemic. Authority-wielding politicians are not experts in health, statistics, or probability, and the few knowledge-wielding bureaucrats with the necessary expertise are mostly incentivized to not make mistakes, and don’t really care if one type of mistake exacts a far higher cost on society than the other. No one is at the helm.
Recommending a low-risk treatment that turns out to be ineffective wastes a modest amount of resources. Delaying a treatment that turns out to be effective may easily kill on a mass scale. But this gross asymmetry in results is not manifested in the decision-maker’s incentives.
So, they inevitably take the safe route of recommending we wait for more information and better quality trials. No one is ever blamed for such a “levelheaded” decision. But in a pandemic, there are no neutral options for decision makers.
This happens time and again. Our analysis of vitamin D is that it is likely highly effective at combating COVID-19. Given its low risk, it should be a top priority for health authorities to ensure no patient is left fighting the disease while being deficient in this essential hormone of the immune system.
The approach taken by both sides is lethal. Both need to change.
To ivermectin’s supporters and the FLCCC: we advise you to stop making overconfident claims on specific drugs. Bold claims such as these may cause people to ignore other effective treatments and they stake all the credibility of a balanced, probabilistic approach on one drug. If that drug turns out to be ineffective, the whole project is undermined. Instead, focus on the strong benefit of a protocol that uses multiple, promising low-risk drugs.
To the skeptics: finding flaws in clinical trials is important and necessary work. But it’s more important to help fix the system so that conducting high quality trials on cheap, repurposed treatments becomes highly profitable. And in the meantime, remember not every flaw means a trial is useless. Help identify promising signals within imperfect trials that indicate the probable efficacy of low-risk treatments, and push for their immediate adoption, so the Fluvoxamine disaster is not repeated.
The ever-present culture war around who is right and the rigid approach maintained by each side is killing people. Please stop.
Since its founding, Rootclaim has tried to bring clarity to areas of uncertainty surrounding world events. Today we are one step closer to that goal, with new discoveries that effectively resolve the major controversy of who was behind the 2013 chemical attack near Damascus.
Responsibility for the 2013 chemical attack has been a hotly contested, politically divisive issue, with a wide consensus in the West that the Syrian government was to blame, while Syria and its allies claimed that it was a “false flag” opposition attack, intended to bring about US intervention.
Rootclaim’s 2017 analysis went against this Western consensus, calculating an 87% likelihood that the Syrian opposition carried out the attack. Following the discoveries discussed below, this has now been updated to 96%, one of our most certain conclusions. Read our updated probabilistic analysis here, including a summary of the main claims of each side.
The new findings are a result of what we believe to be the most impressive independent open-source investigation in history. It was initiated nearly a year ago by several volunteers who reviewed all the evidence from the attack and managed to uncover incontrovertible evidence implicating an opposition faction, confirming Rootclaim’s conclusion. The full report is available here, and following is a summary of its findings.
The investigation began by examining the many videos of rocket impact sites that were uploaded following the attack. Each video was examined for clues pointing to its exact location and the trajectory of the incoming rocket.
For example, in this video, the chemical rocket penetrated a wall on a roof and continued to the floor below. Several landmarks in the background can be matched to satellite photos, identifying the exact location (33.519130°, 36.354841°).
Stitching together a few shots from the video shows that the rocket first hit the far wall and then the floor below. Connecting the two impacts provides an estimated trajectory for the rocket, with a launch location to the northwest.
This location is especially interesting as it singlehandedly invalidates the current hypothesis for government involvement in the attack.
Originally, the common claim was that the attack originated in a Syrian army base. But when the rockets were discovered to have a short range of around 2 km, this claim had to be retracted, as no bases were within that range.
However, this entire area lies east of the blue line we added to the diagram, which shows it could not have been the source, as a rocket shot from there would have penetrated the northern wall of the building rather than the western wall, as seen in the video.
This next video was also reexamined. Its location was identified in 2013 (33.520415°, 36.356117°), and the rocket’s trajectory is clearly evident, since it lodged in the ground without bending, pointing to its source.
In two shots from the video the camera is almost directly behind the rocket, and it is seen to align with a tree and buildings in the background.
Connecting these features in satellite imagery provides the trajectory’s azimuth (towards the yellow building).
Interestingly, the UN misreported this angle by 30(!) degrees (towards the purple building below).
The UN also misreported another trajectory, both of which conveniently intersected at a Syrian army base (which we now know is 5 times beyond the rockets’ range). Subsequently, the New York Times printed these mistaken findings on their front page as “forensic” evidence for Syrian army culpability.
This hijacking of international bodies by political and financial interests is becoming a major world threat, hurting the lives of millions. Additional examples exposed in other Rootclaim analyses include the failure of health organizations to realize the efficacy of vitamin D (and other unpatented treatments) in treating Covid-19, and when the scientific community and the WHO suppressed evidence supporting the hypothesis that Covid-19 resulted from a lab leak (see our analysis).
The open-source investigation repeated the process above for seven impact sites, producing this map of all trajectories (triangles represent uncertainty of a trajectory).
The agreement between the trajectories is remarkable, with all of them converging on a small area that also happens to be at the expected ~2 km range from the impact sites.
It is widely recognized that this location was under opposition control at the time (the significance of this spot was not known until now, so both political sides had no problem agreeing who controls it…).
Right in the middle of the identified area is this small field with enough space from which to launch rockets, whose importance will soon become evident.
Video of the chemical attack
A month after the attack, when the US threat to attack Syria had already been removed, a video surfaced, which was claimed to have been found on the bodies of “Syrian terrorists”. The video shows Islamist fighters in gas masks launching the exact same rockets, identifying themselves as Liwa al-Islam (the dominant opposition faction in the area), and announcing the date as August 21st 2013 (the day of the attack).
The existence of video evidence of opposition fighters carrying out the chemical attack is a remarkable story all by itself. What would normally be considered the highest level of evidence, was here dismissed out-of-hand as fake and wasn’t even mentioned in mainstream media, while overconfident unfounded accusations by the US government and false evidence reported by the UN made headlines.
The Rootclaim method prevents this bias by requiring a thorough investigation of all evidence, without filtering. We carefully examined those videos years ago, and also researched video fabrications in general. We found staged videos to be very rare, and that this video has multiple features that are highly uncharacteristic of a fabrication. This finding was a major factor in our initial conclusion.
Thanks to this new investigation, we now have a much deeper understanding of these videos.
The videos are fairly dark with little detail, but a frame-by-frame examination managed to uncover many features of the launch spot, and they perfectly match that same field where all the rocket trajectories intersect.
For example, in several frames the rocket illuminates the area, exposing details such as trees in the background, a field with low vegetation, and a paved platform where the cameraman stands.
In another shot we see a ditch or edge, while other shots show a few scattered trees and brush.
Other such shots in the video provided further features, which were all modeled in 3D, creating a unified view of the area:
This is a perfect match to our field:
We have a video showing opposition fighters with gas masks launching the rockets used in the chemical attack on the night of the attack. This video strongly matches the characteristics of a small field that lies right at the intersection of seven trajectories calculated from the impact sites, within rocket range of all of them.
Continuing to support the government attack hypothesis in light of this new evidence would require constructing a very unusual scenario. Nevertheless, given the political interests surrounding this issue, we will likely witness such attempts soon.
This breakthrough demonstrates the superiority of Rootclaim’s method, which was able to reach this conclusion years ago, without using the new findings, and with much less information and resources than the Western intelligence agencies who confidently claimed the opposite. That is the strength of probabilistic inference: its ability to extract better insights from less evidence.
Of course, many others also took this position, and have now been proven right, but their position was often politically influenced, causing them to reach the wrong conclusion in cases where the West’s claims happened to be true, such as the downing of flight MH17 over Ukraine (we at Rootclaim concluded that it was downed by the pro-Russian DNR, agreeing with the prevailing narrative in the West).
It is very rare to be consistently correct on contentious issues, when each time the truth supports a different political side. We believe Rootclaim is unique in its consistent success in that aspect.
To summarize the key takeaways from these new discoveries:
Having superior inference methods is far more important than gathering more evidence.
Sometimes, a “smoking gun” is already available, and there is no need to collect more evidence (the information here was all available in 2013). This is especially true for videos and photos, which are so rich in information that there is nearly always another discovery to be made.
The current crisis regarding the public’s trust in authorities and experts is not just about ‘fake news’. Experts are repeatedly failing to serve public interests, due to failures of human reasoning and heavy politicization.
Our society needs to quickly improve its inference methods and especially how our intelligence agencies, courts, international bodies, NGOs, and media operate. The current state of affairs is dramatically increasing the probability of a global catastrophe.
Rootclaim will continue to contribute its part in furthering these goals, by continuing to improve our methodology and by disseminating our analyses to a wider audience.
We’re proud to announce some exciting updates to Rootclaim. Since its inception, Rootclaim has focused on exposing the truth on many issues in public discourse using probabilistic inference. Rootclaim has established an outstanding track record, using proven mathematical models and publicly available information to overcome the flaws of human reasoning. However, this time consuming approach limited us to being able to respond to relatively few events.
We are glad to announce upgrades to Rootclaim that will deliver more agile, simplified, and timely analyses.
Previously, much of the analysis work had gone into the tricky effort of identifying and dealing with evidence dependencies. Going forward, we will group dependent evidence and analyze them jointly. This will allow Rootclaim and the readers to examine the evidence in the context of the investigation that exposed it, easily evaluate interdependencies, and assess the significance of missing evidence. Likewise, previously, a large portion of the effort had gone into the analysis of minor pieces of evidence, which have little effect on the calculated results. When analyzed as a group, minor details often lose their relevance.
These new efficiencies will allow us to publish analyses and provide the most reliable estimates about developing stories within a few days instead of the weeks of work that have been previously required.
The new analyses will be more readable while still maintaining Rootclaim’s high level of accuracy, and they can be read from top to bottom without knowledge of probability theory. Each group of evidence and its effects on the likelihood of the hypotheses will be clearly explained.
We have republished the following two analyses using the new model:
Among the many atrocities of the Syrian Civil War, the one that stood out was the use of chemical weapons, and particularly the nerve agent sarin.
While there is general agreement that there were multiple sarin attacks, most of the Western population has accepted that the attacks were carried out by the Syrian government. This assumption is so entrenched that objections to it are widely considered to be “conspiracy theories”.
Most of Rootclaim’s conclusions on other issues later became the consensus opinion, despite some initial pushback. Since this has yet to happen with regard to the sarin attacks in Syria, we decided to issue an open $20,000 challenge to debate anyone on this matter. This challenge has gone unanswered since April 2018, and we are now presenting it here in more detail, and increasing the bounty to $100,000. By doing so we hope to demonstrate the superiority of reasoning methods that integrate honest consideration of multiple hypotheses, unbiased analysis of evidence, and probabilistic inference.
Update: In June 2021, a video of opposition fighters launching rockets was matched to a field within opposition controlled territory, and that field has been shown to be at the intersection of seven rocket trajectories calculated from images of the impact sites. With this additional evidence we now consider the issue closed, demonstrating again the superiority of Rootclaim’s methods. While the $100,000 challenge is still available, we don’t expect anyone to apply.
Win a debate with a Rootclaim team member about the sarin attacks in Syria and take home $100,000. See our Rootclaim Challenge page for additional topics.
Do you have another hypothesis (e.g. Russia did it, or that some attacks were by the Syrian government while others were by the opposition)? Write to us and we’ll consider it.
The stakes: $100,000 each
This is the first in a series of Rootclaim Challenges, modeled after projects such as James Randi’s million dollar challenge, offered to anyone who can demonstrate paranormal powers in a lab setting (all attempts failed). To deter repeated submissions with the intention of winning by luck, we require the challenger to risk the same amount. Applicants who can’t afford to risk $100,000 are encouraged to pool funds together or even crowdfund it. We are willing to reduce the stakes as low as $10,000 for applicants already involved in public debate on the issue.
Both sides will first agree on two judges with strong analytical skills, relevant experience, no previous endorsement of either side, no relevant political biases, and who declare they will examine both hypotheses equally.
Choosing judges will be done publicly on Twitter, so evasion attempts by either side, such as offering biased judges, are exposed. As an example of our honest approach to this process, in a past discussion, when Nassim Taleb offered Glenn Greenwald as a judge, we agreed to bend the rules and accept him, even though he previously said there is “overwhelming” evidence the government is responsible (contrary to Rootclaim’s conclusion) – because we think he is capable of changing his mind when presented with evidence.
Each side will have 8 hours in total to present its case, including time to respond to the other side’s claims, as part of a two-day event.
The debate will be based on all currently available evidence. The goal here is not to trip up or trap the opponent, but to determine which hypothesis is better supported by the evidence. If you have new evidence, or evidence we overlooked, it should first be shared, so we can update the analysis, and if it doesn’t significantly change the conclusion, the challenge can be accepted. We are not claiming to have better evidence, but rather aim to demonstrate the superiority of probabilistic reasoning over human reasoning, when evaluating the same evidence.
Each judge has to declare which of the two hypotheses is more likely. If both agree, the prize pool, minus the debate expenses, is paid to the winner. Otherwise, it is split.
We are flexible – feel free to contact us with offers.
Who declined so far?
The following people have been sent a tweet offering to participate in the challenge but declined or failed to respond. All of them have publicly expressed very high confidence that the Syrian government is responsible.
Eliot Higgins – Founder of Bellingcat.
Brian Whitaker – Journalist and former Middle East editor of The Guardian.
Chris York – Senior editor of Huffington Post UK.
Josie Ensor – Middle East correspondent for The Telegraph.
Scott Lucas – Editor of EA WorldView and Professor at University of Birmingham.
Richard Hall – Middle East correspondent for The Independent.
Julie Leranz – Senior adviser at The Israel Project and a director at The Human Security Centre.
Kristyan Benedict – Amnesty International UK Campaigns Manager.
Dan Kaszeta – Security and CBRN specialist and writer for Bellingcat.
Tobias Schneider – Research fellow at Global Public Policy Institute (GPPi).
Gregory Koblentz – Director of Biodefense Graduate Program at George Mason University.
Numerous other individuals who were very active on social media discussing this issue.
Have you notified anyone of the challenge and they declined it? Let us know and we’ll add them to the list.
A study from October 2020, the first randomized controlled trial of its kind, showed that high doses of vitamin D (in the form of calcifediol) reduce the severity of Covid-19 in hospitalized patients. The researchers reported a 30-fold(!) reduction in intensive care admissions of Covid-19 patients. At Rootclaim, we analyzed these findings and concluded that even under conservative assumptions accounting for limitations in the study, the effect is still significant and likely around 5-fold. We further demonstrated that since the risks of treatment are low, this treatment protocol should be immediately implemented. Since we published our analysis, additional studies have supported this conclusion.
Many health professionals, government officials, and other decision makers worldwide have seen the studies, but they have yet to update treatment guidelines. This delay may be due to the following:
Not many have the background in statistics and probability required to assess the data, and distinguish it from the dozens of false claims about COVID treatments.
They’re affected by omission bias – they default to the “safe” alternative of inaction, waiting for more data, rather than choosing action. It’s easier to later defend inaction than face criticism for acting too soon.
Their incentives are completely misaligned with the public. The damage to the public from using vitamin D when it isn’t effective is negligible, but the damage caused by inaction in the case that vitamin D is effective is enormous. To the decision maker in a personal capacity, the damage is similar in either case – one wrong decision on their record.
When it comes to low-risk, low-cost treatments, decision-makers hedging their bets on inaction leads to avoidable deaths.
In this particular case the reasons to act now are clear:
Similar treatments have been performed for decades, and the risks are known to be low, especially in this setting, when patients can be monitored at the hospital.
The benefits of the treatment, on the other hand, are potentially enormous, effectively reducing Covid-19 severity to that of the seasonal flu.
While caution is often the correct path when dealing with public health, this is a case where decisions should be made swiftly, using the best available models. At Rootclaim, we develop such models so when our analysis exposed the implications of the new findings, we decided to promote the adoption of the proposed treatment. We hope that this unique challenge will allow the information to reach more decision makers, and save the millions of lives that will likely be lost while waiting for further studies.
Rootclaim is willing to bet $100,000 that vitamin D is effective in reducing the severity of Covid-19.
This is the second in a series of Rootclaim Challenges and is intended to show that the reluctance to implement a vitamin D protocol today is irrational. A decision maker who is not pushing to adopt the proposed protocol is effectively claiming that the probability that this protocol is better than existing treatments is low. But that also implies that taking the bet would be very profitable. Therefore, any professional not accepting this challenge is implicitly admitting that their decision not to promote the treatment is wrong.
UPDATE (Nov 2022): Since this challenge was published, in the early stages of the pandemic, multiple studies have been done on the subject, with the overwhelming majority finding vitamin D effective. Over time, the virus has significantly changed its methods, and the population has changed due to vaccines and immunity. Therefore the analysis of vitamin D’s efficacy on the original virus and population is no longer relevant today and should be updated.
Since the pandemic is no longer a major risk, this update is not a priority for Rootclaim. Nevertheless, if you wish to debate the original analysis, please contact us to set the exact criteria.
The challenger needs to show that they can commit $100,000. We are open to discussing lower or higher amounts, and the funds can be pooled from multiple sources.
Both sides will agree on an arbitrator who will review the evidence.
The challenger needs to declare that they do not have access to any relevant non-public information. This is to protect from abuse in case of unpublished research (there is still a small chance that further research will discover the treatment is ineffective).
For the same reason, we may update these terms or withdraw the offer, as new information emerges. Of course, once a bet is made it is final and cannot be withdrawn.
If you’re not willing to risk your own money betting against vitamin D, why are you willing to risk someone else’s life?
Study participants that received a large dose of vitamin D (as calcifediol) experienced a 50-fold reduction in the odds of admissions to intensive care, which likely translates to a similar reduction in death rates (see further analysis). If these findings are accurate, the end of the pandemic is near. The study has multiple limitations that would normally warrant waiting for studies, but given the circumstances, it is important to dig deeper and accurately assess its implications. We will estimate the probability that the finding is true, and analyze the risks of adopting the treatment now vs. waiting for further studies.
Is the finding true?
A randomized controlled trial, where patients are randomly selected to receive the treatment or serve as control, makes it possible to isolate whether some clinical finding results from the treatment or from another factor. So far, there have been many studies on vitamin D and COVID-19, which demonstrated a strong link between the two, but causality had not been established. For example, people with poor health may have low vitamin D levels due to low sun exposure, creating a correlation with COVID-19 severity that is not causal.
We now have the results of the first randomized controlled trial on the effect of vitamin D on Covid-19 patients. If it was properly conducted, causality has finally been established, and an effective treatment was found. Unfortunately, the study has several limitations that may distort its result.
Let’s review these possible problems, and their significance. A more mathematically rigorous analysis may be found in the appendix below.
1. The sample size is small, so the findings may be due to chance
It is always possible that the patients that were randomly assigned to the treatment group suffered less deterioration by mere chance. This possibility is calculated using the p-value, which measures the probability of obtaining the study result (or stronger) by chance. The authors disclose it only as less than 1 in 1,000, but the actual number is less than 1 in 1,000,000 (can be verified here, using the study results of 13:13 vs 49:1).
It is important to understand that once a p-value has been obtained, the sample size no longer matters. The goal of a large sample is to reduce the random differences between the two groups, thus making the difference in treatment a larger factor in the final result. The p-value improves both with study size, and with effectiveness of the treatment.
In this case, the effect was so strong that the relatively small sample (76 people) turned out to be much larger than required.
It can be said with certainty that if the experimental results are incorrect, it is not because of the sample size or chance.
2. The control group included more people with risk factors
The control group happened to have significantly more people with hypertension, so it is expected they would have more admissions to intensive care. The researchers identified this issue and performed another analysis (logistic regression) that accounted for it, and the findings were only mildly weakened, from a 50-fold to a 30-fold reduction, with 95% confidence that the result is between 4-fold and 300-fold. We will use 12-fold as a conservative estimate.
We performed another analysis, which assumed that only those with high blood pressure could deteriorate (i.e. removing patients without hypertension from the sample), and the findings still remained very significant, with a p-value of 1 in 5,000, far better than the standard threshold of 1 in 20, or 0.05.
Another issue to evaluate is whether this imbalance indicates a deeper problem with randomization or reporting. The reported p-value of the difference in hypertension is 0.0023, meaning a 1:435 chance it would happen in a random assignment. However, this is just one of at least 10 parameters that could affect the study, and the p-value also accounts for an opposite effect (2-sided instead of 1-sided), so the probability that one of them would happen is only around 1:21, meaning 1 in 21 such studies would have such an imbalance by mere chance – hardly remarkable. Given that randomization was done electronically upon patient admission, such a mistake is unlikely, and as fraud it won’t make much sense (especially as it is later reported and corrected for).
This clearly seems like a chance occurrence, and we see no reason to reduce the estimate beyond what the investigators already did.
3. Patients in both groups were also treated with hydroxychloroquine and azithromycin
Patients in both groups received the standard treatment, which at the time was hydroxychloroquine and azithromycin, a treatment that has since fallen out of favor. Could it be that the findings result from vitamin D neutralizing negative effects of those drugs? This option is unlikely. Trials have shown differing results for hydroxychloroquine and azithromycin, with a few pointing only to a mild risk.
It is also possible that vitamin D only works in combination with the other treatments. Given these mechanisms of action, this is highly unlikely.
We estimate that, at most, this possibility reduces the effect from 12-fold to 8-fold (i.e. vitamin D may have neutralized a 50% increase in severity caused by the other drugs).
4. The experiment was not double-blind placebo-controlled
To prevent distortion of the results by the trial participants or the researchers (even unconsciously), it is preferable that neither know which patients were randomized to the control group and which to the treatment group. This was not the case in this experiment.
This is certainly a weakness of this study. It was mitigated by delegating the decision regarding transfer to intensive care to a committee of experts that included members of the hospital’s ethics committee, who were not aware which group the patient was assigned to, and reached decisions based on a structured protocol.
We have reached out to the investigators to learn more about the procedures, and learnt that this was a result of logistic problems in placebo manufacturing. We got the impression that an honest effort was made to mask the data as much as possible, and the two groups were not otherwise treated differently.
We still need to account for the possibility of outright fraud enabled by this weakness, in which case the findings are false. Since there are no commercial interests around vitamin D, and the fraud would be exposed in later studies, we assign this a probability of 10% at most.
5. There may be another, yet unidentified, factor
Of course, there may be another source for the dramatic difference between the two groups, which has not yet been identified. This would usually be the responsibility of the publishing journal to expose. In this case, the publication has been peer-reviewed and published in a small journal specializing in vitamin D. The publisher is Elsevier, which also publishes the Lancet and Cell.
Such a major finding should ideally be published in a world-leading journal, but given the limitations above, and the likely urgency to publish, it is not unreasonable to choose a smaller journal.
Given the relative simplicity of the trial, we do not see unknown factors as a major risk, at most accounting for a further reduction from 8-fold to 6-fold, and a 10% probability of it invalidating the results.
6. Is the prior probability of the study findings low?
Equally important is the likelihood that vitamin D could cure Covid-19, based on the information known before the article was published. For example, if a study finds that five minutes of neck massage cures lung cancer, it is very likely that there is some error in the study, even if its statistical significance was high.
A recent analysis associates COVID-19 severity with a “bradykinin storm”, and offers vitamin D as possible treatment.
Other effects of vitamin D on COVID-19 quoted in the study include: regulating the renin‑angiotensin system, modulating neutrophil activity, maintaining the integrity of the pulmonary epithelial barrier, stimulating epithelial repair, and tapering down the blood’s increased coagulability.
Many previous studies (such as here, here, and here) have already shown a correlative (but not causal) connection between low vitamin D and COVID-19 severity. The new publication only verifies that the connection is causal.
See additional discussions including potential mechanisms here and here.
However, so far the indication has been for a weaker effect – about a 50% reduction in severity, not 30-fold, so the new finding indicating a near cure is initially surprising. But on further examination, there may not be any contradiction between the studies. A re-examination of a study that published detailed data shows that the rate of infection drops to nearly zero with high levels of vitamin D in the blood (above 50 ng/ml). That is, it is possible that the effectiveness increases with dose, and in the very high doses, as used in the study, near healing is achieved.
It should also be noted that the earlier observational studies used vitamin D levels that were measured a significant time before infection. By the time patients got sick their levels may have changed, which would cause a possible strong correlation to appear weaker.
Another possibility is that the use of short term high dose calcifediol is more effective than the long term supplementation of vitamin D3.
Additionally, the investigators have informed us that the protocol has been used on patients after the trial completed, with similar results.
Overall, according to the prior knowledge, a mild effect is more likely than a strong or no effect, reducing our conservative estimate from 6-fold to 5-fold.
Summary – The findings are true
None of the possibilities mentioned invalidates the significant finding that emerges from the experiment. It is very possible that some minor biases occurred that exaggerated the effect, but it is unlikely that vitamin D had no positive effect.
Summarizing the numbers above, we estimate:
20% probability that vitamin D has no significant effect on COVID-19 severity
80% probability that it reduces severity and death, probably around 5-fold, and possibly much more.
In order to make a treatment policy decision, one must consider not only the likelihood that the finding is true, but also the potential harm and benefits of each possible course of action.
Alternative 1 – Wait
The easy decision is to wait for further studies to verify the new finding. This is what medical experts would normally decide after a first publication of a successful trial.
If the treatment is ineffective, there are no costs and risks to this decision.
If the treatment is effective, then based on the analysis above, the results in the study’s control group, and typical outcomes for hospitalized patients, the harm to a typical hospitalized patient, can be estimated as:
Additional 20% chance of suffering severe disease, with likely long-term implications.
Additional 5% chance of death
Alternative 2 – Adopt treatment
The second alternative is to immediately adopt the protocol for hospitalized patients. In this case the harm to patients is from the large vitamin D dose (whether or not the treatment is effective).
As vitamin D is already a popular treatment, there is abundant information on its risks.
The dose used in the study is about 10 times the maximum recommended dose for prolonged use.
However, the treatment protocol in the study is relatively short – until release of the patient or transfer to intensive care. Previous studies on short treatments at similar doses found them to be safe.
The risk in vitamin D is with increased use that maintains very high levels in the blood over a long period of time.
Even then, the risks are relatively limited, and can be corrected by a low-calcium diet and steroids. For hospitalized patients that can be monitored closely the risk is likely further reduced.
Covid-19 specific risk: vitamin D increases the expression of ACE2 in cells, which acts as a receptor for the coronavirus. Therefore, until now, there has been apprehension about its use. Since the new trial focused on COVID-19 patients and doesn’t show such negative effects, the concern seems to have been alleviated. There is still some low likelihood that the study results were completely wrong, either intentionally or due to a catastrophic mistake that hid a worse outcome in the treatment group.
Based on existing knowledge, the risks in the proposed protocol appear to be low.
The risk can be further reduced by monitoring vitamin D levels in the patients’ blood, and keeping them in a high yet safe range, for example 80 ng/ml.
It is safe to assume the risks of the protocol are much lower than:
5% chance of severe complications.
1% chance of death
Given that both:
The likelihood that the treatment is very effective is greater than 50%;
The benefit of the treatment, if effective, is far higher than twice the risk of the treatment;
it is obvious that the right decision is immediate adoption of the treatment protocol.
Hospitals deciding to wait for further studies should have very strong reasoning that outweighs the apparent harm to patients by delaying treatment.
Global Implications on the COVID-19 pandemic
This analysis shows that if the protocol is widely adopted, COVID-19 severity can likely be reduced to that of the seasonal flu, allowing alleviation of certain limitations, which could bring a major improvement in the economy and social health.
A further conclusion, although with lower confidence, is that vitamin D could be effective at earlier stages of the disease. Previous studies have shown a correlation between high vitamin D levels and lower infection rates. The new study establishes a causal connection at late stages, increasing the likelihood that the correlation at earlier stages is also causal. This would mean that widespread vitamin D therapy (e.g. bringing blood levels to a healthy 30-40 ng/ml) could reduce R0. If that reduction is as significant as indicated by the studies, R0 could drop below 1, and stop the pandemic.
Since vitamin D deficiency is already common, and risks are negligible at this dose, governments should immediately encourage and subsidize vitamin D tests and supplementation for the general population.
Update 12 November 2020: An MIT study that went deeper into the statistical aspects of the trial has reached a similar conclusion that the results of the Cordoba trial are reliable, and that vitamin D treatment is the likely explanation for the dramatic reduction in ICU admissions of hospitalized Covid-19 patients.
Update 24 November 2020: Two additional double-blind, randomized, placebo-controlled trials have reached opposing conclusions regarding vitamin D and COVID-19. One study (printed in The BMJ) showed that those receiving a high dose of vitamin D were dramatically more likely to test negative for the virus within 21 days (21% vs 62%). The other study (preprint on medrxiv) reported that vitamin D supplementation did not significantly reduce hospital length of stay for patients with COVID-19.
Our analysis of the second study shows it does not significantly change the picture, and we maintain our contention that vitamin D should be immediately adopted as a treatment and prophylaxis for COVID-19.
The main flaws are that the study was designed in a way that had a low probability of achieving any of its endpoints, and that the treatment protocol itself was not suitable to demonstrate the efficacy of vitamin D.
The study assumed that, if vitamin D were effective, it would reduce hospital stay from a mean of 7 to 3.5 days. That is an unreasonable expectation, especially given that bolus vitamin D takes a day or two to be mostly converted to 25(OH)D (calcifediol), and may take even longer for 1,25(OH)D. Previous trials were based on longer periods, or used calcifediol directly.
Due to low mortality rates in the study, the study was incapable of detecting a significant effect. Even if there were zero deaths in the treatment group, the result would still be considered not significant (i.e. p>0.05), The study had only a marginally better chance of detecting an effect in ICU admissions or mechanical ventilation, and indeed there it showed some effect, although it is still not significant (as expected).
The primary measure was length of hospital stay, which can be misleading. A short hospital stay could signify either a positive outcome due to improved healing or a negative outcome as with a quick death.
Patients were recruited 10 days after symptoms began, and 90% were already on oxygen. It is possible that vitamin D’s effects are not very relevant at that point, so the failure to improve patient outcomes at a late stage does not reflect the efficacy at earlier stages.
The effects of the vitamin D treatment could have been counteracted by the administration of steroids (to 62% of patients in the study), which weaken immune system activity.
While the study cannot demonstrate vitamin D is ineffective, due to its flawed design, it does suggest that at a late stage, alongside steroids, vitamin D is not immensely effective, i.e. it does not reduce the odds of severe outcomes by 5x or more. Our analysis at Rootclaim suggests that a 5x reduction is a reasonable outcome when vitamin D is administered correctly.
Update 24 May: A retrospective study on hospitalized patients with COVID-19 who received calcifediol showed a drop in mortality rate from 20% to 5%.Appendix – Bayesian Analysis
For those with a background in probability, following is a more rigorous analysis using Bayesian inference. By explicitly stating prior probabilities of hypotheses, and calculating the conditional probabilities of the study results under each hypothesis, a more accurate and robust result is achieved, removing the need to analyze sample sizes, p-values, or confidence intervals.
We will define five hypotheses to be considered:
Damage – Vitamin D worsens COVID-19.
Nothing – No effect
2-fold – Vitamin D reduces the odds for severe COVID-19 by around 2.
5-fold – Vitamin D reduces the odds for severe COVID-19 by around 5.
20-fold – Vitamin D reduces the odds for severe COVID-19 by around 20.
First we shall estimate the probability of each hypothesis based on what was known before the new study. As a baseline, few drugs are effective for any specific disease, but as described above, there are multiple studies showing correlation between vitamin D and COVID-19, and several proposed mechanisms of actions. On the flip side, there is the aforementioned risk that vitamin D could actually exacerbate COVID-19 by increasing ACE2.
We will represent these facts with the following prior probabilities:
Damage – 10%
Nothing – 67%
2-fold – 15%
5-fold – 5%
20-fold – 3%
Adjustments to Study
Given the limitations discussed above, we will make the following adjustments to the study results:
Move 2 cases from ICU to non-ICU in the control group, which we attribute to the higher hypertension cases there.
Move 2 cases from non-ICU to ICU within the treatment group, and do the opposite in the control group, due to unknown weaknesses not yet identified.
Estimate at 20%, as above, the probability that the study was grossly mismanaged, and should be ignored.
So instead of the reported matrix of:
Admitted to ICU
Not admitted to ICU
We will use:
Admitted to ICU
Not admitted to ICU
Next we estimate the probability of getting the adjusted study results, under each of the five hypotheses. To do that, we will use the odds of the control group (9:17 = 9/26 = 34.6%), and adjust by the hypothesis factor, to receive the expected odds in the treatment group. For example, the expected odds in the 2-fold hypotheses would be 9:17*2 = 9:34, or a probability of 20.9%. We then use a binomial distribution formula to estimate the conditional probability of getting the exact study result (3 out of 50 trials) given those expected odds. This is then normalized to sum 100%. Lastly we average with the prior probabilities at a weight of 20%:80%, accounting for the 20% possibility that the study is meaningless.
The full calculation:
Odds ratio (OR)
Convert odds to probability = 9/(9+17*OR)
Conditional probability from binomial formula
Posterior = Prior * Conditional
Posterior, normalized to 100%
Account for 20% failed study possibility (final result)
This more rigorous analysis reaches a very similar conclusion of around 80% likelihood that vitamin D is effective against COVID-19, with a 5-fold reduction being the most probable.
There have been many hypotheses surrounding Stonehenge and its long-lost origins. Can a mathematical assessment by Rootclaim help shed light on its original purpose?
The large prehistoric monument in rural England is comprised of a circle of upright stones. It was constructed around 5,000 years ago, under circumstances that have long been lost in the annals of history. Some believe that Stonehenge served a religious function, while others say it was used as a burial site or a place of mystical healing. Yet others argue that it served as a giant calendar. There are also conflicting claims about how the bluestones used to build Stonehenge were moved from their place of origin 150 miles away. One explanation is that a glacier flow moved the stones, while another view says that people transported them from quarries in Preseli.
How can probability theory help?
Probability theory helps researchers measure uncertainty. And there’s plenty of uncertainty surrounding Stonehenge. By using a probabilistic framework, we can model the likelihood of each piece of evidence relating to Stonehenge.Continue reading
The human mind doesn’t deal well with complexity. It seeks shortcuts, often being fooled by one of many cognitive biases. One of the goals at Rootclaim is to reduce uncertainty by breaking down complex questions into more manageable pieces. This whole system is strengthened by the open crowd-sourced approach, which increases the breadth, depth, and creativity of the analysis. A good case in point is the contrast between the recent UN Joint Investigative Mechanism (JIM) report on the Khan Sheikhoun chemical attack, and the Rootclaim analysis of the same incident.
Rootclaim is a collaborative analysis platform that transforms how people understand complex issues by combining the power and reach of crowdsourced information with the mathematical validity of Bayesian statistics.