'GRADE' Methods applied to meta-analysis of Pfizer & Moderna RCTs, rating Quality of evidence: we strongly recommend 'against use' of COVID mRNA vaccines to reduce mortality, very low quality evidence
by Paul Alexander
mRNA vaccine trial data submitted to FDA for EUA, combined in meta-analysis to improve precision and power, finding very low certainty evidence against the use of mRNA vaccines
A recently reported Danish study by Stabell Benn et al. (not yet peer-reviewed) sought to unravel the impact of mRNA and adenoviral vector COVID vaccines on mortality outcomes and found that the mRNA vaccines failed to save lives (RR 1.03, 95% CI 0.63 to 1.71). A clear benefit was however seen for the adenoviral vector COVID vaccine (RR 0.37, 95% CI 0.19 to 0.70). Their analysis was based on available data and while still pre-print, it has had very serious review by senior global scientists/epidemiologists who have pronounced that the rigor and quality is sufficient. The key is this, that the mRNA vaccines do not confer COVID mortality benefit while the adeno-viral vector vaccine do.
Dr. Martin Kulldorff (prominent vaccine scientist) then commented on this pooled Danish study with a very powerful and crisp commentary, asking a very important question: Have populations been mandated vaccines that fail to save lives and just do not work? I am particularly concerned as a skeptic and contrarian given these mRNA vaccines have now been shown to be ineffective on the Delta and Omicron variants and not properly safe. The Pfizer and Moderna COVID mRNA vaccines do not sterilize the virus (are non-neutralizing as based on the initial Wuhan legacy strain and as such do not match the current variants (spike protein)) in that they do not prevent infection, replication, or transmission. If a vaccine does not cut the chain of transmission in a pandemic of infectious disease, then it cannot achieve herd immunity and is essentially failed.
With this background in place, this analysis by me sought to replicate the Danish finding (conducting the actual meta-analysis) as well as apply GRADE methods to rate the quality of the mortality evidence, culminating in a recommendation for or against the use of the mRNA vaccine in reducing mortality. I also include a rudimentary background on GRADE and apply what can be called ‘GRADE lite’. I simply wished to assess what a GRADE type application to the existing trial data (limited as it may be and of poor methodological quality) would look like since the folk in the EBM world will not apply their expertise to the current body of evidence.
The findings as you see below, cuts against their (medical community, academic scientists, EBMers, CDC, NIH etc.) narrative for two years and certainly on the vaccines. I have learnt that the medical/clinician and academic research community actually practices ‘do not bite the hand that feeds you’ very, very well. They are experts at it. The ‘hand’ is the research grants hand, salaries, career positions etc. To advance their agenda, the key is their agenda, and it is led by the CDC (and the NIH/NIAID), who in the US functions as a political arm of government.
The CDC worked under the Trump administration to undermine him, while works under the Biden administration to support him. They use the MMWR reports to hurt an administration as they wish to and do it by producing bogus incomplete reports with only some of the data and with biased unclear misleading conclusions. They mislead the public and whom within the government they wish to mislead. The data they with held is not only the 18-49 years old triple booster vaccine data (this is what we have come to know of, there is major more swept under the rug), they are with holding major data and have been doing this all along and IMO it is a real crime. They must be held to account. The reticence by the FDA to make the data available to the public for 55 to 75 years was shocking and showed that there is serious information there that shows much wrong they did. The FDA, CDC, NIH, Pfizer, Moderna and all of their leaders like Bourla and Bancel etc. must be at some point accountable. It must be investigated.
The financial hand, the fame hand, the ‘keeping my job’ hand is stronger than the ‘be ethical and do all I can to protect my patients and conduct proper medical research’ hand. COVID has revealed the putrid underbelly of medicine and academic research that involves the medical journal publishers and editors etc.
Recommendation: Based on this GRADE assessment, we recommend strongly against the use of the mRNA vaccines to reduce mortality, based on very low quality/certainty of evidence
In making a strong recommendation against use based on very low quality evidence, I am arguing that the recommendation against (strength and direction) will remain regardless of future study. It is however very likely that with future properly designed trials that run to sample size and adequate event numbers etc. as well as all the limitations both the Pfizer and Moderna trials suffered from, then the quality of evidence would increase e.g. “a strong recommendation against use of the mRNA vaccines based on moderate to high quality evidence”.
This would mean that we would be confident that regardless of how many more studies are done, that we would be even more certain in the quality of evidence (the certainty in the evidence informing the recommendation would increase from ‘very low’ as I have rated now). Rating it presently as you would see below, as ‘very low certainty’ evidence, means that we are very uncertain about the underlying estimates of effect e.g. it may not represent the true ‘population’ estimate of effect.
Reasoning: The body of evidence over 1 year since vaccine roll-out shows that the vaccine is ineffective with massive waning vaccinal antibody (Ab) immunity; it does not stop infection or transmission; it confers no benefit in reducing mortality (overall, COVID etc. as seen in presented forest plots)) yet brings known harms to the vaccinee (myocarditis, brain clots/CVST, paralysis, death post vaccine e.g. VAERS, EudraVigilance, British Yellow Card system etc.). Well-informed persons would consider the benefits vs the costs/harms and will likely (if informed) place more value in avoiding the harms and less value in the absence of (or highly uncertain) benefit. Most well-informed persons would want to avoid the vaccine given no benefit in reducing mortality when compared to placebo. Hence the strong recommendation ‘against’.
CRUDE GRADE applied, or GRADE lite (not full GRADE and based on one mortality outcome)
GRADE, what is it?
GRADE's approach to rating the certainty of the evidence is based on a four-level system (high, moderate, low, very low). The GRADE approach classifies bodies of randomized controlled trials as initially starting at high certainty and bodies of observational studies at initially starting at low certainty.
Question: In adult persons (persons at high risk for SARS-CoV-2 infection or its complications) given the mRNA COVID vaccines compared to placebo/control (no vaccine), is there a reduction in risk of death?
2 Pfizer and Moderna RCTs were combined (pooled) in meta-analysis, random effects modelling, relative risk (ratio of the risks), mortality (overall etc.) outcome
mRNA vaccines in pooled analysis:
Pfizer RCT; https://www.nejm.org/doi/suppl/10.1056/NEJMoa2110345/suppl_file/nejmoa2110345_appendix.pdf
Key methods issues:
RCTs start as high-quality evidence; all three forest plots show no benefit of vaccine
GRADE application and explanation of the body of evidence downgrades (quality/certainty rating), 3 levels downgraded across the 5 domains used in rating certainty/quality of evidence (risk of bias, imprecision, inconsistency, indirectness, publication bias):
Risk of bias (two level downgrade for blinding concerns, stopping early and data loss)
i) stopping early which is a red flag for high risk of biased estimates of effect; may have arrived at a ‘random high’ and the benefit may have disappeared had the study gone to the planned sample size
ii)unblinding, giving intervention to placebo, impossible to now assess efficacy or harms; in effect, we are no longer in a study, it is over
iii)data loss due to omission of suspected, but not lab confirmed cases (approx. 3500 cases/Pfizer submission); 95% RR declines to 20% when back calculated
iv)serious concerns with the methods reported in each clinical trial e.g. randomization/allocation concealment, handling and accounting for participants
v)other issues, such as short duration of follow-up to assess safety signals etc. and not ideally a risk of bias but I felt I would note it for the reader as this is a key failure of these RCTs and failure of the FDA to demand appropriate follow- up across longer duration; we in effect, do not know what harms may emerge
Imprecision (one level downgrade as we see pooled estimate’s 95% CI spans both benefit and harms (both sides of the line of no effect); also OIS not reached, small event number)
“Certainty is lower if the clinical decision is likely to be different if the true effect was at the upper versus the lower end of the confidence interval. Authors may also choose to rate down for imprecision if the effect estimate comes from only one or two small studies or if there were few events.” If the 95% CI includes appreciable benefit or harm (we suggest an RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate
Inconsistency (no downgrade)
Indirectness (no downgrade)
Publication bias (no downgrade)
Therefore, 3 downgrades for very low quality of evidence
Example of how the Meta-analyses were performed:
Overall mortality forest plot using a random effects model, relative risk
We see 31 events total in vaccine group, 30 total in placebo; we see non-significant heterogeneity, p=0.89, I 2 of 0%; studies are comparable
The pooled estimate of effect is RR 1.03, 95% CI of 0.63 to 1.71, meaning that with these 2 pooled RCTs of 74, 193 patients, there is no difference in vaccine versus placebo in terms of reducing overall mortality, p=0.90
We see 2 events total in vaccine group, 5 total in placebo; we see non-significant heterogeneity, p=0.81, I 2 of 0%; studies are comparable
The pooled estimate of effect is RR 0.40, 95% CI of 0.08 to 2.09, meaning that with these 2 pooled RCTs of 74, 193 patients, there is no difference in vaccine versus placebo in terms of reducing COVID-19 mortality, p=0.28
iii)OTHER non-COVID-19 MORTALITY:
We see 11 events total in vaccine group, 12 total in placebo; we see non-significant heterogeneity, p=0.85, I 2 of 0%; studies are comparable
The pooled estimate of effect is RR 0.91, 95% CI of 0.40 to 2.08, meaning that with these 2 pooled RCTs of 74, 193 patients, there is no difference in vaccine versus placebo in terms of reducing COVID-19 mortality, p=0.83