Yet Another Study that Exaggerates COVID-19 Vaccine Effectiveness

A recent study titled ‘Averted mortality by COVID-19 vaccination in Belgium between 2021 and 2023’ published in the journal Vaccine claims that 12,800 COVID-19 deaths among people aged 65 years and over were averted, representing a reduction in Covid mortality of 54%. Presumably the study was carried out in good faith and, while it does mention potential limitations related to confounding factors and potential over diagnosis of COVID-19 cases, it also omits to include other limitations which, if considered, would reduce the deaths averted and lower the reduction in mortality.

Even taken at face value, the results may not be especially impressive, raising the question of how effective COVID-19 vaccines are and whether they are worthwhile. The answer to the first question is “not very”. The second can only really be answered by a health economist, but the answer is probably “most likely not”.

We’re well used to hearing, in addition to being very safe, that the COVID-19 vaccines were highly effective, with figures as high as 100% being quoted, but these figures were misleading. Vaccine companies are wont to express vaccine effectiveness in terms of relative risk reduction (RRR) instead of the recommended absolute risk reduction (ARR).

For infections with low mortality, ARR can be much lower than RRR and conveys the true figure for how much less likely a person is to become infected or die if they are vaccinated compared with being unvaccinated. It transpires, compared with other common vaccines, that none of the COVID-19 vaccines are much good. For example, figures of 20%, 5-20% and 5-15% are quoted for ARR to prevent infection of smallpox, cholera and yellow fever vaccines, respectively. In contrast, the ARR of COVID-19 vaccines to prevent infection hovers around and rarely exceeds 1%.

While the ARR of the COVID-19 vaccines (to prevent death) was not reported in the Belgian study, it is possible, with the figures presented, to estimate it at 0.85%. This means that the number of people needing to be vaccinated (NNV) to prevent a single death is 117. To prevent 10 deaths the NNV is 1,170 people.

Accepting that saving life is important, how robust are the results of the study? The limitations already referred to by the authors could potentially reduce the number of survivors and alter the above figures (lowering the ARR and increasing the NNV). However, the further limitations not considered by the authors, include the ‘cheap trick’ and the ‘healthy vaccinee effect’ (HVE).

The cheap trick, described by Martin Neil, Norman Fenton and Scott McLachlan, involves the method whereby people are classified as unvaccinated. This varies across studies but involves the inclusion of people who have received a dose of a vaccine yet remain classified as unvaccinated for a specified period. In the case of studies of COVID-19 vaccines this can be as long as 21 days but, as in the Belgian study, is often 14 days.

While it may make some immunological sense thus to classify, there is no agreement on precisely when protection from COVID-19 purportedly kicks in and ‘a couple of weeks’ is the time usually quoted. The effect of classifying early vaccinated people as unvaccinated increases the apparent effectiveness of the vaccine compared with correctly classifying them as vaccinated. Neil, Fenton and McLachlan estimate that the cheap trick can provide vaccine effectiveness of 70-90% even if a vaccine is completely ineffective.

Contrast the misclassification of the vaccinated people as unvaccinated at the beginning of a study with the use of intention to treat (ITT) analysis at the end of a clinical trial. ITT means the final statistical comparison of the treatment and control groups includes everyone who was allocated to the arms of the study, regardless of whether they dropped out, died or were discontinued for other reasons.

ITT protects against potential differences in the attrition from either arm of the trial. This principle is directly violated by the misclassification of vaccinated people as unvaccinated in vaccine studies such as those used to evaluate COVID-19 vaccines due, amongst other things, to the healthy vaccinee effect.

The HVE is based on the evidence that those who tend to accept vaccination differ from those who refuse vaccination. The former group tends to be healthier and to be comprised of people who make healthier life choices throughout life including (because they assume that it is a good choice) to be vaccinated. Of specific relevance to the Belgian study is the fact that people who are more likely to die soon after – regardless of the effect of COVID-19 – are less likely to accept vaccination.

Very small apparent uptake of vaccine doses can have large healthy vaccinee effects in studies. For example, the apparent effect of a vaccine will be reduced by between 25-75% if the number of doses of a vaccine refused based on the HVE differs by between 0.6-1.9%.

In retrospective studies of vaccination, such as the Belgian study, the cheap trick and the HVE if not considered will work synergistically to exaggerate the apparent effectiveness of the COVID-19 vaccines. If a conservative 15% adjustment is made to the figures in the Belgian study due to the application of the cheap trick, then deaths averted falls to 10,880, ARR to 0.726% and NNV increases to 138. If the effect of the cheap trick is larger and the HVE is large, then the statistics related to the COVID-19 vaccine could be much less impressive.

None of this means that COVID-19 does not have the potential to kill older people or that the COVID-19 vaccines do not work. But the fact remains, if they do work, we still do not know how effective they are. In addition, the harmful effects of the COVID-19 vaccines, including death, which are now well documented, should be considered. Even if the NNT really is 117, as reported in the Belgian study, that means that for each life saved, 116 people have been exposed to potential harm by the COVID-19 vaccines. If one person of the 116 dies of vaccine harm, then the net benefit of the vaccines is zero.

Considering the issues explored — ranging from the modest absolute risk reduction (ARR) reported in observational studies to methodological concerns such as the healthy vaccinee effect and time-dependent misclassification — the effectiveness of COVID-19 vaccines in preventing death, particularly among older people, may have been overstated in key studies. While some level of benefit is plausible, the extent of that benefit must be critically appraised in view of these well-documented biases.

Public health decisions must rest on transparent, unambiguous evidence. If the purpose of mass vaccination is to prevent death at scale, then accurate estimations of the number needed to vaccinate (NNV) and deaths truly averted are essential. Anything less undermines both scientific integrity and public trust.