AstraZeneca is working with the University of Oxford to develop a coronavirus vaccine
DAN HIMBRECHTS / AAP / PA Pictures
Earlier this week, the pharmaceutical company AstraZeneca and Oxford University published positive results on their Covid-19 vaccine. They claimed that it was on average 70 percent effective and could reach 90 percent effectiveness depending on the dosage. However, some questions have now been raised about the results. What’s happening?
How are the results not good news?
As scientists, journalists, and business analysts chewed on the numbers, they noticed some issues with the conduct of the study that cast doubt on the reported results and could hinder the vaccine’s approval.
What kind of problems?
In a word, methodically. “There are many problems,” Paul Hunter, professor of medicine and expert in clinical trials at the University of East Anglia, UK, told New Scientist.
Instead of the results of a large Phase 3 study as we have seen from the results of previous vaccination trials, these results have actually been pooled from two separate studies, one in the UK and one in Brazil. And there was a glaring flaw in the way either of the attempts was conducted. Due to a laboratory error, some study participants in the UK received roughly half the intended dose of their first shot (the vaccine requires two shots at least a month apart).
A mistake sounds bad, doesn’t it?
The mistake violated the protocol that stated how the experiment should be performed, but the experiment continued and the data from these volunteers was included in the analysis. According to a report in the New York Times, vaccine developers noticed the bug – which was the result of a third-party work – during the ongoing process, but consulted regulators and were given permission to move on.
But the mistake caused a happy accident, didn’t it?
Yes. The volunteers who received the faulty first half dose had a higher level of protection against Covid-19 than those who did not. The “full dose” vaccine was 62 percent effective, but the half dose was 90 percent effective. That is the source of the 70 percent average and the idea that the vaccine could be adjusted to get to 90 percent.
Is there a “but”?
There are. Since the dosing error was against the protocol of the experiment, there are reasons to doubt the total number. The details are mathematically difficult, but can be traced back to the fact that a much smaller number of people received half the dose than the full dose, which can result in the sample size becoming too small and “under-challenging” the study.
As Hunter explains, the mistake means that statisticians had to rely on a “subgroup analysis” which may make the experiment too small to draw firm conclusions. It is known that subgroup analysis – especially when introduced post hoc and not as part of the original study protocol – increases the risk of finding false positives, he says. However, he emphasizes that there is nothing suspicious about the methodology.
Oxford and AstraZeneca say all results are statistically significant but the figures have not yet been fully released.
Is that all?
No. Another problem is the fact that the results come from data pooled from separate studies as they were conducted under different protocols. The UK study was a combined phase 2/3 study while the Brazilian study was a pure phase 3 study. They also used different placebos.
Such protocol variations are not necessarily a problem, but they can be. The data needs to be handled carefully to account for the differences, says Hunter. There are established statistical methods for this. “I’m not as concerned about this as I am about the unexpected subgroup analysis,” he says. But here too, details of the methodology were not published.
The dosing error increases this potential problem. This only happened in the UK and only to some of the volunteers, adding to the methodological discrepancies between the two studies.
Have the full results been published in a peer-reviewed journal?
No. The results of the UK study were published in The Lancet, but the full combined results were only published in a press release that did not include detailed figures. Oxford University said the full results would be presented to a journal. On Thursday, AstraZeneca announced plans for a new, global test.
Is it unusual too Press release results this way?
Not for these types of studies, says Mark Toshner, Clinical Studies Doctor at Cambridge University. He is doing any of the UK parts of the study but does not work for or receive funding from AstraZeneca or Oxford University. He’s not involved in whether the vaccine works or not.
The proposal to withhold data collected in multiple Covid-19 vaccine trials is unfair, he says. In fact, it is a legal requirement to avoid insider trading. “The moment you have an interim analysis and a positive result, you have to publish this positive result in order to avoid misconduct,” says Toshner – hence press releases rather than detailed data and analyzes, which costs time.
“It is the responsibility of the scientific community to review the data, but guessing the data a second time before you’ve seen it doesn’t build trust and undermines the scientific process,” he says.
Are there any other concerns?
There are reports now confirmed by AstraZeneca that the people in the half-dose subgroup were, on average, younger than the participants as a whole, suggesting that the 90 percent figure may be exaggerated. Vaccines generally work better in younger people, Hunter says.
Neither AstraZeneca nor Oxford published full information on the age profiles of the volunteers in both studies, with the exception that they were all at least 18 years old.
Another wrinkle is that the volunteers were all healthy or had “stable underlying diseases”. This may mean the results do not reflect the vaccine’s performance in vulnerable groups such as people with serious health problems.
Should we ignore this vaccine trial?
“Absolutely not,” says Hunter. Even if the vaccine is actually only 62 percent effective, that is above the 50 percent threshold considered acceptable by the World Health Organization and the US Food and Drug Administration. If this result had come out three weeks earlier, we would have been over the moon, says Hunter. It’s just that we’ve been spoiled by good news from other countries. Several other vaccine studies reported effectiveness numbers of 90 percent or more.
Sign up for our free Health Check newsletter for a round-up of all the health and fitness news you need to know every Saturday
More on these topics: