DAN HIMBRECHTS/AAP/PA Images
Earlier this week, pharmaceutical giant AstraZeneca and the University of Oxford released positive results about their covid-19 vaccine, claiming that it was on average 70 per cent effective, and could reach 90 per cent efficacy depending on dosing. But now, several questions have been raised over the findings. What’s going on?
In what way are the results not good news?
As scientists, journalists and business analysts chewed over the figures, they noticed some potential problems with the way the trial was conducted, which cast some doubt on the reported results and could be a hindrance to getting the vaccine approved.
Advertisement
What sort of potential problems?
In a word, methodological. “There are lots of issues,” Paul Hunter, professor in medicine and expert on clinical trial methodology at the University of East Anglia, UK, told New Scientist.
Such as…
Rather than results from one large phase 3 trial, which is what we saw from trials of other covid-19 vaccines, these results were actually pooled data from two separate trials, one in the UK and the other in Brazil. And there was a glaring mistake in the way one of the two trials was conducted. Due to a laboratory error, some of the participants in the UK-based trial got roughly half of the intended dose of their first shot (the vaccine requires two shots at least a month apart).
An error sounds bad, is it?
The error violated the protocol setting out how the trial was supposed to be conducted, but the trial continued and the data from those volunteers was included in the analysis. According to a report in the New York Times, the vaccine developers noticed the error – which was the result of work by a third-party contractor – while the trial was ongoing, but consulted with regulators and were given the go-ahead to press on.
But the error spawned a happy accident, didn’t it?
Yes. The volunteers who received the erroneous half-dose first shot turned out to have a higher level of protection against covid-19 than those who did not. The “full dose” vaccine was 62 per cent effective but the half dose 90 per cent. That is the source of the 70 per cent average and the notion that the vaccine might be tweaked to hit 90 per cent.
Is there a “but”?
There is. Because the dosing error violated the trial’s protocol, there are reasons to doubt the overall numbers. The details are mathematically difficult, but boil down to the fact that a much smaller number of people received the half-dose than the full dose, which risks making the sample sizes too small and “underpowering” the trial.
As Hunter explains it, the error means that the statisticians had to rely on “subgroup analysis” which may render the trial too small to draw firm conclusions. Subgroup analysis – especially when it is introduced post-hoc rather than as part of the original trial protocol – is known to increase the risk of finding false positives, he says. But he stresses that there is nothing suspicious about the methodology.
Oxford and AstraZeneca say that all the results are statistically significant, but have yet to release the numbers in full.
Is that all?
No. Another issue is the fact that the results came from data pooled from separate trials, because these were performed under different protocols. The UK one was a combined phase 2/3 trial whereas the Brazil one was a pure phase 3. They also used different placebos.
Such variations in protocol aren’t necessarily a problem, but can be. The data has to be handled carefully to take the differences into account, says Hunter. There are established statistical methods for dealing with this. “I am not as worried about that as I am about the unexpected subgroup analysis,” he says. But, again, details of the methodology haven’t been published.
The dosing error magnifies this potential problem. It only happened in the UK and only to some of the volunteers, which further widens the methodological discrepancies between the two trials.
Have the full results been published in a peer-reviewed journal?
No. The results of the UK trial were published in The Lancet, but the full, combined results have only been revealed in a press release which didn’t include detailed numbers. The University of Oxford said that full results would be submitted to a journal. On Thursday, AstraZeneca announced plans for a new, global trial.
Is it unusual to press release results in this way?
Not for these kinds of trials, says Mark Toshner, a clinical trials doctor at the University of Cambridge. He is running one of the UK parts of the trial, but doesn’t work for or take funding from AstraZeneca or the University of Oxford and doesn’t have a stake in whether the vaccine works or not.
The suggestion that data has been withheld – which has been levelled at several covid-19 vaccine trials – is unfair, he says. In fact, the approach we have seen is a legal requirement intended to avoid insider trading. “The minute you have an interim analysis and a positive result, to prevent malfeasance you have to get that positive result out there,” says Toshner – hence press releases rather than detailed data and analysis, which takes time.
“It is the scientific community’s obligation to scrutinise the data, but second-guessing the data before you’ve seen it does not inspire confidence and undermines the scientific process,” he says.
Are there any other concerns?
There have been reports, now confirmed by AstraZeneca, that the people in the half-dose subgroup were younger on average than the participants as a whole, which suggests that the 90 per cent figure may prove too high. Vaccines generally work better in younger people, says Hunter.
Neither AstraZeneca nor Oxford released full details of the age profiles of the volunteers in both trials except to say that all were 18 or over.
Another wrinkle is that the volunteers were all healthy or had “stable underlying medical conditions”. That may mean that the results don’t reflect how the vaccine will perform among vulnerable groups such as people with serious health problems.
So should we disregard this vaccine trial?
“Absolutely not,” says Hunter. Even if the vaccine is really only 62 per cent effective, that is higher than the 50 per cent threshold deemed acceptable by the World Health Organization and the US Food and Drug Administration. If this result had come out three weeks earlier we would have been over the moon, says Hunter. It is just that we have been spoiled by good news from elsewhere, with several other vaccine trials reporting efficacy figures of 90 per cent or more.
Sign up to our free Health Check newsletter for a round-up of all the health and fitness news you need to know, every Saturday
More on these topics:
