This is one of my older cartoons. Unfortunately, the problem it’s depicting hasn’t gone out of style. But now it strikes me…
Every time a drug company press release of vaccine trial results blasts through the media, it’s followed by criticism of “science by press release”, and people calling for peer-reviewed journal publications instead of press releases – or at least preprints. That’s a cycle we’re going to go through, over and over. While I was writing this post about it, the work was interrupted by a doozy of a press release (Oxford/AstraZeneca’s)! (I detoured into writing a post about those results at WIRED.)
I don’t think that press releases on these phase 3 Covid-19 vaccine trial analyses are inherently a problem. However, that view seems to go against the grain in my circles. The press release scenario can be challenging, but I have 3 reasons for my position, and I think they’re important. If you want to read a well-argued article making the opposite case to mine, check out Julia Belluz’s on “why vaccine science by press release has to stop“. (Disclosure: I’m quoted in that one.)
1. We need radical transparency about these vaccines for critical reasons, including for countries still considering doing big deals with manufacturers: Releasing readouts of interim results are part of that.
Public health physician and leader Ashish Jha wrote in Time that we need “aggressive transparency…unprecedented transparency” through the development process of these vaccines: “We need the companies making the vaccines, and the researchers studying the data, to act differently”, he wrote. Journalist Mark Sumner called for “radical transparency”, to ensure public trust. And so did drug discovery expert, Derek Lowe:
The companies developing vaccines for this pandemic have to break with past behavior and “normal” drug development and be more public about these trials than they have ever been. These are unprecedented times and we need to see some more unprecedented behavior. Fast.Derek Lowe, In the Pipeline, Science blog
That was at the time there was lack of transparency about trial stoppage for an adverse event. That remains a problem. While I was trying to finish this post, a previously undisclosed serious adverse event from mid-October on a trial for the Oxford/AstraZeneca vaccine in India was getting media coverage there. That’s because the trial participant’s lawyer announced they were taking legal action. For a couple of days now, though, there has been silence from the companies and regulators on this, in India and internationally.
Serious adverse events happen in life – so of course they can be completely unrelated to the vaccine. And reports so far say this one wasn’t deemed vaccine-associated internally. But if there isn’t radical transparency, and international drug regulator investigation, without the need for whistle-blowing, it’s going to seed endless conspiracy theories about the safety of that one vaccine, and then, indeed, all vaccines. The traditional route of waiting till there’s a peer-reviewed journal article to say how many serious adverse events there were are over for trials this high profile.
When detailed protocols for several phase 3 trials were publicly released, talk turned to the interim analyses made by the Data and Safety Monitoring Boards (DSMB) in these event-driven trials. Event-driven means the trial is getting its key answer based, not on reaching a target sample size, but on reaching a target of events (in this case, people with Covid-19), with careful rules for determining every step of that process. People have often referred to these as “readouts”. Lowe wrote, “I would expect the companies involved to announce a positive result if they do make any of these interim analysis hurdles, though (wouldn’t you?)”
I did – and I want to hear if they don’t have a positive result, too! “Readouts” from DSMB analyses usually aren’t stand-alone publications. Even if they’re not the stage of results usually published in journals, though, those numbers give us an idea of whether the vaccine is on track to meet the efficacy targets set by the World Health Organization (WHO), the US Food and Drug Administration (FDA), and others.
I’ve seen several people argue that companies are obliged to release this information for reasons related to company regulation. I haven’t looked into that. The companies manufacturing vaccines come from different countries, with different rules presumably, so perhaps that is a factor for some and not others. But even leaving aside the question of whether, say, companies operating in the US must do it, in the current circumstances I think they should.
Each week, countries are making decisions about which vaccines to place vast wagers on. The supply of vaccine is going to be far less than the demand in 2021, and if they don’t buy in advance, there might be little left when they could use it. With the weight of the pandemic on economies, countries have to avoid wasting large sums of money more than ever. Those decisions need to be as well-informed as possible, as soon as possible. And I think this month, knowing there was light at the end of tunnel might have made a difference to many individuals’ morale, motivation, and decisions too.
Consider what it would have looked like otherwise, if there had been no press release, and companies also didn’t divert resources from the regulatory process to producing a journal-style publication. We would have known, for example, on November 20 that Pfizer had lodged an application at the FDA, and that it was scheduled for a hearing on December 10 – but we’d have no clue about the vaccine’s efficacy until the FDA released papers 2 days before that hearing. As it is, we heard the news that a Covid-19 vaccine could work better than we’d hoped a month earlier than that.
It’s a shame we hadn’t already developed a widely-accepted convention for reporting interim analyses in event-driven trials. But we hadn’t. Taken together with their previously published protocols, though, some press releases were very informative and confidence-inspiring. As I wrote in WIRED, though, for one vaccine, it most definitely was not.
2. It’s the reactions to the press releases that are the biggest problem – and that problem holds for preprints, journal articles, conference presentations…
It’s easy to point to uncritical hype and epic fails in media coverage of these “readout” press releases. But let’s face it, how good can it be, when even experts are rushing to opinions without taking time to get up to speed first? Firstly, it requires critical engagement with what’s in the press release. How many experts rushed straight into coming up with explanation for why a low-dose followed by a standard-dose had greater efficacy, without first taking the time to consider whether greater efficacy had even been plausibly established in the press release? Within a few days, AstraZeneca was acknowledging they needed to run a trial before they could justify the claim, but this was already obvious from the press release.
None of these press releases should be read in isolation: They all require context available from the protocols, clinical trial registers, and previous publications. But deep, careful reading of the material has been a problem since the beginning of coverage of vaccine results. That’s inevitable, isn’t it?, when we want instant hot takes. Back in July, I wrote about the inadequacy of coverage of adverse events from the early trials. Something characteristic had already set in for some, but fortunately not all, development teams: High level information in abstracts and article text that was at some odds with the information in the large appendices.
This last week shows that even experts aren’t reading the appendices. When some phase 2 trial results were published in a journal this month, I thought the big news was the trial protocol in the appendix: It had some bombshells in it, that were critical context to the press release that emerged the following week. Yet, you kept seeing people say with confidence that no protocol had been published. It had, but you needed to be paying attention to more than the things the vaccine developers were drawing our attention to. You could see people weren’t engaging closely with the press release contents either in the rush to say something fast. Consider experts, talking about “the trial” as though the data came from 1 trial here, and as though they were from 3 trials here. It was 2 trials, and the press release couldn’t have been clearer on that score.
I’ve gone into that in detail, not only because it’s frustrated me a lot this month in particular. But to illustrate my point. It doesn’t matter whether the source of information is a press release, a conference presentation, a journal article, or a preprint, there has to be critical, non-superficial engagement with the material. And peer review, unfortunately, doesn’t solve this problem. The only form of peer review proven to make a genuine difference is that by statisticians, and you can’t count on that always being done.
What’s more, many journal articles – and even some preprints these days – come with press releases. And the hype and attention-directing they do have more influence in media coverage than the articles themselves anyway. Research by groups like those led by Isabelle Boutron and Chris Chambers are showing you can improve media coverage by intervening in press releases. (See my post on trying to stop science spin.) But the only things that are really going to make a difference are widespread increase in skills at detecting spin and interpreting research results – and getting expert commentary from statisticians more often. Peer-reviewing publications isn’t enough.
3. The drug regulatory process needs to be at the forefront (and transparent) because there’s no time to waste.
It’s astonishing how many people suppose that the medical press is an important part of this process. It isn’t. It’s regulators who matter.Stephen John Senn (statistician, November 10)
I agree with Senn. The level of scrutiny that’s going to be done by people employed full-time to obsess over it is (a) the priority and (b) better than the level of peer review that will usually happen at a journal – especially if the journal is fast-tracking. Take the FDA, for example. They are re-analyzing all the raw data for their vaccine evaluation, and they have 150 people working to do it in a few weeks.
If the vaccine developers could juggle it all – preparing applications for approval around the world, answering regulators’ questions etc – and still put together preprints at the right time without slowing that process down, great! It’s a critical part of acceptance for a lot of people, so eventually those journal publications are necessary too.
But those publications are always only part of the picture, not the final word. You still have to check other trial documentation. Ben Goldacre and his team ran a project called COMPARE, demonstrating that analyzing protocols and trial registry entries often show that the journal publication of a trial has cherry-picked outcomes. I think people want journal publications as a safety net. It’s not that I think they don’t often help a lot. But I think that whole “peer-reviewed journal publication” thing gives people a false sense of security – and since they’re critical to academics’ careers and culture, academics don’t question their assumptions enough.
Back to the current case of the interim analysis press-released by the Oxford/AstraZeneca team, and another leading statistician’s perspective, Natalie Dean:
In that Twitter thread, she points to an article she wrote with colleagues early in the pandemic, on a framework for clinical trials during a disease outbreak – and the unintended negative consequences publishing too-early results can lead to.
When it is time for a “publication”, then I think there really should be preprints first, instead of making everyone wait those extra weeks. (I’m looking at you, NIH/Moderna!!!)
Now a short break to insert a personal bugbear! We have a real problem with an enormous proliferation of places for posting preprints – and they may almost just as well be plonked on someone’s website, because they’re not going to be found.
I’m working hard to maintain a collection that’s open to the public of key documents related to Covid-19 vaccine trials that have published results and/or are in phase 3 trials. (I explain what’s in it and how to use it at the bottom of this post.) And when people don’t post their preprints on bioRxiv or medRxiv, or at least one of the services that Europe PMC indexes, there’s a high risk people won’t stumble across them. (And too often, believe it or not, it’s even hard to tell exactly which vaccine it is so that you can link them together!)
I don’t expect to have convinced everyone, of course, that the real problems here aren’t press releases per se: It’s the quality of them, and inadequate critical, deep engagement with them. That’s not a new problem. This pandemic, though, has provided an information overload stress test, too.
In June, I adapted my cartoon on confirmation bias, with a tweet alert that in this pandemic, an actual study was no longer required to activate scientists’ confirmation bias. I’ll end this post with a further edit to take account of the central role of press releases. And another critical one: It’s almost as though vaccines were national sport teams at the moment! On top of that, there are people whose fervor in bigging up one vaccine and dissing all others must surely be a signal that they have invested in shares. So be careful out there, people: The threat level for bias on vaccines and vaccine trial results is extreme!
[Update, December 6] Added sentence on FDA’s evaluation based on subsequently published information.
Disclosures: My only interest in Covid-19 trials is as a person worried about the virus, as one of my sons is immunocompromised. I have worked for an institute of the NIH in the past, but not the one working on the vaccine (NIAID). More about me.