Skip to content

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

PLOS BLOGS Absolutely Maybe

What’s Happening When Living Systematic Reviews…Stop?

Cartoon about calling a living review that stopped updating a zombie

I’m not sure when the name “living systematic review” was coined, but it was fleshed out in an article in 2014 by Julian Elliott and co. The name was new, though the concept – staying on top of new clinical trials and keeping meta-analyses very up-to-date – has been around for over half a century. As Elliott’s article pointed out, it was the idea Iain Chalmers was immersed in back in the 1980s, making it a reality in care around childbirth for a while. And he excited a bunch of us to set out to achieve it across all of healthcare in the Cochrane Collaboration in the early 1990s.

I didn’t know it then, but the US Congress had mandated its National Cancer Institute at the NIH to do something similar even earlier. The NCI started with just a database of its own clinical trials in 1977, eventually expanding to all cancer trials. In 1980, they started summarizing the content for clinicians, and along the way created patient versions, too. In 1984, they moved to formal evidence assessment methods, and the project was named PDQ, for Physician Data Query. PDQ went online in 1995. As far as I know, it’s still the only large-scale non-commercial effort that’s pulled something like this off across decades.

The journal for Cochrane reviews was launched in 1995, too. We didn’t call them living systematic reviews, though that’s what they were meant to be. We just declared that all reviews “must be prepared systematically and they must be up-to-date to take account of new evidence.” Cochrane reviews, we said, would be “updated and amended as new evidence becomes available and errors are identified.” Ha! You can’t fault us for lack of ambition, that’s for sure!

Over a couple of decades, I tabulated how dramatically far we’d fallen short on that goal: Cochrane reviews that genuinely stayed up-to-date were the exception, not the rule. And no wonder: the list of hurdles this faces is longer than a gibbon’s arm. There are groups, though, from time to time, that succeed in keeping a systematic review over the long haul in a field with lots of clinical trial activity.

What’s different in the new iteration? New technologies are the key that make living systematic reviews a reachable goal, wrote Elliott. There’s a ton of interesting stuff going on in the LSR community. As with the OG Cochrane plan, I reckon a lot of good will come of it. And at least this time, they’re thinking through this how this ends. There’s Cochrane guidance on planning for the demise, and a process for moving in and out of “living” status. The jargon for that is “transitioning”. Cool! It’s a nice, feasible, orderly picture – the authors report on a final search and the review’s status change, and drop or add the word “living.”

I discussed one of the Cochrane LSRs in a talk I gave to a Cochrane meeting in April 2021. I wrote it up here a few weeks later, but the story has moved on since then.

To recap: Cochrane got some Covid LSRs underway when the pandemic struck. The one I’d been following was for convalescent plasma, and in my opinion, it’s an excellent review. They had their first version online in May 2020, and the evidence they gathered wasn’t strong enough to draw conclusions. Their first update followed quickly: July 2020. Again, the conclusion was, we don’t know if it works or not yet – still no trial. A second update followed in October 2020. Now there were 2 very small trials, too shaky to base a conclusion on.

Meanwhile, another group published a classic systematic review on the same question in July 2020. However, they also included trials for using convalescent plasma for non-Covid respiratory infections, and that was helpful. Their conclusion? “No convincing evidence of benefit,” with the possibility that it “has minimal or no impact.”

Then in January 2021, the picture had changed so dramatically, it meant the LSR’s conclusion on effectiveness was now definitely wrong. The thumping great RECOVERY Trial – with over 11,000 participants – announced the treatment definitely wasn’t working in that trial. A preprint swiftly followed in March. But the LSR stayed as it was until May 2021.

In April, it felt like the lengthening interval between LSR updates – and none since the previous October – was a bit of a worry. And in pandemic time, it felt like ages! But in hindsight, incorporating results from March into an update in May was pretty fast, given there was lots more than the RECOVERY Trial to add. Still, when it came to the question of mortality and the LSR’s new conclusion – “high certainty” of no benefit – that was, predictably, down to that single trial, which carried 96% of the weight in the meta-analysis. (See my explainer on what that means here.) In effect, the RECOVERY Trial made the LSR instantly redundant on the question of benefit.

Cartoon of a meta-analysis with a big trial throwing its weight around

If the result of that single trial hadn’t been so dramatic, though, the LSR would have been critical to making sense of how the evidence was traveling. And it still was super important for adverse event questions, as well as addressing the claims the treatment’s proponents made based on non-trial data. So I count this LSR as a solid example of the potential of LSRs. Although when there’s a whopper trial like RECOVERY in the mix, it’d be good to see a plan for how to react when it drops its massive weight.

What about LSRs outside Cochrane that stop? I’ve seen one with a final update named a “concluded living systematic review”. They won’t all end up so tidily, though. What are we going to call them when updates are long overdue, or they’ve apparently been abandoned? How “alive” do they have to be, to still justify the “living” label? It presumably won’t be all that long till plenty of so-called living systematic reviews sit in some kind of zombie zone – not living, and not “transitioned” either.

The first review named an LSR that I found on PubMed was in 2014. It might be the first zombie too. As of September 2022, its last update search was a little over 3 years ago. Then, the authors listed 1 trial from 2018 as needing to be added, with several others lined up for assessment. Using the links they provided for keeping up, you can see there have been more trials since. The question is still living, even if the review is not. From a quick look, there seem to have been at least another 4 systematic reviews or updates since the LSR’s last update search, too.

If only the evidence community was as good at pooling their efforts as they are at pooling data, eh! Back in 2010, there were at least 66 organizations internationally regularly searching and screening literature for clinical evidence – and about 10% of them running searches every month. In 2014, the NCI’s PDQ team were doing 200 searches monthly. It’s pretty mind boggling.

Can technology make a big dent in all this? As much as it could help make it easier, I don’t think technological assists can make up the ground lost by the colossal waste of human effort – or remove all the grinding tedium involved in constant surveillance for new evidence and absorbing it. It might give people a false sense of assurance that this can be achieved without NCI-level resourcing. Pooling resources and sharing data – and not just within organizations – seem critical to me. I still think that we need meta-collaborations for each stage of the process that involves intellectual effort, from the ground up, to keep up on a major scale. Meanwhile, I look forward to the benefits of having at least some living systematic reviews, and the spin-off advantages from the current wave of methods development. I expect we’ll see quite a lot of zombies, though.

~~~~

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny.)

Disclosures: I gave an invited talk at a Cochrane virtual conference on systematic reviews in the pandemic in 2021 (which I wrote up in a blog post here). I’m a consultant to Cochrane on stakeholder engagement for high-profile controversial reviews. I studied methodological aspects of shifting evidence that affect the reliability of systematic reviews for my PhD, including several studies of Cochrane reviews. I am one of the founders of the Cochrane Collaboration. I collaborated with the NCI’s PDQ team when I worked at another Institute of the NIH for a few years up to 2018 (the National Library of Medicine).

Leave a Reply

Your email address will not be published. Required fields are marked *


Add your ORCID here. (e.g. 0000-0002-7299-680X)

Related Posts
Back to top