PubMed is incredibly reliable. And a lot depends on it. It’s an ecosystem built around MEDLINE, the steady feed of new publications…
When Journal, Scientific Society, and Community Values Clash

A process I’ve been involved with at a journal recently exploded. It was meant to resolve a controversy about a publication, not make a new public conflict unavoidable. So I wanted to understand how it happened, and what it would take to resolve the problems we now face. At the heart of this rupture lie questions about unacknowledged competing values when a scientific society’s journal is struggling with openness and post-publication criticism. It’s hard to see how things can improve if they remain unspoken.
The scientific society I’m talking about is the Cochrane Collaboration, and its journal is its raison d’être. I was one of its founders. So I have a long list of disclosures at the foot of this post. The journal publishes systematic reviews, and there is just one of them on any specific question. The reviews are considered current unless otherwise specified. They are intended to influence health care practice, and they do. The controversial one in this saga is about exercise and people with ME/CFS.
The methods and scientific standards of Cochrane reviews have been evolving for over 30 years. The cumulative effect of this development is meant to increase the reliability of the reviews’ results. Cochrane reviews are expected to be updated, as the evidence base grows or the evolution of methods mean reviews no longer reflect current standards – and in response to important criticism. The organization describes them this way:
“Cochrane Reviews are internationally recognized as the highest standard in evidence-based health care… We update Cochrane Reviews regularly to incorporate new research, so that you can base treatment decisions on the most up-to-date and reliable health evidence.”
The organization has published a lot about its values. Its longstanding foundational principles, for example, include fostering “open and transparent communication and decision-making,” and “being open and responsive to criticism.” There are principles on consumer involvement, with assurances that “Cochrane will value its consumer volunteers,” and communicate effectively with them. There are lots of commitments, too, at the individual level for staff, the board, and members, to the foundational principles, and to accountability.
In that spirit, when a Cochrane leader asked me to help them “embed a patient-focused contemporary perspective” in a complete overhaul of the ME/CFS review, I agreed. That’s how I came to lead an Independent Advisory Group (IAG) of stakeholders. It is a privilege to get to work with this extraordinary group. There are people affected by ME/CFS, practitioners, and scientists, including several representatives of impressive national and international groups. A trio of us are systematic review experts with no personal connection to the condition, but with long and deep Cochrane leadership histories.
The project was supposed to take two years, starting in 2020. But then came the pandemic, and Cochrane turned a major amount of its attention to that. The organization took a major hit in funding and restructured, with massive implications. We had to go back to the drawing board to re-configure this project a couple of times. Advocates of the intervention launched a full-on bid to try to stop the project. Personal tragedy in my life slowed down progress of IAG activities for some time. In 2023, Cochrane was rocked by a major controversy over a pandemic-related review, and their handling of post-publication criticism of it.
In 2024, though, after all those delays and false starts, I thought I had reached an agreement with Cochrane on a detailed path to finish the project, and resolve major outstanding issues related to criticisms and complaints about unaddressed criticisms. Cochrane’s reluctance to respond to critics in accordance with its own policies had been an especially perplexing challenge, but I thought we were on track. Then this happened in the week before Christmas:
- The IAG was blindsided by a very brief email from the journal’s Office of Editor in Chief, with no signatory, telling us the update of the review was cancelled.
- The decision was announced publicly shortly afterwards, with a brief message indicating that complaints could be sent via the organization’s complaints webform, but correspondence to individuals at Cochrane “will not be considered.”
- Without notice, the history of the project, including the IAG’s public reports, was wiped from Cochrane’s website, with the pages re-directed to the public announcement. (It’s preserved, though, on the Internet Archive.)
- A day or so later, Cochrane issued a new 2024 version of the ME/CFS review, ostensibly implying an update, although the review was unchanged except for the addition of an editorial note. The note announced the update was cancelled, and Cochrane was standing by an outdated version of the review. It had previously acknowledged the review required a serious update, and there was a link to that. This note was issued by the commercial publisher, in agreement with the scientific society.
How do you get from a bold commitment to responsiveness to this? Let’s start with why there was an IAG and planned re-do of the review in the first place.
The review about exercise and ME/CFS has a long history of controversy, with deeply entrenched resistance to criticism of it. The last time it was fully updated was in 2015, based on a search for studies the year before. More than a decade since the last search in an area of active controversy is awfully long in the tooth. One “rule of thumb” has been that it’s likely a systematic review in health is outdated after 5 years. In the case of this review, it wasn’t just the usual issues changing across the decade. Understanding of the disease, and the intervention in relation to it, had moved on dramatically.
After 2015, more precise diagnostic criteria for this disease were adopted. In 2015, an Institute of Medicine report had designated post-exertional malaise (PEM) as the hallmark defining symptom that differentiates ME/CFS from other chronic fatigue conditions. That of course had profound implications for the approach to exercise for people with this diagnosis. In 2016 in the US, after an NIH meeting considered evidence presented by AHRQ, that agency added a statement to their systematic review pointing out that trials to that point had “a high risk of including patients who may have an alternate fatiguing illness or whose illness resolves spontaneously with time.”
PEM is more than you can imagine if you have only ever known normal fatigue, as Ed Yong eloquently explained at The Atlantic. Sharing people’s descriptions of how it feels, he wrote, “…where fatigue usually sets in right after exertion, PEM might strike hours or days later, and with disproportionate ferocity. Even gentle physical or mental effort might lay people out for days, weeks, months… It’s the annihilation of possibility.” [Non-paywalled link.]
The elemental shift of this diagnostic change in the ME/CFS field has gradually been reflected in clinical practice guidelines internationally. Now, exercise is no longer recommended as a standard approach, and there’s a stress on taking care to reduce the harm it can cause – by NICE in the UK, the CDC in the US, and in Germany, for example.
Meanwhile at Cochrane, from 2015 on, detailed and important critiques from people with ME/CFS began to be published through Cochrane’s formal post-publication commenting system, firstly from Tom Kindlon, and then in 2016 by Robert Courtney. They were largely brushed off. In one response, the authors wrote, “In the spirit of openness, transparency and mutual respect we must politely agree to disagree.” Ouch.
In 2018, Cochrane’s then editor in chief, David Tovey, stepped in, in response to a complaint by Courtney. A trail of notes in the review followed, including reporting on a rejection of authors’ revisions. A full update, along with a proper response to criticism, was needed, Tovey noted. Negotiations dragged on. I weighed in on this in a blog post here early in 2019, adding my voice in support of major aspects of the Kindlon and Courtney critiques.
To cut a very long story short, an amendment to the still-outdated review was published in 2019. I wrote a post here arguing the amendment helped, but was only a beginning. “It’s tough,” I wrote, “watching this play out slowly, step by painful step.” When the new editor in chief, Karla Soares-Weiser, announced the amendment, she also announced the full update, and later, the IAG. In March 2020, Cochrane announced the review’s authors had stepped down, and a new author team was being assembled. Now those authors have been dismissed and the update project cancelled, but even more unaddressed criticisms have accumulated at the journal. Where does all that leave this review, and what does this mean for Cochrane?
In the last few weeks, I’ve been analyzing those five years in the project, and what was happening at Cochrane during that time. I found that key aspects of my experience of Cochrane’s difficulty with criticism are not unique. Some of my expertise is specific to understanding problems like these. I have studied updating of Cochrane reviews for over 20 years, post-publication actions in systematic reviews are the focus of my PhD dissertation, and I worked for several years in post-publication issues for PubMed, at the US National Library of Medicine’s NCBI (NIH).
While the senior inside team has pulled up the drawbridge, I’ve spent hours in confidential and frank conversations with other senior Cochrane people. The IAG has been in intense discussions about next steps too. The first of these steps is an open letter to Cochrane. However, the views I express in this post are entirely my own.
I considered Cochrane’s decisions and actions in three separate categories: The new/updated review project; the editorial note on the current review; and a cluster of communication and accountability issues, around dealing with criticisms, complaints, and controversies.
The first of these three is the most straightforward. In cancelling the review project, Cochrane’s announcement attributed the decision to “insufficient new research in the field and a lack of resources to oversee this work.” That explanation sidesteps the fundamental reasons given for needing an update in the first place, and it’s misleading to imply the evidence and other content of this review is current. It’s not. What’s more, methods being out of date matters.
Of course, not wanting to commit resources to oversee staff members actually doing the update is fair enough – some were authors on staff time, too. That’s obviously their call to make. But if that were what this was primarily about, then there could have been a discussion about potentially going ahead in the normal way for a Cochrane review, without special resources from Cochrane to conduct it. It doesn’t explain Cochrane leadership’s salt-the-earth approach.
The second issue, the editorial note, is a critical one. Early in the project, the IAG gathered all the critiques of the review it could find, to ensure that the new authors could attend to them. Our content analysis identified over 100 criticisms. Consideration wouldn’t result in action for all of them. Indeed, some were contradictory. But en masse, the criticisms highlighted the consequences of outdated methods and bias in a particularly stark way.
Since NICE updated its review and guidance on ME/CFS, there have been calls for Cochrane to withdraw the review. In August 2023, for example, Science for ME (S4ME) launched a petition, now signed by many groups and thousands of individuals internationally. There was a story about this earlier today in Retraction Watch. Cochrane has made it clear that the review is staying. The journal used to withdraw reviews liberally, but in 2019 changed policy to align withdrawal more with retractions at other journals. When I last checked, no Cochrane review had been withdrawn since 2019.
Withdrawal isn’t the only mechanism for addressing the problem of an outdated review that won’t be updated. In late 2023, agreement was reached between the editors and the IAG that an editorial note to reduce problematic use of the review was appropriate while waiting for completion of the update. In 2024, the IAG recommended Cochrane issue an editorial note saying the review was out of date and should not be used, with information to support this recommendation. This has been done for other Cochrane reviews, including in 2021 on another ME/CFS review. It was a shock to see what they did in December instead.
Retiring outdated systematic reviews is standard in similar settings, like some health technology assessment agencies. Considering Cochrane’s principles and past practice, this shouldn’t be controversial. Saying something old is out of date doesn’t even cast judgment on a review. But choosing to remain an active player with a very old misleading review enables cherry-picking in a controversy when there are more up-to-date ones with different conclusions, as there are on this topic. That is antithetical to the goal of contributing to a well-informed community.
The promises Cochrane makes to the community about its reviews must supersede, for example, a concern about the impact on how the journal looks if people come across multiple “too old – do not use” notices. To be confident about the values driving these decisions, we need more transparency and accountability. The December note on this review was issued in the name of the commercial publisher (Wiley), in agreement with the society owner of the journal (Cochrane). The society is dependent on royalties from Wiley. Yet, the reputation of a medical journal depends on both the reality and the appearance of editorial independence from its owners and associated financial interests.
This brings us to my final category: How Cochrane is choosing to handle controversies, its contributors, and its critics.
When I summarized Cochrane’s communication strategy in December above, I mentioned that the brief corporate email sent to the IAG had no personal signatory. That’s because this is Cochrane’s policy now, even though faceless corporate communication runs counter to the principles of responsiveness and accountability. The organization brought in consultants in 2023 to develop this new approach, and more. Minutes of a governing board meeting say that to “depersonalize the process” of complaints handling, contributes somehow to making it “more effective and responsive to stakeholders’ needs.” When I asked if they had any evidence to support their new direction, they had none to pass on. Back in the mists of time, I was the Coordinating Editor for Cochrane reviews on the effects of communication, so that interaction quite pained me.
The signatory issue may seem to be a trivial point, but if I fully unpacked the contents of that previous paragraph, I would be making a case that Cochrane has taken the wrong lessons away from the controversies they have recently struggled through. And in December, their approach created a new one.
After all those months and years of delays, there was something particularly cruel about dumping all this on a patient community in the week before Christmas – and announcing that the people who are supposed to be accountable will ignore correspondence directed to them. This, while multiple properly lodged criticisms, and complaints about the lack of response to them, have remained unanswered for years.
It’s even more unacceptable for this particular patient community. As Yong has pointed out, “ME/CFS patients face extreme dismissal and disbelief” about their suffering, from many directions, and it hurts. “Societal dismissal of this kind,” he said, “leads to scientific neglect.” Cochrane is compounding this.
I quoted Yong’s description of PEM above, and what can cause it. As serial bad news came from Cochrane across that week, the distress and emergency emails and calls and meetings had predictable results for some: They crashed. Even Cochrane’s duty of care to the consumers it appointed to its IAG wasn’t honored. I don’t know how to put my sorrow and anger about this into words.
Back in 2016 I gave a talk I called “Openness and Consequences.” In it, I argued that the historical pressures on science have been moving it away from its origins in individualism and secrecy, towards collaboration and openness. But openness and accountability of journals is still far behind where it needs to be for science’s credibility. Thomas Südhoff articulated the need for more journal accountability in 2016 too. Nearly a decade later, these critiques still apply.
Many journals resist change, or eventually move with the times just a little, never really catching up. The Cochrane Collaboration had a head start, though. The journal’s fully electric origins in the early 1990s were already quite progressive in several respects. When members established an electronic system for criticisms, they worried, not about fending off critics, but about how to increase criticisms of Cochrane reviews. The Collaboration has long awarded an annual prize in honor of criticism of its methods or of the organization itself named for Bill Silverman, a champion of criticism “as a form of troublemaking that can help to drive progress.” Now, though, on the fundamental principle of being responsive to criticism, the Cochrane Collaboration is going backwards. It’s not just about one review. And that is more existential for this journal than it may be for others.
Unlike many scientific societies, this one doesn’t have a financial base in membership fees and commercial sponsorships. The journal doesn’t sell advertising. So its reputation and resulting influence are particularly dependent on the enthusiasm of its contributors. It cannot afford to burn through social capital acquired over decades. It needs to change course. The new conflict it has created around the ME/CFS review presents it with a valuable opportunity to do so.
Absolutely Maybe posts tagged ME/CFS are here. I have a “talk page” for the ME/CFS project here.
You can keep up with my work at my newsletter, Living With Evidence. I’m active on Mastodon: @hildabast@mastodon.online, and less so on Bluesky.
~~~~
Disclosures: I am an Emeritus Member of the Cochrane Collaboration. I was a member of the founding group of the organization and founder of its Consumer Network (1993). I was a member of its governing board (then called the Steering Committee) and Coordinator of the Consumer Network until 2001, and the editor of the quarterly Cochrane News from 1994 to 2000. I was the coordinating editor of a Cochrane review group for 7 years, involved in some Cochrane methods groups, and an employee of the Australasian Cochrane Centre. My PhD research was on some factors affecting the reliability of systematic reviews, including a focus on post-publication events (such as retractions and updates) and Cochrane reviews.
I am currently the lead of an Independent Advisory Group for a pair of Cochrane reviews on the HPV vaccine (a network meta-analysis [protocol] and a review on effects on community rates of related disease and vaccination harms [protocol]), with some financial support from 2021 to 2023. I was also appointed the lead of the Independent Advisory Group (IAG) for the review on exercise and ME/CFS in February 2020, with some financial support from 2020 to 2022, resuming in 2024. I am the spokesperson for the IAG. I have a talk page that I began for responding to questions about the Cochrane ME/CFS project.
I have frequently written about the Cochrane Collaboration and Cochrane reviews, both in support (such as a series of 6 posts here on the controversy around a previous HPV vaccine review) and critically, including strong criticism of both of the reviews mentioned in this post (the controversial pandemic-related review, for example here and here, and ME/CFS, for example here and here). I have also participated in criticism of policy in recent years (for example, as a co-signatory of a letter to the BMJ). With this decades-long involvement, I have a considerable track record with many individuals, including staff and authors of Cochrane reviews. My ME/CFS-specific declaration of interests as at November 2023 is here. I keep my financial disclosures up-to-date here.
Back to the top of the article
The cartoon is my own (CC BY-NC-ND license). (More cartoons at Statistically Funny.)