Values and Conflicts at Journals

Image credit

Image by Hilda Bastian, CC BY-NC-ND 4.0

What makes an argument ‘correct,’ or, at least, reasonable? We can debate about each other’s claims endlessly, and there’s an interdisciplinary field focused on the forms of logic we use when do it. It’s called argumentation theory. Seemingly intractable ongoing controversies with lots of public back and forth offer people who work in this field plenty of fodder for developing their own methods and arguments.

Sally Jackson has studied randomized trials and Cochrane reviews in this way, as well as controversies around ME/CFS. She recently published a paper that crosses all those streams at once, analyzing threads of post-publication critique of the Cochrane review about exercise and ME/CFS. Jackson’s ethnographic case study raises issues about journal culture and post-publication criticisms that apply widely. (The study includes a comment for which I was the lead author. I’ve discussed the particular controversy Jackson is unpacking a few times at this blog, most recently a year ago, and six months prior to that.)

The corpus of data Jackson analyzed included all versions of the Cochrane review, 20 comments published at the Cochrane journal (plus author or editor replies), and the series of editorial notes and statements attached to the review. Those showed that Cochrane announced that the review was so problematic it needed to be replaced by a new version—then did an about-face, announcing that it was fine as it was. In addition, Jackson included Cochrane’s guidance about their reviews and commenting at the journal to ascertain the “official” norms relevant to Cochrane reviews and post-publication critique at the journal. She also read some documents linked in this material if they were needed to understand the arguments, and added some of those to the corpus.

Jackson classifies these post-publication discussions as an “argumentative polylogue” (a “dialogue” carried on by more than two parties). Her analysis is not a thorough summary of all that we are arguing about, and she is not always picking up on key points at issue when she discusses a thread. It is very useful, however, for how we’re arguing, and what this means for the journal and the Cochrane organization.

Jackson draws on Gilbert’s work to grapple with the norms and values underpinning post-publication discussion at the journal. That’s based on “interwoven components” of goal, context, and ethos. The ethos here, she points out, goes beyond concerns simply about the particular review. The impact of not dealing well with criticisms of a journal article, or not maintaining espoused standards, can ripple out across the networks of the people involved and beyond.

Thus, writes Jackson, “One plausible design aim for Cochrane’s debate space is to manage disagreements that might undermine trust in Cochrane or its products when released into this broader sphere of participation.” People interested enough to read the history of Cochrane’s actions with this review via the comments and editorial notes and statements would see what Jackson calls “a collapse” of Cochrane’s post-publication norms. “Why,” she asked, “would Cochrane put its reputation at risk in this way?”

Her conclusion was that Cochrane appeared to be more willing to cop the reputational damage associated with the problematical article and unresponsiveness to criticisms, than to risk reputational damage in another sphere: Reputational damage to their authors, which would have “eroded scientists’ willingness to trust Cochrane as a partner.” Jackson summarizes these clashing interests like this:

“Cochrane’s interest in maintaining the trust of deliberative bodies is about getting its products used (helping to justify whatever funding it receives from whatever source). Its interest in maintaining the trust of the scientific community is about being able to attract reviewers.”

I wasn’t just a commentator in this situation. I was an insider who knows more about what happened than is in the public domain. I think Jackson is right, but only to a point. Part of why I think so comes from the limitation of studying only one contested review—a limitation she stresses herself. Across the same time frame as the editors were making their final decisions in this saga, they were taking actions on their most controversial and high-profile review ever: A review dealing with masks, and an update to it in the height of the Covid pandemic.

In that case, Cochrane issued a statement criticizing the review and announcing they would make changes to it—without even discussing it with the review’s authors first. They were “throwing us under bus because of ‘pressure’,” wrote the lead author. The editorial about-face aspect featured here again: They changed their mind about making changes in the end. (A comment about all this submitted to the journal post-publication system by the author was not published.) Regardless of what you think about that review, protecting the interests of authors was not consistently privileged.

Across the period I was involved, Cochrane’s responses to controversial reviews were situational, and not in accordance with values, policies, or precedents. I agree with Jackson that the actions represented choices in the face of conflicting values. However that doesn’t exclude the possibility that editors’ conflicts of interests weighed in the balance as well. I think that raises a structural question for journals.

Years ago, Cochrane had an oversight committee for The Cochrane Library (the name of the journal). Its primary purpose was to provide “a mechanism for avoiding or resolving disputes about the content of The Cochrane Library or any editorial positions taken by the Editor in Chief in respect to the Library.” That is gone, and so is the role of Ombudsman. The role of the current Editorial Board does not include dispute resolution independently of the editor. Given the harm that this kind of situation can cause to whichever publics are affected by a journal’s contents and not just to a journal’s credibility, there really should be a trustworthy knowledgeable group to which disputes of this scale can be referred. Editors need independence from journal owners, but editorial authority should not be absolute. They are not infallible and impervious to conflicts of interest.

Jackson points out that ways of managing competing values in argumentation “must be faced not once, but over and over.” “Norms,” she writes, “live and die in actual practices, including argumentation practices.” Meanwhile, online forums, social media, and more are changing relationships between journals and the public. It’s more likely that critiques will be seen widely than in the old days. Quoting Jackson again: “Contemporary public argumentation is rife with still-new inventions and incomplete experiments that aim to improve societies’ capacity for reasonableness.” Improving journals’ capacity for responding reasonably needs to keep pace.

Cochrane continues to promote its reviews as “…the highest standard in evidence-based health care. We update Cochrane reviews regularly to incorporate new research, so that you can base treatment decisions on the most up-to-date and reliable health evidence.” Meanwhile, this out-of-date Cochrane review continues to mislead readers by stating that exercise is a treatment for this condition, recommended by treatment guidelines which in fact, do not recommend it. That could be resolved quickly and simply. If you need a poster child to show why the model of uncontestable authority of a single person and their support network at a journal is outmoded and extraordinarily risky, this is a good one.

You can keep up with my work at my newsletter, Living With Evidence. I’m active on Mastodon: @hildabast@mastodon.online and on Bluesky.

~~~~

Disclosures: My comments were part of the analysis in the case study I discuss. My thanks to Tom Kindlon for alerting me to the article. I was also appointed the lead of the Independent Advisory Group (IAG) for the review on exercise and ME/CFS in February 2020, with some financial support from 2020 to 2022, resuming in 2024.

I am an Emeritus Member of the Cochrane Collaboration. I was a member of the founding group of the organization and founder of its Consumer Network (1993). I was a member of its governing board (then called the Steering Committee) and Coordinator of the Consumer Network until 2001, and the editor of the quarterly Cochrane News from 1994 to 2000. I was the coordinating editor of a Cochrane review group for 7 years, involved in some Cochrane methods groups, and an employee of the Australasian Cochrane Centre. My PhD research was on some factors affecting the reliability of systematic reviews, including a focus on post-publication events (such as retractions and updates) and Cochrane reviews.

I have frequently written about the Cochrane Collaboration and Cochrane reviews, both in support and critically. Absolutely Maybe posts tagged ME/CFS are here, and all posts tagged Cochrane here. My ME/CFS-specific declaration of interests as at November 2023 is here. I keep my financial disclosures up-to-date here.

The cartoon is my own (CC BY-NC-ND license). (More cartoons at Statistically Funny.)