Should We Trust Meta-Analyses with Meta-Conflicts of Interest?

 

There are a couple of angles to look at researcher conflict of interest from. One is that a conflict could distort their work, tilting findings and claims away from “the truth”. The other is for the way the work is received, not how it is done: authors’ perceived conflicts could damage credibility.

How does this translate to authors of systematic reviews and meta-analyses? Are the issues the same, no matter the type of study?

I’ve been thinking about that a lot lately. I was one of the external stakeholders consulted as part of the Cochrane Collaboration’s review of its conflict of interest policy for their systematic reviews editorial teams. As they explain, they are looking to strengthen their approach to financial conflicts, and “consider a wider range of possible inherent biases”.

In biomedicine at least, systematic reviewers/meta-analysts are widely seen as arbiters on the state of knowledge. Their work often guides individual decisions, policy, and funding. I think that makes conflict of interest an acute issue for them – and credibility particularly critical.

There’s the usual string of potential conflicts of interest authors of any studies can have – like financial ones. But authors of systematic reviews and meta-analyses can have a type of potential conflict that’s totally meta, and unique to them: they could be an author of the very studies they are choosing and weighing up.

I think it’s problematic, and the systematic reviewing community has a blind spot about it. A serious analysis of non-financial conflicts of interest in systematic reviewing by Meera Viswanathan and colleagues, for example, didn’t even mention it [2014]. But there are several reasons to be concerned. The first is the possible impact on credibility.

 

 

In Brian D. Earp’s article, “The Unbearable Asymmetry of Bullshit“, he skewered the practice of

…hyper-partisan and polarized, but by all outward appearances, dispassionate and objective, “systematic review” of a controversial subject.

When people are reviewing their own studies, it’s got the feel of an echo chamber to me, instead of an objective analysis of a body of evidence. Is it really the people responsible for the research who are most likely to be able to keep an open mind, and see if an entire field has overlooked critical issues, or keeps making the same mistake? That goes to the core of one of the definitions of conflict of interest cited by the National Academy of Medicine (then called the Institute of Medicine):

A financial or intellectual relationship that may impact an individual’s ability to approach a scientific question with an open mind.

 

 

John Ioannidis and Peter Gøtzsche have written about experiencing this problem as co-authors of meta-analyses:

Primary authors are likely to defend their results and see the meta-analysis as an opportunity to advance their views.

From my point of view, author bias of this sort can just about scream out from a systematic review. Check out my previous post for an example. There, a single deeply flawed trial concluded there was an adverse effect of an intervention. It didn’t have much impact. Years later, the author participated in a systematic review on the subject, doing the quality appraisals of the included studies – concluding that his own trial was of high quality. Unlike the single study, the systematic review had a massive impact – even though the single study remained the only evidence for the adverse effect.

It’s tough, because I can see the value of researchers systematically reviewing to keep on top of the evidence to do their next study, for example. You can make a strong case for all researchers having to do systematic reviews, at least occasionally. And people who have done the primary studies being reviewed obviously have a lot of insider knowledge. Yet it seems to me that people are under-estimating how subjective many of the steps in systematic reviewing and meta-analysis are. And so they underestimate, in turn, the potential for being taken off course by confirmation bias or allegiance bias [PDF].

Allegiance hasn’t been studied anywhere near as much as financial conflicts of interests – and even financial conflicts of interest haven’t been studied as much in systematic review authorship. So we don’t know much about whether these anecdotes and theoretical concerns translate to serious bias in systematic reviews. One thing we do know, though: authors evaluating their own studies is common. And that can indirectly be a financial interest, too, if the systematic review helps them get funding for the type of study the review says is needed.

A 2016 study by Maram Hakoum and colleagues found that 9% of a sample of 100 Cochrane reviews disclosed that one or more of the review’s authors was also an author of a study or studies included in the review – and 15% had authors of relevant studies that were not included in the review. Cochrane authors are supposed to disclose this, but that’s unusual for a journal. In the Hakoum study, none of their sample of 100 non-Cochrane reviews disclosed authorship of the studies being studied.

Hakoum’s study only looked at disclosures: they didn’t trawl through the included studies looking for self-authorship that hadn’t been disclosed. Klaus Lieb and colleagues did that, though, in their 2016 study of 95 systematic reviews of psychotherapies. They found that 36% included studies authored by one or more of the systematic review authors – but this relationship was disclosed in only 2 reviews, which were published in journals with policies requiring it to be disclosed.

Lieb’s study rated the amount of spin in the reviews’ conclusions, too: they found it in 28% of the 95 reviews. There was some suggestion that spin was more likely when authors were reviewing their own studies [OR=2.08 (CI 0.83 to 5.18)], but we can’t know for sure without bigger and further studies in a variety of subject areas.

[Update 22 March 2020]  In 2020, Lorenzo Bertizzolo published a study looking at what is going on when more than 1 Cochrane review includes a trial, and the risk of bias is rated differently. They found that when an author is an author of both, the trial is more likely to be rated as at low risk of bias.

It would help, too, to have research on how often the reviewers’ conclusions were in line with those of their own studies in contested areas, for example, and whether their own studies get a critical enough assessment. We need more meta-research on this type of meta-conflict. The little we have suggests there could be a problem here. And there’s a big problem with lack of awareness and disclosure.

 

 

Viswanathan and colleagues argued that there are several options for managing non-financial conflicts of interest in systematic reviewing:

…disclosure followed by no change in the systematic review team or activities, inclusion on the team along with other members with differing viewpoints to ensure diverse perspectives, exclusion from certain activities, and exclusion from the project entirely.

Cochrane already requires disclosure, but that’s not commonplace for journals, and I think it should be. Aiming to make sure that the team overall has people without conflicts is a common approach to financial conflicts, for example – and it’s Cochrane’s too. It’s not clear, though, that it nullifies the impact.

“Exclusion from certain activities” seems to me to be the obvious minimum. If an author has done any of the included studies – or any that might be eligible – I think readers deserve to be sure that they had no role in choosing the studies, no role in determining the criteria by which quality will be assessed, and no role in assessing the quality. Come to that, what about choosing the outcomes, extracting data, and making recommendations about future research? The more I think about it, the more it seems to me that the conflict is too great to play much role in a systematic review at all. It certainly leaves a systematic review open to criticism of its objectivity.

This is Cochrane’s current policy on this [PDF]:

Cochrane authors who include primary studies (which they had conducted) in their review should declare this in the review in the Declarations of Interest section. The CRG should ensure that an editor checks the included data and interpretation against the study report and any available study registration details or protocol.

The CRG (Cochrane Review Group) is the editorial group responsible for the review. That’s a little analogous to a role that data monitoring committees play in clinical trials: mitigating “risk to trial credibility” [PDF]. Someone has to do it, though – as well as checking the assessments of quality or risk of bias of the studies. A single person wouldn’t be regarded as enough for a typical clinical trial: why is it for a systematic review? That said, why increase the workload like this, when stepping aside and letting others do the review is the safer course?

I’ve concentrated in this post on this issue that gets less awareness than financial conflicts of interest. But that doesn’t mean I think it’s more important, or that it was the sole focus of my contribution to the Cochrane Collaboration’s consultation. I came down in the camp of people who think that the policy on financial conflicts needs to be tighter, with more of a differentiation between degrees of financial interests. There’s a world of difference between holding a patent or working at a drug company, and speaking at a conference that’s pharma-sponsored, for example.

I also argued the policy needs to explicitly include consumer organizations under institutional conflicts, and that you need to disclose financial ties of the consumer organizations you represent – even if you aren’t an employee. The degree of commercial sponsorship of patient groups has become truly alarming, and we need to be more transparent about it at every level. (See here, here, here, and here.)

And I questioned the section of the policy on royalties of reprint sales: what is going on there? As Richard Smith pointed out back in 2005, this is a key way that journals have become “an extension of the marketing arm of pharmaceutical companies”. I don’t know what Cochrane’s practice is with reprints, so it’s definitely something that needs more transparency.

My final thought, though, isn’t about editorial practice or policy at all. Having dug deeply into some arguments about financial conflicts of interest among authors of Cochrane reviews in the last year, it’s striking to me that there’s another serious weak link. And that’s among researchers/authors.

People are going to hunt for authors’ undisclosed activities – even trivial ones. Researchers really need to keep a scrupulous list of everything they do that comes even the tiniest bit close to a conflict of interest, so that full disclosure is easy when the time comes. If you don’t pay close attention to how close you come to something that was commercially sponsored, write it down, and keep track of your disclosures, you’ve got to rely on racking your brains somewhere down the line. And that’s risky. Trust can become fragile so very quickly. Avoiding even the appearance of a conflict of interest is probably impossible. Full disclosure, however, should be an achievable goal.

 

~~~~

 

Update 29 April 2019: Explanatory sentence about CRGs added.

Update 10 August 2019: Link on studying financial conflicts in systematic review authorship updated – full review published. Original link had been to the protocol for that review.

Update 1 October 2019: Added to the disclosures that I drafted the first conflict of interest policy for the Cochrane Collaboration (which I had forgotten).

 

Disclosures: In February 2019 I was interviewed on the Cochrane Collaboration’s conflict of interest policy by Kirsty Loudon, Project Officer for the organizational policy review. I am one of the members of the founding group of the Cochrane Collaboration and was the coordinating editor of a Cochrane review group for 7 years, and coordinator of its Consumer Network for many years. I drafted the organization’s first conflict of interest policy. I am no longer a member, although I occasionally contribute peer review on methods or proposals, and participate in consultations. I don’t peer review Cochrane reviews because the Cochrane journal is not open access. In my day jobs from 2004 to 2018 (at IQWiG and then with PubMed projects at the National Institutes of Health), I worked to increase access to systematic reviews, including partnership with the Cochrane Collaboration. I was a co-signatory to this 2018 letter in the BMJ, challenging the organization’s approach to open science. In 2018, I wrote a series of posts about unfolding controversies in the Cochrane Collaboration (here, here, here, here, here, and here.) Some of my recent posts have defended and criticized specific Cochrane reviews (here, here, and here). I am currently working on a PhD on some issues affecting the validity of systematic reviews. I have never accepted funding from a manufacturer of a drug, device, or similar health product. More than 20 years ago, I received funding from a not-for-profit health insurer, and from a private health insurers’ association for participation in a conference.

 

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)