Let’s Be Serious About Bias in Protocols of Systematic Reviews

A detailed pre-specified protocol is supposed to be the superpower of a systematic review: its contents literally put the “systematic” into review. And it is expected to do an awful lot of the heavy lifting on minimizing bias. There is even PRISMA guidance that claims that the protocol “prevents arbitrary decision making with respect to inclusion criteria and extraction of data”, and an Agency for Healthcare Research and Quality (AHRQ) one that making a protocol publicly available at the start of a review will “prevent the effects of author bias”. I think we’ve been so complacent about the hypothetical bias-reducing properties of protocols, that we haven’t paid enough attention to bias in them.

I’ve been a bit pre-occupied with this issue lately. I’m involved in establishing a process to tackle a Cochrane review that’s been highly criticized, including by me. It’s about exercise therapy and Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS). And it means going back to the protocol for a comprehensive update.

The Cochrane Library‘s editor-in-chief, Karla Soares-Weiser, pointed out that the protocol is based on old methods as well as definitions of ME/CFS from the 1990s. These are important issues – the protocol was published in 2001 – but as we approach the starting blocks for a new protocol, it seems to me that the current approach to protocols doesn’t prevent the problems I see in that protocol either.

Of course, one of the points of pre-specifying exactly what you will do in a review is to actually follow through precisely. It’s not only possible, it’s even probable, that protocols won’t do that: meticulously following the pre-specified rules might be more the exception than the rule. Although whether this skews reviews’ conclusions isn’t clear, it’s not reassuring. Reviews with pre-published protocols haven’t been to shown to be decisively more reliable than than those without, as far as I can see. Which means the theories about the bias-preventing power of protocols may be more an article of faith than a reliable reality.

That doesn’t mean that the theory is wrong. You could minimize bias in a systematic review with a solid protocol that is only deviated from with equally solid – and reported – justification. It might be hard, though, to ensure that people believe it. After all, even when people say they believe a study will provide the answer to a question, if they don’t like the answer, some will furiously denounce the study as fundamentally flawed and doomed from the outset anyway. Once a systematic review is mired in as much controversy as this one about ME/CFS, developing an approach to it that doesn’t have one or all sides convinced the game is rigged is a colossal challenge. Short protocols thin on justification with closed peer review surely don’t help though.

One of the reasons I don’t think a lot of the current guidance on protocols goes far enough is that it’s often explicitly predicated on the idea that you should write it before you know what the evidence is, as though (a) that’s realistically the default scenario, and (b) that somehow guarantees impartiality.

I suspect that’s rarely, if ever, if the case. I don’t think you can get as far as writing a protocol without having any clue at all about what’s out there, even if you don’t know the entirety of it. Years ago when my job involved sorting out why systematic reviews on similar questions came to different conclusions, one of the things I found striking was how often systematic reviews were apparently undertaken to challenge the findings of another review. And I’ve written before about the problem of researchers doing systematic reviews where their own studies could be included.

Those are just a couple of the ways that a systematic review starts with anything but a blank slate. So developing a model of how a protocol should be developed based on a hypothetical that’s rather fictional isn’t a good starting point. Better, it seems to me, is to engage with the potential controversies around a body of evidence rather than pretend a blindness that’s not real. A protocol should show us why we can have confidence that good effort is being made to minimize bias in that specific review.

The protocol sets out the context in which the review is being conducted. It presents an opportunity to develop ideas that are foundational for the review… A key purpose of the protocol is to make plans to minimize bias in the eventual findings of the review.

Cochrane Handbook

One of the fundamental flaws in the practice of protocols at the moment seems to me to be the idea that it’s enough to simply report on the methods that are chosen. You don’t actually have to support those choices in detail. I think one solution to protect against bias and the appearance of bias, is to spend time in the protocol justifying those choices.

Another option is to plan sensitivity analyses on contested aspects – that’s a way of assessing whether looking at the data this way or that changes the results. I don’t know how often that’s done, but it’s an important methodological option.

All this requires of course, that authors are genuinely striving for rigor and fairness. People who use or are affected by the review deserve to have confidence that that’s what’s happening. A fundamental problem with the original exercise therapy and ME/CFS protocol, it seems to me, was evident author bias in the text about both the intervention and the condition. Some of the type of bias in that text is actually encouraged by the current structure and guidance for Cochrane protocols, and it shouldn’t be.

There is a section hard-coded into the Cochrane review software for the background of a protocol and the subsequent review called “How the intervention might work”. That’s inherently biased towards the assumption not only that it could work, but also that this is a bigger issue for consideration than harm it could cause. You have to be interested in the section on logical models – which aren’t typically used in Cochrane reviews – to ever get to a reminder that considering the potential harm matters.

Logic models are representations of how interventions are intended to ‘work’, but they can also provide a useful basis for thinking through the unintended consequences of interventions and identifying potential adverse effects that may need to be captured in the review.

Cochrane Handbook

The guidelines that operationalize the Handbook don’t go that far, leaving the obligation to consider harm a poor cousin to anticipated benefit. That’s obviously not the intention, but it is the result. And it’s emblematic of the relative superficiality of the current approach to the construction and content of protocols. For the ME/CFS protocol, there was no mention of the potential for harm in the description of possible effects of the intervention.

I think a protocol should include a justification of the methodological and analytical choices it presents. It is sometimes there – a citation that supports the use of a particular method, for example. Even then, though, that can just be a description of it, not evidence of its robustness or that it’s fit for purpose in this particular review. There can be discussion of methodological strengths and weaknesses of the review’s methods in the discussion section after it’s done, but surely that belongs in the protocol, too.

The protocol is also a place to discuss the approach to data openness. When I thought about what deepens my confidence in the rigor and fairness of a systematic review, the amount of access I have to check what was included and what was left out matters – even when I don’t actually double-check or re-analyze any of it. Detailed and open reporting that’s also well-synthesized is confidence-inducing.

Reading the guidance from various systematic review bodies offers several solutions that have the potential to help ensure robustness and credibility. One is to ensure that every stage of the process has hands-on involvement and oversight from people with methodological expertise who don’t have a stake in the review’s outcome. Another important strategy is the approach many public agencies use, described here by AHRQ:

For publicly funded reviews, the public should also have the opportunity to comment on the protocol to improve the acceptability and transparency of the SR process. The review team should be responsive to peer reviewers and public comments and publicly report on the disposition of the comments. The review team need not provide a public response to every question; it can group questions into general topic areas for response. The period for peer review and public comment should be specified so that the review process does not delay the entire SR process.

AHRQ Standards

One advantage of the Cochrane review is that any version can be updated, keeping the previous one available online, and that includes the protocol. So this is possible. What the AHRQ advises is a major effort and commitment, though, especially given the magnitude of the public response that’s likely for a controversial review.

I’ve engaged in these processes many times, both as a consumer advocate doing the commenting and being on the receiving/responding end at public agencies. It can be a complete waste of everyone’s time. But when it’s not tokenistic whitewashing, consultation can be powerful. The establishment of the Independent Advisory Group in this case is a commitment to genuine accountability and responsiveness: it provides a mechanism for establishing open discussion of the protocol, beyond the confines of peer review that’s outside public scrutiny. It’s too early to say what that will look like – the Group hasn’t even been fully established yet, although it’s on the way. I’ve made a start though, by gathering the critiques of this review from the past so we can respond publicly to those and feed into the protocol development process.

The origins of the word “protocol” are ancient, and represent important metaphors for what we need to do here. It’s made up of words for “first” and “glue”. A protokollon was a sheet glued into first place on papyrus scrolls. Because it was the outermost part of the scroll when rolled up, writes Keith Houston, “it received the brunt of any rough treatment at the hands of its readers”. It used to be blank, but eventually the waste of scarce space bothered people, and it was used to record details of what was to be found within.

Capable of withstanding rough treatment from readers is exactly what a protocol for a systematic review has to be. And glue is apt for what should follow if a systematic review is to be trustworthy: the reviewers should stick to the protocol, with transparency and very solid justification for any time they don’t. We should be far more serious about how to achieve all of this. It’s critical to confidence in a systematic review’s integrity.

~~~~

The views expressed in this post are my own, and do not necessarily reflect the views of the Cochrane Collaboration or anyone else involved in the update of the Cochrane review on exercise and ME/CFS.

Disclosures: I am one of the founding members of the Cochrane Collaboration, and was a member of the organization’s governing body that established the system for protocols of Cochrane reviews. I have contributed to the Cochrane Handbook on occasion since then, including peer review of a section of the current version unrelated to protocols. I was a co-author of PRISMA for abstracts, but was not involved in the development of PRISMA for protocols. I lead the Independent Advisory Group currently being established for the update of the Cochrane review on exercise and ME/CFS. I have written blog posts criticizing that review, here and here: see the disclosures in either of those relevant to the issue of exercise and ME/CFS.

The cartoon is my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)