3 Things Expressions of Concern Reveal About the Research Publication System
An editorial expression of concern is a way to alert readers to behind-the-scenes worries about the integrity of a publication. It emerged in 1997, from the influential International Committee of Medical Journal Editors, but was hardly ever used for years. Use is growing now, but they are still rare.
Melissa Vaught, Diana Jordan, and I have just published an article in Research Integrity and Peer Review about expressions of concern in PubMed/PMC, the major biomedical research database. We hunted through Google Scholar, Retraction Watch, and publisher and journal websites as well as PubMed/PMC. (Data at the Open Science Framework).
We found 320 publications that were affected – mostly in the last few years. Not many, is it? They represent a microcosm of the possibilities and problems with post-publication systems. Here are 3 problems that struck me, while looking for, and at, editorial expressions of concern – with some ideas for solving them. (They are my personal views only.)
1. Alerting readers to publication problems is taking too long.
From the point of view of people reading and acting on publications, if part or all of a paper is unreliable, the sooner we’re warned the better. Full investigation of potential misconduct takes months or years, though (see here, here and [PDF]). And even resolving errata when an author is pressing for them can take quite a while.
An expression of concern offers a solution to this problem. For articles that were later retracted, the expression of concern arrived on the scene months earlier: a median of nearly 9 months, with the longest nearly 3 years earlier. And they rarely need to be retracted: we only found 6 out of 300 – although there probably have been some more that have been issued and removed without leaving a trace.
There have been thousands of retractions and a vast number of slow-moving errata. And there are sure to be many publications where journals have serious concerns that could never be resolved. We shouldn’t be kept so completely in the dark – or for so long.
Solution? There should be a lot more editorial expressions of concern. (As well as improving response times to resolution of issues, of course.)
2. Many systems aren’t set up to handle rare research integrity events – and individuals don’t get much experience with them, either.
If you have never tried to do one of these studies, you mightn’t realize how hard it is to find some extremely serious notices. We wanted to confirm each single case and extract as much information about them as we could. It was often amazingly hard to find notices we knew existed.
Some journals’ content management systems just aren’t set up to accommodate these rare events. They could be buried away in tiny print somewhere on a screen. If you were encountering that article at the journal, there’s not a snowball’s chance in hell you would notice it. And there are retractions and expressions of concern wedged into an erratum or a letters to the editor section, so you could easily overlook the seriousness of the event.
The rarity also means that the people who will be handling these events along the publication chain don’t have much previous personal experience of what to do – and most won’t have any experience at all. That’s a recipe for inconsistency, too.
Solutions? Publishers, journals, and content management system providers need to adapt to better accommodate post-publication events. And auditing rare events is essential.
3. Distributed responsibility for publication integrity and reluctance to act create large cracks that publications can fall through.
A lot falls between the cracks of journals, authors, authors’ institutions, and funder and other agencies with a responsibility for research integrity. Some of this is to do with stigma and concerns about liability. But there are conflicts of interest, too. No doubt there are other reasons for a reluctance to act.
A variety of issues on the sharing of investigation reports between journals and universities are being explored in CLUE – “Cooperation and Liaison between Universities and Editors”. Liz Wager and colleagues have a preprint out now for discussion on this.
However, many authors – and journals, too – aren’t responsive to concerns about major problems in publications. The jurisdiction of universities and agencies are limited. Public agencies can be subject to transparency requirements, but universities and journals? Not so much. There is a lot of room for issues to slip through the cracks between these structures and processes.
Many are worrying about, and working on, these “territorial” and transparency issues. Ginny Barbour and colleagues from COPE (Council on Publication Ethics) have a preprint out on trying to reduce the stigma, by re-branding. It seems to me, though, that re-branding isn’t going to change underlying attitudes.
Years ago, when I was part of a group starting a journal, we established an independent ombudsperson role: if you were an author or reader with a concern that editors weren’t addressing, there was somewhere for you to go. I did a stint a few years ago on the BMJ’s ethics committee – an impressive resource for the editor, that introduced a similar kind of independent dynamic into the process. Those processes convinced me that there is a critical role for independent mediators and arbiters to play.
I hope some of the developments to come address responsibilities more directly, with some institutions breaking ground in the direction of more transparency, independence, and action.
~~~~
Editorial expressions of concern are now prominently identified in PubMed/PMC, and are being progressively processed. You can search for publications affected by them with:
You can find editorial expressions of concerns themselves with:
The cartoons are my own (CC-NC-ND-SA license). (More cartoons at Statistically Funny and on Tumblr.)
* The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal, and do not necessarily reflect the views of the National Institutes of Health or the U.S. Department of Health and Human Services.
I completely agree that EoCs could and should be used much more often. They offer several benefits:
1) They alert readers much sooner to potential problems, as you point out. Journals should care more about their readers, who are often also their customers!
2) Pressure to resolve problems is applied to the authors, exactly where it belongs. Under the current procedure they have every incentive to drag things out; not so if they are trying to resolve an EoC.
3) Journals have less work to do. I’m a bit surprised that they haven’t realised the savings inherent in using EoCs.
I guess the problem is fear of legal issues.
I agree fear of legal issues fuels some of the reluctance to take post-publication action. For the US, at least, though, editors have been supported by a court for EoCs (discussed in our article). I was surprised too that they were so little used.