Skip to content

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

PLOS BLOGS Absolutely Maybe

Systematic Reviews & Meta-Analyses: A 5-Step Checkup


It’s easy to be a little blinded by the specialized statistical techniques in systematic reviews and meta-analyses. As with any type of study, though, there are bad ones that can lead you down a garden path.

So let’s look at some simple – or at least less-than-hellishly-complicated! – ways to spot reviews that are worth taking more seriously than others.


Cartoon: An ounce of prevention is worth a pound of miltivariable linear regression


You don’t start with the statistics. No amount of fancy statistical legwork can save a study that’s biased by its design. That’s true of individual studies, and it’s true of meta-studies that analyze groups of them, too.

A bonus of this phenomenon is that even without knowing a lot about methods of analysis, you can whittle down the number of reviews and meta-analyses that could cause you problems.

But first, some shortcuts that I don’t think are a reliable help.

Many argue that there is a “brand” that should give you confidence. I don’t agree (even though I participated in establishing some of those brands).

And don’t believe it must be good just because it says it was “systematic”, or something like “followed PRISMA (or Cochrane) methods/principles”. That alone is no guarantee that the quality is higher. (See for example this study on quality and mentioning PRISMA).

A last proviso before we start: a perfectly well-done study can land on the wrong answer. And a perfectly crappy study can nonetheless “be right”. Still, we really need time-savers, and steering clear of worse studies is a way to cope with an information glut.

Let’s get to my 5-step check-up. Most systematic reviews and meta-analyses won’t pass all 5 of them.


1. Are there clear, pre-specified, eligibility criteria for studies being chosen or rejected for the review?


Ideally, the review makes it quick and easy for you to answer this one, with a really clear list of inclusion and exclusion points – or at least a clear description of the grounds for accepting or rejecting a study. The criteria might be extremely broad, or very specific.


Cartoon: Does it work? It depends what you mean by does, it, and work


The easiest way to be sure details like this were pre-specified is if there was a fully worked out study protocol before the reviewing steps began. That might have been published in a journal, or in a registry like the one specifically for systematic review protocols, PROSPERO. Or they may mention that they had one. Sometimes, the language is the best guide you have to go on.

If there are very clear criteria, but I can’t be sure if they were pre-specified, I tend to give them the benefit of the doubt at this point, but it’s a point for caution.

While it’s a good sign, having a protocol is not enough. You can totally rig a protocol to get you where you want to go. And as with anything else, just because you said you would do things, doesn’t mean you did them – or did them well.


2. Did they make a strong effort to find all the studies which could have been eligible?


Cartoon ancient statue of Greek statistician


Once you have a clear question, and a clear idea of what studies are eligible, then you need a good search strategy to find them.

There should be a good search strategy. That’s an organized plan of what databases and other sources are being searched, and with what terms. It might be in an appendix. A rule of thumb is it should search, at the very least, in more than one place. The less detailed and organized this is, the less confident we can be that this review has really captured the relevant research.


3. Were the searches done within the last 5 years?


Cartoon of document pile-up


This one is a rule of thumb, too. In some areas, even a year ago might be too late, if the field is moving fast. If the only review is an old one, then it might still be useful. There’s a pretty high chance that there’s been a game-changer since, though.

Tip: the date of the review’s publication and the date of the search could be very, very different – the contents of a review can even be years older than the article in which they are published.


4. Can you see a list of the studies that were excluded from the review?


It really helps if there is a flowchart, something like this one:


Diagram showing number of studies found and sifted through at each review stage


(This flowchart one comes from a systematic review I participated in.)

But while the flowchart is good, looking to find a list of the rejected studies is a good shortcut for assessing the transparency and thoroughness of a review.


5. Have they given you some indication of how good they think the studies they included are?


Cartoon about assessing study quality


I’ve written more about this over at Statistically Funny (and here, too, in looking at meta-analyses in psychology). There can be quality problems that affect the whole of a study – for example, whether the researchers assessing outcomes in a clinical trial knew what comparison group a patient was in. And there can be quality problems with a specific question or measurement within studies, too.

You need, though, to know how much weight can be put on the studies in a review or meta-analysis. If you can’t do that, then you really have to go back and look at all the studies yourself, pretty much.

There are many different ways that people could have done this, depending on the kind of review and study types involved. For example, they could assess quality, risk of bias, or particular hurdles a study has to clear. The quality criteria are sometimes also entry criteria for the review. In a case like that, just being included already tells you something.

What’s more, the review’s conclusions should take the quality of the evidence into consideration. This isn’t just a numbers game.

Want to go into this more deeply? The most commonly used method for formally assessing the quality of a systematic review in health care is AMSTAR. Another is ROBIS. Julian Higgins and colleagues developed a checklist for assessing the quality of a meta-analysis. The GRADE Working Group have a range of methods for assessing quality of evidence, including one for assessing syntheses of qualitative research. And Sally Thorne has proposed some interesting questions for qualitative meta-syntheses, too.

Once you’ve gotten as far as a meta-analysis you want to make sense of, check out my tips for understanding data in meta-analyses.


Happy sifting!



The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)


* The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal.


  1. Isn’t the elephant in the room reporting bias? In other words unpublished studies and relying on journal articles (which are summaries of studies) for your summaries?

    You can have the most rigorous methods (fancy stats or not) but if it relies on published journal articles it has the potential to be heavily biased.

  2. “It’s easy to be a little blinded by the specialized statistical techniques … No amount of fancy statistical legwork can save a study that’s biased by its design.”

    Well, good statistical legwork can often lessen the bias while poor statistical legwork can increase the bias of an ideal RCT to be worse than that of a poor observational study. Ignoring the reality of needing fancy enough (but not too fancy) statistical methods is just poor science. Unfortunately, though I am sure your post did not mean to suggest that pretty much any statistical methods will do, it may very well to some readers.

    Part of the larger problem in discerning what to make of multiple studies may be an overemphasis of the synthetic to the (complete?) neglect of the analytical role that statistics can play.

    Unfortunately this is encouraged by the debatable but common definition of meta-analysis as (just) statistical synthesis That is, the idea that statistics is about getting weighted averages of the study estimates.

    Analytic roles for statistics focus on “contrast[ing] results from different studies and identify[ing] patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies”

    For instance, investigating heterogeneity rather than just estimating it and extrapolating from the idiosyncratic sample of studies done rather than just getting an idiosyncratic average effect. In the latter, perhaps assessing the _quality_ of studies done and extrapolating to better studies.

    Here by quality is simply meant as whatever leads to more valid results. Some may find the word offensive and may wish use risk of bias though I think is becoming understood that qualitative only assessments are not adequate.

    I will close by agreeing with Jon Brassy, given what’s available or not from published journal articles there is a limit to fancy enough statistical methods and often simple methods take you about as far as you can get. Unfortunately that often is not far but highly uncertain.

Leave a Reply

Your email address will not be published. Required fields are marked *

Add your ORCID here. (e.g. 0000-0002-7299-680X)

Back to top