What Does the PREDIMED Trial Retraction & Reboot Mean for the Mediterranean Diet?

A very influential nutrition trial just tanked. It was retracted from the New England Journal of Medicine (NEJM) on 13 June, and re-published with new analyses and toned-down conclusions.

Both Gina Kolata, writing in the New York Times, and Alison McCook, writing at NPR, imply, at least to some extent, that it might make no difference to the evidence. But I disagree.

Here’s what’s happened to the trial, and where I think it leaves the overall evidence.

Called PREDIMED, it was a multi-center trial from Spain, with the NEJM final report published in 2013. Altogether, 7,447 people at risk of cardiovascular disease (CVD) – heart attack and stroke – were reported as randomized to one of 3 groups:

• Mediterranean diet with free olive oil provided, along with individual and group training sessions at the start, and then quarterly;
• Mediterranean diet with free nuts provided, along with individual and group training sessions at the start, and then quarterly;
• Advice to reduce fat intake, with a leaflet – but after the first 3 years, people in this control group were also offered individual and group training sessions.

The primary endpoint for the trial was a composite one of major cardiovascular events: myocardial infarction, stroke, or CVD-related death. And the trial was stopped early. More people dropped out of the control group than the Mediterranean diet groups.

There are several alarm bells here already, and we’ll come back to those. (For some background, I’ve written before about the care you need to take with relying on composite endpoints – here – and trials that get stopped early – here and here.)

The road to retraction of PREDIMED didn’t start with those issues, though. It began with a piece of meta-research by John Carlisle in 2017. He listed PREDIMED as an example of a trial branded “randomized”, where the data didn’t look consistent with randomization to him. NEJM followed up – and the authors dug into their data.

It turned out Carlisle was right. The authors have now reported that randomization had gone awry for 21% of the participants – 1,588 of the 7,447 people in the trial. About a third each were for one or more of these reasons:

• When more than one person in a house was recruited, instead of randomizing each, they were all assigned to the same diet;
• At one site, the randomization table hadn’t been used properly; and
• At one site, clinics were randomized instead of people.

The authors have gone to a lot of trouble to re-analyze. And they say that even without these 1,588 people, the results are much the same. They tone down their conclusion because as the trial wasn’t properly randomized, the evidence is now weaker.

Unfortunately, though, I couldn’t find data showing what the impact of these 1,588 people were on the components of the composite endpoint. (The 3 components are all secondary endpoints, but I couldn’t find a sensitivity analysis for them – let me know, please, if you see them!)

Even for the full group, there was no statistically significant difference on myocardial infarction or CVD mortality – just for stroke. And in the supplementary information, there wasn’t a difference in the Kaplan Meier analysis for stroke either.

So what have critical systematic reviewers had to say about the quality and reliability of the PREDIMED trial, even before this bad news landed? And how strong is the evidence overall for the Mediterranean diet after this retraction?

The authors list a large number of systematic reviews/meta-analyses in their supplementary information, to support their certainty that health benefits for the Mediterranean diet are established. But they don’t report the basis for selecting that list. And there are, generally, more unreliable systematic reviews than very good ones.

I looked at 2 sources of reviews for a quick check: the systematic reviews listed in PubMed as citing the 2013 paper (n = 9) and cited in the HANDI assessment of the Mediterranean diet (n = 1). The systematic reviews listed with the trial in PubMed are those selected for PubMed Health with citations accessible. That means they are more likely to be higher quality systematic reviews – here’s a post from me of some rules of thumb on this. (Disclosure: In a previous role, I had recommended the providers of systematic reviews included in PubMed Health.)

Out of those 10 reviews, 6 of them were directly relevant and of acceptable quality. Only 1 of them was included in the PREDIMED paper’s list of systematic reviews/meta-analyses. (You can see a summary of all 10 reviews below this post.)

I think the strongest and most recent of these reviews were from organizations that specialize in systematic reviews: 2 from the Cochrane Collaboration and 1 from NICE. The 2 Cochrane reviews rejected the PREDIMED trial from consideration, even before these revelations, primarily because of taking issue with the control group.

The reviewers from NICE assessed the PREDIMED trial as at “serious” risk of bias for individual CVD outcomes, with “low or very low quality” data – and underpowered for mortality outcomes (even at the full complement). In addition, they wrote, the trial

was difficult to interpret because the control group was advised to reduce their fat intake and to follow some of the components of the Mediterranean diet.

Even before this retraction, the evidence base for the Mediterranean diet wasn’t really a slam dunk. There have been several trials, but they are riddled with problems – and what’s called “the Mediterranean diet” varies a lot.

So how does this evidence apply to you? Keep in mind, these people were at relatively high risk of CVD, with the men 55 and older, and the women 60 and older. So if you’re at less risk than that, these results don’t necessarily apply anyway. And we don’t really know what aspects of a Mediterranean diet might make a health difference to people from other countries.

If you are keen on trying to stick to a Mediterranean diet, HANDI has a guide here. It’s not lots of pasta and pizza and red wine! And speaking of red wine, check out this article by Julia Belluz in Vox, on the shifting evidence about even moderate alcohol use. If there does turn out to be an adverse effect to the popularization of the Mediterranean diet, it could be in the idea that the best diet includes regular red wine.

The Mediterranean-food-and-red-wine hypothesis is really appealing – at least to some social groups. But that could be a problem. If we’re not particularly careful with health claims that are attractive to us, there’s a good chance we’re going to be disappointed.

~~~~

Disclosure: I don’t believe I have any critical conflict of interest about the Mediterranean diet – other than a personal bias towards the health of Asian diets. I do eat pasta frequently, though, so I would be delighted if that was good for me!

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.) I borrowed the notion of there being no use crying over spilled milk, to represent the damaged trial, not just the trial. Here’s a piece on the interesting history of the spilled milk analogy.

[CORRECTION] The original version of this post mistakenly described the control group as participating in the same training sessions as the experimental groups after the first three years. However, they were only of the “same frequency and intensity”. Thank you to Angeliki Papadaki for pointing out the error.  

The systematic reviews I assessed are listed below: * indicates the systematic review is directly relevant and of acceptable quality.

Listed in PubMed as systematic reviews citing 2013 PREDIMED trial:

1. Kane (2017) (not listed in PREDIMED): cognitive outcomes, not CVD.
2. Palmer (2017 – Cochrane) (not listed in PREDIMED): kidney disease, not CVD.
3. * Bloomfield (2015 – VA) (not listed in PREDIMED): Mediterranean diet compared to other diets. For CVD mortality, they rated PREDIMED as “overall low risk of bias. Consistency is unknown and there was imprecision”. For overall mortality, “There is large imprecision and inconsistency, and overall risk of bias is medium”.
4. * Hooper (2015 – Cochrane) (not listed in PREDIMED): diets low in saturated fats, CVD: excluded PREDIMED because “Total fat goals in the low-fat arm were unclear and authors confirmed that aims were non-specific (if aims < 30%E this study would be included)”.
5. NICE (2014) (not listed in PREDIMED): obesity, not DVD.
6. * NICE (2014) (not listed in PREDIMED): lipids, CVD: PREDIMED assessed as “serious” risk of bias for individual CVD outcomes – all included studies categorized as under-powered for mortality, with “low or very low quality” data. In addition, “The most up to date study from Spain that examined ‘Mediterranean diet’ was difficult to interpret because the control group was advised to reduce their fat intake and to follow some of the components of the Mediterranean diet”. There was, they concluded, critical differences in what constituted a Mediterranean diet in included studies.
7. * Lara (2014) (not listed in PREDIMED): people “of retirement age” (54 to 70 years), used Cochrane risk of bias tool, but did not report individual results or take data quality into account in drawing conclusions. However, authors reported “None of the studies satisfied all of the criteria of the quality assessment tool”. Concluded evidence of effectiveness for the Mediterranean diet.
8. Benatar (2013) (not listed in PREDIMED): dairy food, PREDIMED cited but not included.
9. * Rees (2013 – Cochrane) (listed in PREDIMED): Mediterranean diet to prevent CVD. PREDIMED “did not meet our strict inclusion criteria as the comparison group was not minimal”…. “Low-fat diet arm had face-to-face nutritional advice as well as leaflets, therefore, not a minimal control”.

Cited in HANDI:

1. * Nordmann (2011) (not listed in PREDIMED): Mediterranean to low-fat diets, CVD. An author of PREDIMED is one of these reviewers, although did not participate in quality assessment. However, as PREDIMED was subsequently terminated early for benefit, the completed trial would not have met this review’s inclusion criteria. The authors concluded that “None of the included trials was powered to detect any differences in clinical outcomes between the 2 diets”. Quality assessment identified some problems. PREDIMED was classified as concealing allocation, but not reporting on whether outcome assessment was blinded; there was <10% missing data (but trial was ongoing); and “the method used to account for missing data remained unclear”.