Saturated Biases: Where the AHA Advice on Coconut Oil Went Wrong
Even for an evidence food fight – and those get plenty heated – the hue and cry about the new American Heart Association (AHA) advice on coconut oil has been quite something. “We advise against the use of coconut oil”, they wrote, and boy, it was ON.
A quick recap in case you don’t know what the fuss is about. In 1961, the AHA called for us all to swallow less saturated fat to reduce our risk of cardiovascular disease (CVD – heart/blood vessel disease and stroke). By the 1970s, whole milk consumption in the US was plummeting.
The diet and economies of developing nations big on palm and coconut oil – both saturated vegetable fats – changed course. The backlash against fats helped fuel the increase of carbs that in turn contributed to rises in obesity and diabetes – both of which increase CVD risk.
In 2013, the AHA took a stand again: in favor of “a dietary pattern that….includes…low-fat dairy products,…nontropical vegetable oils…” and a belated warning about carbohydrate increase as a potential adverse effect – with no specific shots fired at coconut-eaters. But along the way there were some giant plot twists: coconut-everything became a trendy “superfood” – and the “butter’s not so bad after all” camp grew and got public traction.
So why the new statement? Because, they write, “discordant conclusions” from meta-analyses have “created confusion among patients, their physicians and the public”. (Explainers I’ve written about meta-analysis are here and here.)
I’m going to list 4 reasons I believe the AHA gets it wrong on coconut oil. They are really dimensions of a single problem, though: not applying careful and transparent scientific methods to get on top of bias. In a hot zone of conflict, that’s a recipe for more conflict and confusion, not less.
Of its 24 pages, the AHA report devotes more than 1 to disclosing conflicts of interest of the members of the writing panel (but not others), and 6 pages to references. But there’s only half a page reporting on their methodology of choosing, analyzing, and drawing conclusions from those references, addressing very little of what we need to know. Impeccable credentials can’t substitute for strong scientific methods and reporting. I think we’re all particularly biased when it comes to what we eat and drink, and it’s a notoriously difficult topic to get reliable research results on, too. We all need guardrails when we barrel down these roads.
Systematic reviewing is a painstaking, methodical process you go through to try to minimize your biases in assessing evidence, including meta-analysis. The US National Academy of Medicine has standards for developing trustworthy guidelines that apply those principles. The GRADE Working Group does too, including how to minimize bias in going from evidence to recommendations. (*Disclosure: I was a member of the GRADE group when these were developed.) Following a rigorous process along these lines would at least have led to a more transparent report.
1. Inadequate search and method of choosing studies
The abstract of the AHA report states that they discuss “the scientific evidence, including the most recent studies” and make their conclusions taking into consideration “the totality of the scientific evidence, satisfying rigorous criteria for causality”.
They are asking us to take their word for it here, because they don’t tell us how they tried to determine the totality of the evidence and find recent studies. This means the adequacy and potential bias in finding and choosing studies can’t be fully assessed. We can check the adequacy of their results, though.
For their coconut oil section, they don’t cite any study outside a single review, which includes evidence up to the end of 2015 (Eyres, 2016). The Eyres review has a very specific remit, so it only covers a patch of the questions related to coconut oil and CVD, not all of them, and not all health issues. By virtue of its exclusion criteria, this review cannot support these claims in the AHA report:
Clinical trials that compared direct effects on CVD of coconut oil and other dietary oils have not been reported.
..coconut oil…has no known offsetting favorable effects…
I haven’t attempted a systematic search for relevant trials related to coconut oil and health, but I did have a quick and dirty look for ones on CVD. It’s easy to find some that aren’t included. In fact, there are more participants in the randomized studies I found that aren’t included in the Eyres review than there are randomized participants in it. These were:
- Cardoso (2015) studies people with coronary artery disease
- de Paula Franco (2015) tests coconut flour in overweight women
- McKenney (1995) studies coconut oil in people taking statins
- Enns (2015) studies health outcome measures in people with peripheral arterial disease
- Ganji (1994 and 1996), Assunand Shedden (2017) are trials with coconut oil supplements
That last one was published after the AHA panel was finished its work, which was sometime in 2016. [Update: a commenter on this post notified another recent one, Irawati, 2017.] There are at least 2 more on the way – in Canada and England. And then there’s another category of study I didn’t look for, that’s often raised as relevant: comparing the effect of medium chain triglycerides on metabolic measures, including blood pressure, like St. Onge (2008).
I checked to see if there had been an adequate search and method behind the choice of systematic reviews and meta-analyses in the main part of the report. For those reviewing trials, they refer to 4:
- Mozaffarian (2010, date of last search June 2009)
- Chowdhury (2014, date of last search July 2013)
- Hooper (Cochrane Collaboration) (2015, date of last search March 2014)
- Mensink (WHO) (2016, date of last search December 2013)
They cite the re-evaluation of a primary trial in Ramsden (2016) – but not the systematic review with meta-analysis that’s in that paper as well (date of last search, March 2015). [Update: See a critique of this in comment below.] I found another 7 in a quick search with a search date in 2009 or later, one from 2009 that didn’t report the date of search, and another published after AHA was finished. A systematic search could find more, but again, it seems there are more eligible studies in this category outside the AHA report than in it.
(Skeaff, 2009; Smart, 2011 which found no trials on its question; Stroster, 2013; NICE, 2014; Schwingschackl, 2014; Bueno 2015 and Mumme 2015 on medium chain triglycerides and weight loss; Pimpin, 2016; Hamley, 2017).
2. Not being equally critical of all the studies
It’s not easy to disentangle biases in trials, or get a grip on why meta-analyses are discordant. (An explainer from me here.) But at the very least, you need to subject each study to the same rules and standards.
That didn’t happen here, in 3 determinative ways:
- No systematic assessment of methodological quality of reviews and meta-analyses was reported and we can’t be sure it happened;
- The quality of only 2 of the 8 studies in the review of coconut oil were discussed, and only 7 of the 8 were mentioned at all;
- The same standards of methodological quality applied to trials on other questions weren’t applied to studies on coconut oil.
The AHA report calls the Eyres coconut oil review a systematic review – but the authors do not. It has some of the elements of a systematic review. If you call it a systematic review, though, then it’s a very poor quality one: pretty unfair, since they don’t claim to have done one. (I used AMSTAR and came to a 1/11 score, though you could apply the criteria loosely to get to 3 – still bad.)
One of the things the coconut oil review doesn’t do is systematically assess the methodological quality of the studies included, and then take that into account when coming to individual conclusions. And the AHA panel doesn’t either – although they exercised a lot of critical precision on studies responsible for others’ conclusions that conflict with the AHA.
In addressing other studies, the AHA sets this bar:
…a randomized trial of a food or nutrient must achieve a biologically meaningful difference in intake between treatment and control groups and sustain it for a long enough time to deliver a valid result.
Large randomized trials that went for years and are accepted by others were rejected as a basis for AHA decision-making in the main part of the report. Whereas, of the 7 coconut oil trials they accepted for decision-making, only 3 were classified by Eyres as randomized. The 7 studies ranged from tiny to small, and went for only days to a maximum of 8 weeks. (The eighth, to reduce the amount of coconut oil in a high coconut diet, went for longer.) As Eyres writes:
Much of the research has important limitations that warrant caution when interpreting results, such as small sample size, biased samples, inadequate dietary assessment, and a strong likelihood of confounding.
3. Incorrect representation of the results of the coconut oil review
According to AHA,
The authors [Eyres et al] also noted that the 7 trials did not find a difference in raising LDL cholesterol between coconut oil and other oils high in saturated fat such as butter, beef fat, or palm oil.
That one has showed up in a lot of coverage, and even headlines like “Nutrition experts warn coconut oil is on par with beef fat, butter”.
But that’s not what Eyres says. Firstly, 7 trials didn’t compare coconut oil to other saturated fats: only 5 did (3 randomized, 2 non-randomized). And there were differences: this wasn’t a meta-analysis with a single, summative and definitive result. (Just before this statement about the 7 trials, the AHA reports that one of them found butter raised LDL more than coconut oil: another did too.)
Eyres concluded that, on the question of the comparative effect on LDL cholesterol of coconut oil and other saturated fats, “the results are largely inconsistent”. I agree – they’re not consistent at all. But then, these studies are so small and biased, the real question is far more basic: do we know enough about the effects of coconut oil on CVD risk factors and other important issues to conclude anything at all?
4. Making strong conclusions based on weak evidence
The AHA’s main conclusions are based on 4 trials with the tail end of the latest one in 1971. Diet has changed so dramatically since then, that their analysis isn’t compelling. (See Nina Teicholz and Eric Thorn, Gary Taubes and Larry Husten for very critical discussion on this.)
On coconut oil, the AHA has taken a stand on very shaky ground with some major claims – as though they had a very strong systematic review of reliable research on all possible health consequences of dietary coconut oil. They don’t.
The people arguing the opposite – that coconut oil is so healthy you should try to use it every day – are also on shaky ground. The first time a “nutrient as disease prevention” idea took off in a major way was probably dietary fiber to prevent colorectal cancer. I wrote a post a few years ago about the history, why that didn’t work out as expected, and why that’s typical of these kinds of nutritional claims.
Preventing actual disease or reducing risk factors via dietary change is tough. Unbalancing a diet can go wrong quite easily, though. If you wanted to increase your coconut oil and did it by having a couple of piña coladas every day, for example, you’d have a whole new set of problems! Dietary changes usually come with unintended baggage.
On the other hand, the occasional piña colada or coconut cream curry in a healthy diet isn’t going to cause a heart attack or stroke.
It’s not at all clear why AHA singled out coconut oil for a specific analysis, but no other non-dairy saturated fats. I didn’t even know for sure what dietary coconut oil meant when I read this report. Do I eat a lot of it? Is it a common ingredient in foods that I don’t think of as coconut-y? I had no idea.
It really doesn’t look as though coconut oil is a big dietary factor on average in the US – and if this data is right, the amount consumed hasn’t changed much. That might not include coconut milk, but all non-dairy milk isn’t a large proportion of milk either (and it’s mostly soy and almond).
Maybe the coconut “craze” is more hype than behavior – and socioeconomically specific in a way that’s unlikely to match CVD risk. On the other hand, palm oil use has apparently soared and it would be hard to avoid: it’s about the food industry using it. And then there’s the bacon boom, and super sizing…
If we’re concerned about America’s big public health problems with saturated fats, the AHA’s coconut oil salvo is distracting us from what matters most.
Continued in Circling the Wagons, Science Style: AHA Saturated Biases Redux.
~~~~
On 29 June, I posted a comment on the AHA Advisory at PubMed Commons. I submitted the section of my comment on coconut oil to the AHA, together with a link to the PubMed Commons comment and this blog post. And I emailed the journal with a link to the comment and post. The journal replied that there was a 500-word limit to what the AHA would consider, and I should re-submit.
On 24 July, the lead author and chair of the AHA Advisory replied to my comment at PubMed Commons, acknowledging no error, and firing shots at me for statements I did not make. I replied at PubMed Commons on 2 August, with a blog post detailing the issues in dispute. (Shortened formal letter to the AHA pending.)
* Disclosures: I have no income or livelihood bias about saturated fats. But I’m saturated in plenty of bias nonetheless, even though I’ve never given much thought to coconut oil before this last week. Some of my family are deeply committed to veganism and environmental activism, and a friend is involved in the manufacture of vegan foods using coconut oil. I was a member of the GRADE Working Group when it was developing guidance on rating evidence and recommendations, and methods for going from evidence to recommendations. I was a member of the Australian National Health and Medical Research Council working party on guidelines for developing, evaluating and implementing clinical practice guidelines (1996-1998). I participated in developing the Cochrane Collaboration, and participated in summarizing the original Cochrane review on dietary fat for consumers that is discordant with the AHA. (My day job includes working on curating and disseminating systematic reviews and guidelines at PubMed Health.)
[Updates 29 June]: Working on a comment for PubMed Commons I realized I had left Assun
[Update 30 June 2017] George Henderson notified another relevant trial in a comment on this post: I added it to the post. Thanks, George! Note: In order to be consistent with which studies I include on this post, I decided on including any report of a controlled trial specifically related to CVD, that has cholesterol levels or more direct health measures as study outcome, regardless of the trial results – and to note here any reports of trials that I exclude. That meant that I added Ganji (1994), and note here one trial I did not include because of its outcome measures: LaBarrie (2017 [PDF]). I did not examine trials on non-CVD related health issues to check if they included CVD outcome measures, and I did not keep a list of them.
[Update 31 July 2017] I corrected the search date for studies in the Eyres review: I had overlooked that they updated the search at the end of the 2015. This post originally said it was 2013, which was likely to be out of date.
The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)
* The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal, and do not necessarily reflect the views of the National Institutes of Health or the U.S. Department of Health and Human Services.