It shocks me when a professional group circles the wagons and starts firing shots at people who point out undeniable errors in their research.
I appreciate, at least a little, what it’s like for insiders in these processes. I’ve been there dozens of times, including having chaired the development of national guidelines* in a hot button area. Once upon a time, I was even among the people sued by some companies because of a recommendation that didn’t go their way. I know what it’s like to feel besieged or bombarded.
Regardless of how heated others’ rhetoric becomes, though, when it comes to their scientific claims, I believe major organizations’ spokespeople have to be able to acknowledge error, and maintain scientific rigor in responding to methodological criticisms. I don’t think the author response to my comment on the 2017 American Heart Association (AHA) position paper on dietary fats managed to meet that standard.
This is a cloud with a silver lining for me, though! It provides an excellent example for one of my favorite things to do: discussing methods for detecting bias in research – and how to try to minimize your own biases while doing it.
Unfortunately, there isn’t really any way to completely extinguish bias. And even if it were possible for research to be totally free from any bias, people can’t be. What’s more, it’s probably easier to make mistakes when you’re dealing with lots of studies than not to.
When reviewing evidence, the best way we know to deal with these problems is to start by setting clear and explicit ground rules for every step – using methods that are validated and widely used rather than inventing new ones if at all possible. Then, at least two people undertake key steps, cross-checking with each other to try to reduce error and increase consistency. And finally, it’s all reported in enough detail for others to assess. Someone else should be able to reproduce what was done, which means they need to be able to see how A led to B.
These days, clinical practice guidelines are expected to meet this standard too. The National Academy of Medicine calls for guideline development processes to be systematic, explicit, and transparent to minimize biases.
The recent AHA Presidential Advisory on dietary fats does not demonstrate that level of rigor or transparency. I wrote a blog post about why I think this made their recommendation on coconut oil unreliable, concluding:
On coconut oil, the AHA has taken a stand on very shaky ground with some major claims – as though they had a very strong systematic review of reliable research on all possible health consequences of dietary coconut oil. They don’t.
The people arguing the opposite – that coconut oil is so healthy you should try to use it every day – are also on shaky ground.
The AHA statement doesn’t have enough detail on data, and the goal posts seem to shift around inside the report. The goal posts veer around in the author response. For those playing along at home, I’ll explain how I’ve gone about trying to gauge the strength of the arguments in that response. (There’s a lot of detail – if you would like a quicker take on the issues, check out my original post.)
Firstly, though, I’ll lay out the errors I mentioned so you can see why I, at least, think they are undeniably errors of material fact. These errors result in an exaggeration of the strength of the evidence.
There’s only 1 source reported for studies on the effects of coconut oil on people’s health. It’s this, a review by Eyres and colleagues. Those authors were looking for studies of coconut oil consumption on cardiovascular risk factors, particularly lipid profiles – not health effects generally. And they included a slice of evidence up to the end of 2015 (although apparently less thoroughly after 2013). They selected 8 clinical trials meeting their criteria, only 5 of which compared coconut oil to other saturated fats. There was no meta-analysis. The Eyres team categorize the body of evidence this way:
This review identified a limited amount of human research on which to assess the merits of coconut oil or coconut products in relation to cardiovascular health. Much of the research has important limitations that warrant caution when interpreting results, such as small sample size, biased samples, inadequate dietary assessment, and a strong likelihood of confounding. There is no robust evidence on disease outcomes, and most of the evidence is related to lipid profiles.
Here are direct quotes from the AHA statement on the issue of coconut oil versus other saturated fats, starting with a discussion of 2 of the studies included in the Eyres review – emphasis from me:
A carefully controlled experiment compared the effects of coconut oil, butter, and safflower oil supplying polyunsaturated linoleic acid. Both butter and coconut oil raised LDL cholesterol compared with safflower oil, butter more than coconut oil… Another carefully controlled experiment found that coconut oil significantly increased LDL cholesterol compared with olive oil… A recent systematic review [Eyres] found 7 controlled trials, including the 2 just mentioned, that compared coconut oil with monounsaturated or polyunsaturated oils…The authors [Eyres &co] also noted that the 7 trials did not find a difference in raising LDL cholesterol between coconut oil and other oils high in saturated fat such as butter, beef fat, or palm oil.
As there was no meta-analysis of these trials, there is no single estimate to discuss. Of the 5 that did include a comparison with saturated fats, there were differences among their results: the AHA had pointed out 1 of them just a few sentences previous to their “no difference” statement. As there aren’t 7 trials showing equivalent effect on LDL cholesterol for coconut oil and saturated oils, of course Eyres didn’t “note” this. Here’s what Eyres wrote:
In comparison with other fat sources, coconut oil did not raise total or LDL cholesterol to the same extent as butter in one of the studies by Cox et al., but it did increase both measures to a greater extent than did cis unsaturated vegetable oils…[W]hen the data from the 5 trials that directly compared coconut oil with another saturated fat are examined collectively, the results are largely inconsistent.
Basically, coconut oil could be worse than other saturated fats, could be the same, could be better – we need stronger evidence if we want to know.
The AHA author response doesn’t include an acknowledgement of error in their description of what Eyres and colleagues found and wrote.
Next issue: was there enough evidence in the Eyres review to support the AHA’s broad conclusions?
I had a quick look to see if there was more recent evidence, and relevant cardiovascular trials that did not fit in the scope of the Eyres review. It was a plausibility check of whether the Eyres review was likely to cover the cardiovascular turf. I found a bunch. So I concluded that the AHA did not have a basis for these statements:
Clinical trials that compared direct effects on CVD of coconut oil and other dietary oils have not been reported.
[Coconut oil] has no known offsetting favorable effects.
That last statement isn’t limited to favorable cardiovascular effects. You would need rigorous up-to-date systematic reviewing of all health effects to address both of those questions. Eyres isn’t that.
I have no idea if there are any benefits of dietary coconut oil. I’m not hazarding a guess without an adequate systematic review, either. I just know that enough people think there might be health benefits, that they do trials related to weight loss, diabetes, and cognitive impact, and there are at least 2 more on cardiovascular issues on the way – aiming for 24 people in Canada and 90 in England.
The AHA author response provided no additional information on what they actually did. And it included a few shots fired at me, like this one:
Bastian employs a tactic in common with some other critics of good nutritional science, namely, to a) disparage and misrepresent high quality studies that show harmful effects of saturated fat; b) promote and misrepresent seriously flawed and irrelevant studies that report the opposite; and c) cite meta-analyses with faulty designs often based on inclusion of flawed studies.
In my comment, I had pointed to the conclusions of the Eyres review about the weaknesses of the body of trials they looked at, agreeing with that assessment. The AHA response:
Bastian implies that this systematic review is composed of weak studies…There is no objective scientific reason to disparage them.
Hm. Does the AHA working group genuinely think the 8 trials in the Eyres review are sufficiently strong to answer questions about all possible non-LDL-cholesterol health effects of coconut?
When the AHA described their reasons for rejecting trials on quality grounds on a different issue in their statement, they included some of the same weaknesses that concerned the Eyres team – like inadequate dietary assessment. Although Eyres does not regard the evidence as strong, the AHA author response re-states this strongly (although it gets the number wrong):
[T]he 8 trials in the Eyres et al. systematic review provide strong evidence that coconut oil increases LDL-C levels compared with unsaturated oils.
(Only 7 of the trials had such a comparison.)
The response then goes on to look at the examples of studies I cited to substantiate my conclusion that there isn’t a broad-ranging systematic review available to come to a conclusion about health effects – and implies that I am claiming something about their results. Over 400 words are spent describing why some of the studies I pointed to strengthened their conclusions – and why those that don’t support their conclusions can be dismissed. We’ll come to that shortly. But since that’s moving into territory where we don’t have a systematic review to support us, I’ll illustrate the issues involved in assessing study quality using systematic reviews in another part of the AHA statement.
The heart of the AHA position on dietary fats overall revolves around trials AHA selected from 3 systematic reviews. They didn’t leave a list of criteria and paper trail for how and why they chose those 3 systematic reviews: Mozzafarian (2010), Chowdhury (2014), and Hooper (2015). (If you’re interested in this, I recommend having a look at Table 2 in a review by Steven Hamley that shows more than 3 systematic reviews of trials, and all the trials they include.)
Here are the 4 they regarded as the most rigorous/reliable. They named them “core” trials:
- Dayton (1969) (“Veterans Administration”)
- Leren (1970) (“Oslo”)
- Medical Research Council (MRC) (1968)
- Turpeinen (1979) and Miettinen (1983) (Finnish Mental Hospital Study)
In the table below, I’ve translated the judgements about the quality of those 4 trials made by the authors of the 3 systematic reviews. All 3 of these systematic reviews used formal and validated methods to do this: it’s a basic standard expected of a systematic review.
The reviews used different assessment methods. Mozzafarian used a simple score that looks at a few issues around randomization and so on (the Jadad score). They considered studies scoring 3-5 as high quality, and studies rated 0-2 as low. They have scored each of the 4 “core” trials either 2 or 3. If Mozzafarian scored it 3, I’ve called it “higher”; the 2’s got “lower”.
The other 2 reviews reported risk of bias ratings using the Cochrane method, which rates categories of bias, but not the study as a whole. I mapped “higher” quality here as a high risk of bias rating in at most one category. One trial was excluded by a review (Hooper) because it didn’t meet a design threshold they had set for adequacy.
|Systematic Review||Included Trial 1||Included Trial 2||Included Trial 3||Included Trial 4|
Some of the trials the AHA rejected are rated more highly by others than some “core” trials. Check out the risk of bias rating table in the Hooper review (Figure 2), for example.
For some of the criteria the AHA considered deal breakers, systematic reviewers made a different judgment call, in addition to that exclusion of a study.
For example, the Hooper review gave the MRC trial an “unsure” on completeness of outcome data and whether a reduction in saturated fats was actually achieved in the trial. They also concluded there was a high risk of systematic differences in care other than the intervention between the groups – a concern Hooper & co expressed about the Oslo trial as well. In other words, if others applied the criteria for “core”, a different set of studies would probably be chosen.
So let’s get back to the AHA author response to my comment. They rejected the judgment by Eyres (and me) that there were weak studies in the Eyres review. When they turned to the 8 trials I had cited to show that the AHA’s broad statements couldn’t be assessed based on Eyres alone, a range of criteria for judgment emerge. (Note: There are 8 trials in 7 papers.) There are 3 clear exclusion criteria for controlled trials in this statement in the AHA response:
[A]mong the 7 studies…4 would appropriately be excluded as result of being non-randomized, uncontrolled, using a very small amount, not including a control group or not even being a trial of coconut oil.
I don’t really know what to make of “uncontrolled” and “not including a control group” as 2 criteria, given all these trials are controlled: the final 3 that aren’t rejected don’t make it clear to me either.
These are the 3 criteria I derive from this, for a controlled study to be regarded by the AHA authors as valid enough for consideration:
- Uses “enough” of a dose of coconut oil (but a threshold for enough isn’t given).
- A separate coconut oil group.
The author response says that a small trial can be adequately powered for a conclusion on effect on LDL-cholesterol – but they don’t say how big or how long it needs to be, so I can’t apply a criteria we both agree on there. However, LDL-cholesterol isn’t the only outcome to consider on the question of any possible favorable effects, so I consider it a moot point.
Out of the author response and those in my own comment, I picked criteria to extract data from these 2 groups of 8 coconut trials (“Eyres” and “non-Eyres”). I picked ones that I think are clear and fairly objective, given I’m the sole data extractor (but let me know please if you spot errors!). Here they are:
- Randomized: yes or no. I didn’t know what to do about cross-over trials, given the fuzzy description of controls in the AHA author response and that one cross-over trial made it through their sifting. So I called them “yes” if they were randomized and noted that (“X-over” below).
- Number of people participating.
- Length: this is the time they were supposed to be adhering to the diet. (I do not note washout periods in cross-overs or run-in times.)
- Separate coconut arm: yes or no.
- Comparison of coconut against unsaturated fat (“Non-SFA” below): yes or no – a comparator group had to be only unsaturated fat to be yes.
- Comparison of coconut against saturated fat (“SFA” below): yes or no – a comparator group had to be only a non-coconut saturated fat to be yes.
- An outcome other than measurement of blood lipids: yes or no (called “health” outcome below). Note: this included body weight or BMI.
So….what happens if you apply these objective criteria to the 2 groups of trials even-handedly? Remember that not being randomized and not having a separate coconut arm are, according to the AHA response, knock-out criteria for non-Eyres trials. If so, then no Eyres-included trials should fall on those knock-out criteria if the AHA authors are being even-handed.
[UPDATE on 3 August] Matt Hodgkinson drew my attention via Twitter to another trial of coconut oil: Vijayakumar (2016). It’s the largest, longest so far. It’s been added in italics* below, to make it clear it was not part of the AHA author response.
The 8 trials in the Eyres review
|Number of People||
|Coconut vs Non-SFA||
Coconut vs SFA
|Cox (1995)||Yes (X-over)||28||3 x 6 wks||Yes||Yes||Yes||No|
|Cox (1998)||No||41||3 x 6 wks||Yes||Yes||Yes||No|
|Mendis (2001)||Yes||54||52 wks||Yes||No||No||No|
|Mendis (1990)||No||25||2 x 8 wks||Yes||Yes||No||No|
|Ng (1991)||No||83||3 x 5 wks||Yes||Yes||Yes||No|
|Reiser (1985)||Yes (X-over)||19||Varied (up to 3 x 5 wks)||Yes||Yes||Yes||No|
|Fisher (1983)||No||9||4 x 9 days||Yes||Yes||No||No|
|Voon (2011)||Yes (X-over)||42||3 x 5 wks||Yes||Yes||Yes||No|
The 9* trials outside the Eyres review
|Number of People||
|Coconut vs Non-SFA||
Coconut vs SFA
|Ganji (1994/1996)||Yes (X-over)||10||4 x 7 days||Yes||Yes||No||Yes|
|McKenney (1995A)||Yes (X-over)||11||3 x 6 wks||Yes||Yes||No||No|
|McKenney (1995B)||Yes (X-over)||17||4 x 6 wks||Yes||Yes||Yes||No|
|Assunção (2015)||Yes||40||12 wks||Yes||Yes||No||Yes|
|Cardoso (2015)||Yes||114||3 mths||Yes||Yes||No||Yes|
|de Paula Franco (2015)||No (X-over)||42||3 x 3 mths||Yes||No||No||Yes|
|Enns (2012)||Yes||50||8 wks||No||Yes||No||Yes|
|Shedden (2017)||Yes||42||8 wks||Yes||No||No||No|
|Vijayakumar (2016)*||Yes||200||2 yrs||Yes||Yes||No||Yes|
Things which really stand out here:
- The Eyres group has fewer randomized trials: 4/8 compared to 7/8 in the non-Eyres group (or 6/7 for non-Eyres after knocking out the trial with no separate coconut arm). [New total with additional trial: 4/8 vs 7/8.]
- There are fewer randomized participants in the Eyres group: 143 compared to 234 in 6 non-Eyres trials with a separate coconut arm. [New total with additional trial: 143 vs 434.]
- All the trials in the Eyres group only look at blood lipid profiles whereas most in the non-Eyres group assess at least 1 non-blood-test outcome (5/8 or 4/7). That is in part because of the Eyres exclusion criteria (such as rejecting any trial in a specific population or clinical subgroup, such as overweight people). [New total with additional trial: 0/8 vs 5/8.]
- The Eyres group tends to be smaller and shorter than the non-Eyres group. [The largest longest trial is now the additional “non-Eyres” trial: it is the only coconut trial that is long enough to meet the duration criteria the AHA set for a “core” trial.]
It’s important to keep 3 additional things in mind:
- I only did a quick search for trials. I was only assessing the validity of the claim that “the totality of the evidence” had been considered, and whether or not there was non-lipid evidence at all. The 16 studies here are not all the trials that have been done. [17 as of 3 August: there may still be others.]
- I specifically did not collect trials of dietary coconut oil for non-cardiovascular health effects.
- You can’t just apply some kind of vote-counting on these results. (More on why that gets you into hot water here.) There is no study by study risk of bias assessment reported (by Eyres, the AHA, or me), and no appropriate meta-analyses that take the studies’ precision into account.
Where does this leave us? If the AHA authors applied the same level of critique even-handedly to the Eyres group as they did to the non-Eyres group in their response to me, much of the evidence they currently accept on coconut oil would fall by the wayside. That wouldn’t leave them with much of an evidence base at all.
Would that – and acknowledging the basic error they made on communicating about coconut comparisons with saturated fat – change their conclusion? That’s impossible to know. The circle-the-wagons-and-fire response to my comment so far doesn’t induce confidence. (Although I hope that will change: I’ll keep you posted on how this goes.)
Specialist groups resisting adapting in response to evidence on issues of existential importance to them is a common problem. To minimize the chances of this happening, the National Academy of Medicine has detailed recommendations about how societies should evaluate evidence, and how they should guard against conflicts of interests – not just financial ones, but intellectual ones as well. The AHA has enough resources to adhere to them. It should.
Having valid and solid goal posts for assessment and decision-making processes also makes disputes after publication much easier to manage. But either way, dealing scientifically with criticism without circling the wagons needs both skills and willingness.
Rigorous, transparent, and responsive processes all the way down the line are difficult and resource-consuming. But many people are affected by statements like these. We need to be able to trust that professional groups are being guided by what’s in our interests as consumers/patients, not theirs.We can’t afford mysterious arbitrary processes protected by interested parties circling their wagons. The road to hell is paved with biased assessments of biased research by influential groups.
UPDATE 3 August 2017: An additional coconut trial was reported by Matt Hodgkinson on Twitter. Thanks, Matt!
UPDATE 6 August 2017: My original letter to the AHA was rejected for being over 500 words and having more than 5 references. In re-submitting a shorter letter, I followed through on a range of issues that I saw with the analysis of the core trials, while I was assessing differences in approach between that and the coconut oil analysis. This is the text I submitted to the AHA:
The Advisory does not adhere to expected methods to minimize bias. (1) That has contributed to errors and methodological problems in analysis and reporting.
- The basis for selecting only 3 systematic reviews for the central analysis is not reported. It is claimed that all 3 “showed significant and substantial reduction in CVD”: 1 did not (except for a statistically significant finding in a sensitivity analysis).
- The report states 6 non-core trials in those 3 reviews were excluded, providing a rationale for excluding a 7th. Reasons are not given for exclusion of another 3 trials. (2)
- Criteria were inconsistently applied. Trials were excluded for mixed dietary interventions. However, the intervention in both the Oslo and MRC trials including intensive counseling, and it was not reported that only saturated fat was addressed. The Oslo diet additionally restricted pure sugar and provided free sardines in cod liver oil: influence from these cannot be excluded. (3) The Oslo trial did not meet the criterion for reporting control of the control group’s diet.
- Outcomes ranging from possible signs of heart disease to death were combined to use “the primary outcome chosen by the trial”. However, only 1 trial (Dayton) included a primary outcome stated as such in references provided. In addition, 2 different outcomes are combined for the Finnish trial (Fig 1), while stating that the primary outcome was CHD death. Pooled outcomes are therefore neither comparable nor trial-specified.
- Data from a cluster crossover trial (Finnish) are meta-analyzed as though it is from an individual trial, without sensitivity analysis. (4) A significant difference is reported for women, which was not a conclusion of the trial.
- The only data on women are from the Finnish trial. The weakness of the selected clinical trial evidence for women is not accounted for in conclusions.
- Statements about absence of trials on broad health outcomes of coconut oil are made without a systematic review addressing broad health outcomes. The claims rest on a CVD-only review whose scope is limited (e.g. excluding studies “in a special clinical population”, such as those in the core trials above). There are at least 8 relevant trials that are not considered, (5) more of which are randomized than in the Eyres review (7 with 434 participants vs 4 with 143). Some assess non-lipid outcomes.
- It is reported that Eyres et al noted no difference in 7 trials comparing coconut oil to other saturated fats. However, Eyres reported only 5 trials making this comparison and concluded “the results are largely inconsistent”.
- It is stated that the 8 trials in the Eyres review provide “strong evidence”, but there is no assessment of the strength of evidence in these trials, in contrast to the extended exclusion criteria for trials in the earlier analysis.
Disclosures: I have no personal financial, livelihood, or intellectual conflicts of interest in relation to coconut or other dietary fats.
- Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. 2011. Washington DC: National Academies Press. https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0079468/ PMID:24983061
- Hamley S. The effect of replacing saturated fat with mostly n-6 polyunsaturated fat on coronary heart disease: a meta-analysis of randomised controlled trials. Nutrition Journal 2017; 16:30. https://nutritionj.biomedcentral.com/articles/10.1186/s12937-017-0254-5 PMID: 28526025
- Leren P. The effect of plasma cholesterol lowering diet in male survivors of myocardial infarction. A controlled clinical trial. Acta Med Scand Suppl 1966; 466: 1-92. http://onlinelibrary.wiley.com/doi/10.1111/j.0954-6820.1966.tb05097.x/full PMID:522820
- Donner A, Klar N. Issues in the meta-analysis of cluster randomized trials. Statistics in Medicine 2002; 21(19): 2971-2980. http://onlinelibrary.wiley.com/doi/10.1002/sim.1301/abstract PMID:12325113
- Bastian H. Circling the wagons, science style: AHA saturated biases redux. Absolutely Maybe, PLOS Blogs. August 2017. Internet: accessed 6 August 2017: https://blogs.plos.org/absolutely-maybe/2017/08/02/circling-the-wagons-science-style-aha-saturated-biases-redux/
Disclosures: I have no income or livelihood bias about saturated fats. But I’m saturated in plenty of bias nonetheless, even though I’ve never given much thought to coconut oil before this last week. Some of my family are deeply committed to veganism and environmental activism, and a friend is involved in the manufacture of vegan foods using coconut oil. I was a member of the GRADE Working Group when it was developing guidance on rating evidence and recommendations, and methods for going from evidence to recommendations. I was a member of the Australian National Health and Medical Research Council working party on guidelines for developing, evaluating and implementing clinical practice guidelines (1996-1998). I participated in establishing the Cochrane Collaboration, and participated in summarizing the original Cochrane review on dietary fat for consumers that is discordant with the AHA. (My day job includes working on curating and disseminating systematic reviews and guidelines at PubMed Health.)
* The national clinical practice guidelines I’m referring to here were on care around preterm birth, covering obstetric, midwifery, and neonatal intensive care issues (in 1996). Also in the dim dark past, I chaired a national ethics working group on limiting life-sustaining treatment, and another on community participation in clinical research. (The lawsuit I referred to was around recommendations on the health effects of passive smoking.)
** The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal, and do not necessarily reflect the views of the National Institutes of Health or the U.S. Department of Health and Human Services.