We got a review back recently for a paper we submitted to the New Zealand Medical Journal.  We were questioning the ongoing obsession with saturated fat and disease, especially heart disease.  We were rejected – a common thing (at least for us) when you talk about the evidence not being sufficient to making recommendations for what the public eats.

Just for your interest, and if you aren’t familiar with the academic world, here are some snippets of what you get back when you submit and have a paper rejected.

You will see why it is so frustrating, and that the process itself is mostly broken, as it doesn’t encourage debate, it stifles it. In my view we need to move to a more open review system.

I’m not saying our paper was “worthy” and I’m just bitter about it being rejected (although that is a minor factor).  I’m just saying what happens.

The aims of this blog are to:

• Show you what happens in the murky academic nutrition world.
• Look further into the actual evidence of the latest meta-analysis. Once you delve into the actual techniques you will see the house of cards the “eat less saturated fat” dogma is built on.

This will be of interest to those into the science and academic side of nutrition. 

1. The review process

Reviewer snippets…and what we’d like to say back.

Reviewer 1:

This article does not add much to the debate that it addresses.  The authors mistakenly believe that randomized trials are inherently less biased than nonexperimental research, but that is not the case.

Randomized trials are effective at controlling confounding, but there are other ways to control confounding (e.g. matching and restriction)

and they are still subject to other biases.  For example, the MRFIT trial mentioned in the article did not find a positive association between smoking and lung cancer after 16 years of follow-up, but that

reflects on inadequacies of the trial, not on the limitations of the nonexperimental studies that have elaborated the smoking-lung cancer association.

Our immediate reaction….duh, that’s why we have meta-analyses, and if in the end all of the trials don’t show anything then how is it likely that changing nutrition guidelines will do any better?

The authors also wish to equate a lack of statistical significance with a null effect, which is erroneous, damaging their argument and their credibility.

Our reaction: Say what? The trials show nothing, but we shouldn’t confuse that with any possibility of anyone ever showing an effect? When evidence gets to that point, then we’d reconsider our view.  In the meantime, no evidence = no evidence.

A small, non-significant effect in a meta-analysis of heterogeneous results (some trials result in harm, some in benefit) is not a meaningful result. If results were homogeneous (all trials having similar results), a non-significant trend might be given more credibility.

Nonetheless, the authors do offer a few criticisms of earlier writers in

the journal that are worth considering.  However, the length of this article is unnecessary to make those criticisms, and the arguments are far from clear. On balance, I think that this debate is important but this particular piece does not advance it.

Us: Damned by faint praise? Not even.

Reviewer 2:

The author[s] appear[s] to regard the Cochrane Review as being particularly relevant to their argument. I would refer them to the updated Cochrane Review published in the past few weeks by the same lead

author, which reports a significant 17% reduction in total CVD events (fatal and non fatal) and unplanned cardiovascular interventions, with saturated fatty acids (SFA) reduction.  A decrease in all cause and CVD mortality is not ruled out by this metaanalysis. Indeed it would be surprising given that the mean duration of the included trials was 52 months, with only 2 included trials of more than 5 years’ duration, had the reduction in cardiovascular diseases been reflected in a significant reduction in total mortality. 

Us: First bit, good point, we will look more at the latest Cochrane review (second part of this blog). Second bit, but really – the low fat, low saturated fat intervention isn’t shown to be effective, but that’s because we don’t have long-term evidence, but we should carry on anyway?

What to take from this?

I may as well write a blog about this.  More people will read it than the subscription-only NZ Medical Journal.  It will be judged by people who haven’t spend decades defending an outdated proposition!

I’ll explore the latest meta-analysis and uncover whether the various assumptions that derive comfort from it are justified. I suspect that, no matter how many meta-analyses are done, the nutrition establishment won’t change their mind (“science advances one funeral at a time” Max Planck).

2. What is a meta-analysis anyway?

A meta-analysis is a way to bring together all of the studies which have studied the same thing, to produce an averaged result from a larger data set. The Cochrane collaboration is a virtual community of academics who carry out such analyses, and the processes they use are widely regarded as “gold standard”.

The meta-analysis of saturated fat reduction has  just been updated (2015) by Hooper et al, in the Cochrane reviews site.  In this case, the meta-analysis is of the effect of 13 trials which reduced saturated fat and recorded cardiovascular endpoints.

What do Hooper et al (2015) show?

They show no effect of interventions for the reduction of saturated fat intake on 

• All-cause mortality
• Cardiovascular mortality
• Myocardial infarctions
• Non-fatal MI, Stroke
• CHD mortality
• CHD events.

There was a significant effect for combined cardiovascular events (R 0.83 (0.72 to 0.96)) of reducing saturated fat over a standard diet, however this limited effect (mainly a reduction in the incidence of angina) was solely restricted to studies where polyunsaturated fat replaced saturated fat. (Note that in our second reviewer’s reading of the Cochrane review they mentioned combined fatal and non-fatal events. Fatal events are easily distinguished from the more common non-fatal events; combining the two makes the results look more impressive than they really are.)

Here’s their actual comments on the only significant finding they have

“This systematic review of long-term randomised controlled trials of SFA reduction suggests that reducing saturated fat for at least two years had no clear effects on all-cause or cardiovascular mortality, but a 17% reduction in combined cardiovascular events with important heterogeneity. This clear effect on cardiovascular events was not lost on sensitivity analyses….”

This is more detail in their Table 6 (click below to enlarge) which shows the individual studies used, as well as the combined results – this is what a meta-analysis does.

Hooper et al’s (2015) Table 6 showing the individual and pooled effects reduced saturated fat on combined CVD events (click to enlarge)

I first made a crude attempt to understand exactly what this “17% reduction in CVD events” meant in absolute risk. The 17% is relative risk which tends to make the actual population effect sound bigger than it was because the actual events are only a small proportion of the total people (one person can have more than one event, of course, but we’ll pretend otherwise for simplicity).

So I simply used the numbers at the bottom of the table – in total in the “Reduce saturated fat (treatment)” there were 21 791 people, of which 1774 (or 8.14%) had some sort of cardiovascular event over the study period (52 months).

In the “usual diet (control)” there were 31509 people, or which 2603 (or 8.26%) had some sort of cardiovascular event over the same period. (Of course, the same person can have more than one event, and probably does; I’m expressing it in terms of people for simplicity’s sake, this is complicated enough already!).

To more directly compare what might happen here we can extrapolate. If we extrapolated and pretend we randomised 20 000 people – 10 000 to a reduced SFA diet and 10 000 people their usual diet then we would expect to see 826 CVD events for the usual diet and a reduction to 814 for the reduced saturated fat. That’s a difference in 12 CVD events in 10 000 people over just about 4.5 years, because of eating less saturated fat (compared to doing nothing).

Underwhelmed? Hell yes.  So am I.

But that’s not how you do these analyses. Even although those are the actual numbers from the actual people involved in the studies, there is a confounding issue.  You see the bigger studies end up with perhaps more weight than they should. There needs to be some way of understanding that better.

On balance though you’d think the numbers would show something similar, right? How does something which when aggregated shows nothing, somehow turn into something which shows something?

This opened up some discussion between myself and the other authors of our paper Dr Simon Thornley and George Henderson.

Grant: “Guys – why do the actual numbers from the actual trials have no apparent resemblance to the relative risks they worked out? You’d think that actual people in actual experiments counts for something?  There’s clearly nothing happening here?”

Simon: (Public health physician and PhD Epidemiologist). Basically, each trial is considered a “strata” and the relative risk (or absolute risk, if you’d like to calculate it) is summed by applying an inverse-variance weight to the calculated relative risk of each trial. That is, the studies with narrowest confidence intervals are given more weight than those with wider intervals. This is a reflection of the overall statistical power of the trial (number of events and number of participants). This is like a stratified Mantel-Haenszel epidemiological analysis, to account for confounding. This is for fixed-effects analysis, and there is additional step to account for heterogeneity between trials, which is what they did in this analysis (random effects analysis)….

 Grant:  Statistical techniques or not, the reality doesn’t play out for the casual observer, even if there is confounding? OK the simple addition I was engaging in clearly is confounded, but its also what has happened to actual people in reality. only 4 of the 13 trials show an effect with a total N of less than 1500 people, when there are over 53 000 in the meta-analysis?

Let’s look at the MH weights its the Oslo diet heart study which contributes quite a bit to this effect.  Plenty of people have written about the multifactorial (several dietary factors, exercise etc etc) nature of this trial before.  It was men only, secondary treatment, and they only had diet information for 17/206  people in the intervention.

The same criticism applies to the STARS trial in 1992 – men only, secondary care, multifactorial .

So how is that the maths makes two trials which have about 450 people in total, which are multifactorial, men only, secondary care able to be included and then essentially guide public policy on food?  Meta-analytic techniques or not that makes no sense to me.

Simon: It might be worth emphasising this aspect… That the majority of trials returned disappointing results, and pooling the results only leads to showing a benefit for the least reliable outcome… CVD events… The most clinically important and most accurately assessed outcomes – CVD death and overall mortality – are still frustratingly resistant to statistical jiggery…!

As Malcolm Kendrick says… “The best this result suggests is that avoiding saturated fat might change the cause of death on your death certificate, but not the date of death”… If this is worth pushing to the masses, then at least they should be open about this….

 George: Apart from the fact that most of the saturated fat reduction trials also involved in practice some sugar reduction, which we don’t hear about from Hooper et al., there are two interesting findings in the sub-group analysis.

1) the benefit of reduced CVD events only belongs to the saturated fat reduction trials that increased polyunsaturated fat. Replacing saturated fat with carbohydrate, even when it lowers cholesterol, isn’t associated with any benefit.

2) there is no reduction of CVD events in the 3 trial groups, including the very large WHI trial (49,000 people) published since 2000.

The authors hypothesise that this second result is due to increased statin use in recent times giving a blanket protection against CVD events, regardless of diet. Well, maybe. But there’s another explanation that’s consistent with finding 1): polyunsaturated fat consumption has been creeping upwards since the 1960’s. The baseline PUFA intake in the WHI trial was 7.8% of energy, in other words, the amount of PUFA consumed by controls (the “high saturated fat” group) in WHI was probably similar to the amount consumed by intervention groups in earlier trials.  So the whole thing may have nothing to do with saturated fat, only additional polyunsaturated fat.

Grant: Good points guys, and there’s also the quesiotn on trans-fat which wasn’t controlled for in earlier trials. I guess I’ll get comments and emails about how bad polyunsaturated fats are, the O6 to O3 ratios and so forth. These may all be true, but this is the way the epidemiology is playing out right now.  That may change of course.  See more on polyunsaturated fats below. …

PUFA changes in the US population (click to enlarge)

In Hooper et al. 2015, a more up-to-date data set has indeed been examined (and it is noted that there are no longer any new studies underway), and in our opinion the results seem to bear out best Lee Hooper’s 2010 prediction.

Take homes

The saturated fat reduction in the diet recommendations are at best based on scant evidence. Even if we take the non-significant trends and the minor events in Hooper et al. at face value, which is probably making more of them than we should, the results suggest one thing – dietary interventions of the “reduce saturated fat” type are so ineffective that it’s surely long past time to try something else.

What I’d suggest – real food, unprocessed. If you need a star rating system or a food label it’s not in that category. Diets that help people balance their blood sugar and not constantly produce high levels of insulin are a public health priority.  Yet this seems to never get mentioned. Cut out sugar, forget about saturated fat if you eat actual real food.