How to interpret the latest meta-analysis on saturated fat and health
We got a review back recently for a paper we submitted to the New Zealand Medical Journal. We were questioning the ongoing obsession with saturated fat and disease, especially heart disease. We were rejected – a common thing (at least for us) when you talk about the evidence not being sufficient to making recommendations for what the public eats.
Just for your interest, and if you aren’t familiar with the academic world, here are some snippets of what you get back when you submit and have a paper rejected.
You will see why it is so frustrating, and that the process itself is mostly broken, as it doesn’t encourage debate, it stifles it. In my view we need to move to a more open review system.
I’m not saying our paper was “worthy” and I’m just bitter about it being rejected (although that is a minor factor). I’m just saying what happens.
The aims of this blog are to:
- Show you what happens in the murky academic nutrition world.
- Look further into the actual evidence of the latest meta-analysis. Once you delve into the actual techniques you will see the house of cards the “eat less saturated fat” dogma is built on.
This will be of interest to those into the science and academic side of nutrition.
- The review process
Reviewer snippets…and what we’d like to say back.
This article does not add much to the debate that it addresses. The authors mistakenly believe that randomized trials are inherently less biased than nonexperimental research, but that is not the case.
Randomized trials are effective at controlling confounding, but there are other ways to control confounding (e.g. matching and restriction)
and they are still subject to other biases. For example, the MRFIT trial mentioned in the article did not find a positive association between smoking and lung cancer after 16 years of follow-up, but that
reflects on inadequacies of the trial, not on the limitations of the nonexperimental studies that have elaborated the smoking-lung cancer association.
Our immediate reaction….duh, that’s why we have meta-analyses, and if in the end all of the trials don’t show anything then how is it likely that changing nutrition guidelines will do any better?
The authors also wish to equate a lack of statistical significance with a null effect, which is erroneous, damaging their argument and their credibility.
Our reaction: Say what? The trials show nothing, but we shouldn’t confuse that with any possibility of anyone ever showing an effect? When evidence gets to that point, then we’d reconsider our view. In the meantime, no evidence = no evidence.
A small, non-significant effect in a meta-analysis of heterogeneous results (some trials result in harm, some in benefit) is not a meaningful result. If results were homogeneous (all trials having similar results), a non-significant trend might be given more credibility.
Nonetheless, the authors do offer a few criticisms of earlier writers in
the journal that are worth considering. However, the length of this article is unnecessary to make those criticisms, and the arguments are far from clear. On balance, I think that this debate is important but this particular piece does not advance it.
Us: Damned by faint praise? Not even.
The author[s] appear[s] to regard the Cochrane Review as being particularly relevant to their argument. I would refer them to the updated Cochrane Review published in the past few weeks by the same lead
author, which reports a significant 17% reduction in total CVD events (fatal and non fatal) and unplanned cardiovascular interventions, with saturated fatty acids (SFA) reduction. A decrease in all cause and CVD mortality is not ruled out by this metaanalysis. Indeed it would be surprising given that the mean duration of the included trials was 52 months, with only 2 included trials of more than 5 years’ duration, had the reduction in cardiovascular diseases been reflected in a significant reduction in total mortality.
Us: First bit, good point, we will look more at the latest Cochrane review (second part of this blog). Second bit, but really – the low fat, low saturated fat intervention isn’t shown to be effective, but that’s because we don’t have long-term evidence, but we should carry on anyway?
What to take from this?
I may as well write a blog about this. More people will read it than the subscription-only NZ Medical Journal. It will be judged by people who haven’t spend decades defending an outdated proposition!
I’ll explore the latest meta-analysis and uncover whether the various assumptions that derive comfort from it are justified. I suspect that, no matter how many meta-analyses are done, the nutrition establishment won’t change their mind (“science advances one funeral at a time” Max Planck).
- What is a meta-analysis anyway?
A meta-analysis is a way to bring together all of the studies which have studied the same thing, to produce an averaged result from a larger data set. The Cochrane collaboration is a virtual community of academics who carry out such analyses, and the processes they use are widely regarded as “gold standard”.
The meta-analysis of saturated fat reduction has just been updated (2015) by Hooper et al, in the Cochrane reviews site. In this case, the meta-analysis is of the effect of 13 trials which reduced saturated fat and recorded cardiovascular endpoints.
What do Hooper et al (2015) show?
They show no effect of interventions for the reduction of saturated fat intake on
- All-cause mortality
- Cardiovascular mortality
- Myocardial infarctions
- Non-fatal MI, Stroke
- CHD mortality
- CHD events.
There was a significant effect for combined cardiovascular events (R 0.83 (0.72 to 0.96)) of reducing saturated fat over a standard diet, however this limited effect (mainly a reduction in the incidence of angina) was solely restricted to studies where polyunsaturated fat replaced saturated fat. (Note that in our second reviewer’s reading of the Cochrane review they mentioned combined fatal and non-fatal events. Fatal events are easily distinguished from the more common non-fatal events; combining the two makes the results look more impressive than they really are.)
Here’s their actual comments on the only significant finding they have
“This systematic review of long-term randomised controlled trials of SFA reduction suggests that reducing saturated fat for at least two years had no clear effects on all-cause or cardiovascular mortality, but a 17% reduction in combined cardiovascular events with important heterogeneity. This clear effect on cardiovascular events was not lost on sensitivity analyses….”
This is more detail in their Table 6 (click below to enlarge) which shows the individual studies used, as well as the combined results – this is what a meta-analysis does.
I first made a crude attempt to understand exactly what this “17% reduction in CVD events” meant in absolute risk. The 17% is relative risk which tends to make the actual population effect sound bigger than it was because the actual events are only a small proportion of the total people (one person can have more than one event, of course, but we’ll pretend otherwise for simplicity).
So I simply used the numbers at the bottom of the table – in total in the “Reduce saturated fat (treatment)” there were 21 791 people, of which 1774 (or 8.14%) had some sort of cardiovascular event over the study period (52 months).
In the “usual diet (control)” there were 31509 people, or which 2603 (or 8.26%) had some sort of cardiovascular event over the same period. (Of course, the same person can have more than one event, and probably does; I’m expressing it in terms of people for simplicity’s sake, this is complicated enough already!).
To more directly compare what might happen here we can extrapolate. If we extrapolated and pretend we randomised 20 000 people – 10 000 to a reduced SFA diet and 10 000 people their usual diet then we would expect to see 826 CVD events for the usual diet and a reduction to 814 for the reduced saturated fat. That’s a difference in 12 CVD events in 10 000 people over just about 4.5 years, because of eating less saturated fat (compared to doing nothing).
Underwhelmed? Hell yes. So am I.
But that’s not how you do these analyses. Even although those are the actual numbers from the actual people involved in the studies, there is a confounding issue. You see the bigger studies end up with perhaps more weight than they should. There needs to be some way of understanding that better.
On balance though you’d think the numbers would show something similar, right? How does something which when aggregated shows nothing, somehow turn into something which shows something?
This opened up some discussion between myself and the other authors of our paper Dr Simon Thornley and George Henderson.
Grant: “Guys – why do the actual numbers from the actual trials have no apparent resemblance to the relative risks they worked out? You’d think that actual people in actual experiments counts for something? There’s clearly nothing happening here?”
Simon: (Public health physician and PhD Epidemiologist). Basically, each trial is considered a “strata” and the relative risk (or absolute risk, if you’d like to calculate it) is summed by applying an inverse-variance weight to the calculated relative risk of each trial. That is, the studies with narrowest confidence intervals are given more weight than those with wider intervals. This is a reflection of the overall statistical power of the trial (number of events and number of participants). This is like a stratified Mantel-Haenszel epidemiological analysis, to account for confounding. This is for fixed-effects analysis, and there is additional step to account for heterogeneity between trials, which is what they did in this analysis (random effects analysis)….
Grant: Statistical techniques or not, the reality doesn’t play out for the casual observer, even if there is confounding? OK the simple addition I was engaging in clearly is confounded, but its also what has happened to actual people in reality. only 4 of the 13 trials show an effect with a total N of less than 1500 people, when there are over 53 000 in the meta-analysis?
Let’s look at the MH weights its the Oslo diet heart study which contributes quite a bit to this effect. Plenty of people have written about the multifactorial (several dietary factors, exercise etc etc) nature of this trial before. It was men only, secondary treatment, and they only had diet information for 17/206 people in the intervention.
The same criticism applies to the STARS trial in 1992 – men only, secondary care, multifactorial .
So how is that the maths makes two trials which have about 450 people in total, which are multifactorial, men only, secondary care able to be included and then essentially guide public policy on food? Meta-analytic techniques or not that makes no sense to me.
Simon: It might be worth emphasising this aspect… That the majority of trials returned disappointing results, and pooling the results only leads to showing a benefit for the least reliable outcome… CVD events… The most clinically important and most accurately assessed outcomes – CVD death and overall mortality – are still frustratingly resistant to statistical jiggery…!
As Malcolm Kendrick says… “The best this result suggests is that avoiding saturated fat might change the cause of death on your death certificate, but not the date of death”… If this is worth pushing to the masses, then at least they should be open about this….
George: Apart from the fact that most of the saturated fat reduction trials also involved in practice some sugar reduction, which we don’t hear about from Hooper et al., there are two interesting findings in the sub-group analysis.
1) the benefit of reduced CVD events only belongs to the saturated fat reduction trials that increased polyunsaturated fat. Replacing saturated fat with carbohydrate, even when it lowers cholesterol, isn’t associated with any benefit.
2) there is no reduction of CVD events in the 3 trial groups, including the very large WHI trial (49,000 people) published since 2000.
The authors hypothesise that this second result is due to increased statin use in recent times giving a blanket protection against CVD events, regardless of diet. Well, maybe. But there’s another explanation that’s consistent with finding 1): polyunsaturated fat consumption has been creeping upwards since the 1960’s. The baseline PUFA intake in the WHI trial was 7.8% of energy, in other words, the amount of PUFA consumed by controls (the “high saturated fat” group) in WHI was probably similar to the amount consumed by intervention groups in earlier trials. So the whole thing may have nothing to do with saturated fat, only additional polyunsaturated fat.
Grant: Good points guys, and there’s also the quesiotn on trans-fat which wasn’t controlled for in earlier trials. I guess I’ll get comments and emails about how bad polyunsaturated fats are, the O6 to O3 ratios and so forth. These may all be true, but this is the way the epidemiology is playing out right now. That may change of course. See more on polyunsaturated fats below. …
Commenting in a 2010 editorial on an earlier meta-analysis of saturated fat replacement RCTs by another group (Dariush Mozaffarian et al.), Lee Hooper had this to say:
“However, dietary patterns have changed over the 20–50 years since these studies ware carried out.It would be useful to examine the full data set, including more recent trials before concluding, as the abstract does, that “a shift toward greater population PUFA consumption in place of SFA would significantly reduce rates of CHD.”
Such a shift has already occurred since these trials were carried out, and further shifts may be unhelpful.”
In Hooper et al. 2015, a more up-to-date data set has indeed been examined (and it is noted that there are no longer any new studies underway), and in our opinion the results seem to bear out best Lee Hooper’s 2010 prediction.
The saturated fat reduction in the diet recommendations are at best based on scant evidence. Even if we take the non-significant trends and the minor events in Hooper et al. at face value, which is probably making more of them than we should, the results suggest one thing – dietary interventions of the “reduce saturated fat” type are so ineffective that it’s surely long past time to try something else.
What I’d suggest – real food, unprocessed. If you need a star rating system or a food label it’s not in that category. Diets that help people balance their blood sugar and not constantly produce high levels of insulin are a public health priority. Yet this seems to never get mentioned. Cut out sugar, forget about saturated fat if you eat actual real food.
The New Zealand medical establishment followed the lead of the United States in believing that saturated fat is bad for us. It should now follow the lead of the United States as it backs away from that stance, as exemplified by these two articles that indicate the USDA is losing interest in saturated fats as a nutrient of concern:
My personal kudos to you for your work, which is helping lead people back to a healthier, low-carb diet, rich in the natural saturated fats nature intended us to eat. Thank you!
Thanks for the kind words Lynn
So if I discovered penicillin today I would be told to go away and start with clean equipment … In my view science only progresses when you discover something no one expected. In so many trials today the outcome is agreed first then the experiment conducted in away that will produce the supporting data. Obviously the science is settled and we don’t want young upstarts rocking the boat.
It is critical when looking at replacing saturated fats with polyunsaturated fats that you compare all cause mortality; higher non-CVD mortality more than compensates for the (tiny) reduction in CVD mortality. It is quite hilarious to see people try to recommend something that means you die less of heart disease because you die more from cancer. Warning: meta-studies of all cause mortality will often exclude “suicide, violent and accident” deaths because of course these have nothing to do with nutrition (systemic inflammation? Different brain fats? Effect on myelin sheaths?) . As a road safety expert I can tell you that car crashes do not happen ‘randomly’; they are clustered around a whole set of behavioural issues.
Pingback: How to interpret the latest meta-analysis on saturated fat and health | Wellbeing & Common Sense
Tell the lions and tigers in the wild to leave the fat behind.
A nice blog Grant!
Thanks Hong, keep up your awesome work in ophthalmology. I still think their is some place of low carb diets in some of the eye diseases especially those in retinal degeneration which may be related to mitochondrial issues
Dear Grant, forget wasting your time tackling the entrenched establishment.your success will be amongst the masses as the word spreads. after a life time of running,tennis and a constant and largely unsuccessful effort to lose weight your gospel has been like a breath of fresh air.my medical history includes a week in a coma inAuckland hospital, 2hip replacements and taking100mg of Voltaren for psoriatic arthritis for 28 years until my kidneys got a bit shaky. Anyway a vegetarian diet and finally your logical ideas and I feel great. 91kg down to 83.5 ,kg and i feel consistently great I am 75 and look forward to a healthy and productive old age keep up the good work cheers jim Fergusson Sent from my iPad
Thanks for your comment Jim, its made my day!
How do people get away with reporting relative risk as opposed to the more meaningful absolute risk? 8.14% vs 8.26%? Really?? So for every 830-odd people treated just 1 will be saved from a CVD event? Nothing to get excited about. Underwhelmed indeed.
Would that small benefit justify a diet full of refined carbs and Omega 6 PUFA and a daily dose of your favourite statin?
Well done on the review. Pity about being rejected. The reviewers didn’t give particularly strong reasons, but they had probably already made up their minds. Will you try another journal?
As an aside, it’s interesting that even though the recent Cochrane meta-analysis was about reduced SFA (it wasn’t updated, this one was separate) all of the trials they included were either SFA>PUFA, Mediterranean or reduced fat diet trials, suggesting there may not be any relevant trials that have reduced SFA in isolation (therefore not a smart aim for a meta-analysis). With this in mind, I wonder how the SFA debate will play out given the title/conclusions (all people seem to read) of this meta-analysis vs. the current positions of the ADA, Darisuh Mozafarrian, etc. Will this point even be noticed?
Thanks for the link love
I am reading “The Big Fat Surprise” by Nina Teicholz at the moment and she explains in great details how the fat heart disease hypothesis perpertuates, and why discenting opinions were/are always rejected in mainstream journals. It remains incredible that a handful of “experts” under the leadership of Ancel Keys could get away with it for so long. We the public are suffering the consequences 40/50 years later.
The good thing to come out of the internet is, you aren’t dpendent on scientific journals any more to get your opinion across. Just create your own website and publish. You will surely get a following of thousands, if not millions, of like minded people, who refuse to take as gospel what those experts advocate.
Keep up the good work.
Just one small sample from Nina’s book:
An Extreme Case of Selection Bias
There has been a lot of selective reporting and ignoring of the methodological problems over the years. But probably the most astonishing example of selection bias was the near-complete suppression of the Minnesota Coronary Survey, which was an outgrowth of the National Diet Heart Study. Also funded by NIH, the Minnesota Coronary Survey is the largest-ever clinical trial of the diet-heart hypothesis and therefore certainly belongs on the list along with Oslo, the Finnish Mental Hospital Study, and the LA Veterans Trial, but it is rarely included, undoubtedly because it didn’t turn out the way nutrition experts had hoped.
Starting in 1968, the biochemist Ivan Frantz fed nine thousand men and women in six Minnesota state mental hospitals and one nursing home either “traditional American foods,” with 18 percent saturated fat, or a diet containing soft margarine, a whole-egg substitute, low-fat beef, and dairy products “filled” with vegetable oil. This diet cut the amount of saturated fat in half. (Both diets had a total of 38 percent fat overall.) Researchers reported “nearly 100% participation,” and since the population was hospitalized, it was more controlled than most—although, like the Finnish hospital study, there was a good deal of turnover in the hospital (the average length of stay was only about a year).
After four-and-a-half years, however, the researchers were unable to find any differences between the treatment and control groups for cardiovascular events, cardiovascular deaths, or total mortality. Cancer was higher in the low-saturated-fat group, although the report does not say if that difference was statistically significant. The diet low in saturated fat had failed to show any advantage at all. Frantz, who worked in Keys’s university department, did not publish the study for sixteen years, until after he retired, and then he placed his results in the journal Arteriosclerosis, Thrombosis, and Vascular Biology, which is unlikely to be read by anyone outside the field of cardiology. When asked why he did not publish the results earlier, Frantz replied that he didn’t think he’d done anything wrong in the study. “We were just disappointed in the way it came out,” he said. In other words, the study was selectively ignored by its own director. It was another inconvenient data point that needed to be dismissed.
Need I say more?!?!
Pingback: How is data manipulated? | No Fructose