I was prompted to write this post following a question in a sports nutrition group that I’m a member of. The particular question was in reference to a fad diet, but the thrust of it is one that as a dietitian and as a personal trainer I’ve been confronted with a number of times, that being; “if nutrition research is good, why isn’t everybody on the same page regarding diet”.
A complete answer to that question might have to address things such as the imperative to stand out if you sell things (whereby having a “point of difference” might mean being deliberately contrarian or just flat out wrong), the cultural/social/economic underpinnings of our nutrition choices and so on. Without considering that, though, the largest part of the answer is that nutrition research is often ambiguous and hard to interpret.
So this post is a primer (a complete answer would probably take a textbook, and being honest I don’t have the background in either research or stats to write the definitive article) on why it is that we don’t see clear, easy to interpret results in nutrition, why we often see conflicting results, and what it takes to draw firmish conclusions from what we read in spite of that.
Firstly – philosophically science and scientific thinking is actually ABOUT uncertainty. We generate hypotheses based upon observations or the evidence available and then go about systematically trying to falsify them. Hypotheses that survive repeated testing are maintained, and those that are falsified are either modified in light of new information or discarded.
“Laws” in science describe an aspect of the universe. They do not explain “why”, they simply provide a framework of how things behave. For example, the laws of thermodynamics describe the dynamics of energy. You will not find experimental evidence that runs counter to the laws of the universe – that’s a surefire indication that another factor is at play. If a dietary strategy/framework is predicated upon abandoning a law of the universe, you’re probably best to run away screaming.
“Theories” build upon multiple well-substantiated facts to explain why phenomena in the natural world occur. Theories can be falsified – they are like a meta-hypothesis – but as with hypotheses they are often modified to accommodate new information that does not conform perfectly with their current framework. To say “x is just a theory” is a non-argument, as theories should be built off of well substantiated facts and some (eg evolution) are very comprehensive and supported by multiple streams of evidence, even if the particulars may at times be refined. Rather, demonstrating that the premises of a theory are incorrect, or that predictions/hypotheses that would be generated based off of a given theory are false can lead to their abandonment/modification.
Novel research identifies areas that our best current theories do not describe well, or potential avenues of falsification that have not been explored, and tries to find out more about them. Researchers may also base hypotheses off of prevailing theories and explore them to see whether the theory allows us to make sound inferences.
So when if you consider all of the above, when we appraise science, it’s not just a matter of what we DO know, it’s a matter of what we DON’T. Scientific thinking often involves us saying “well, we have pretty good reason to believe that x is true under y conditions, but we can’t be sure if that’s always the case”, or “x appears to be related to y, but how that is so isn’t clear yet” and so on. We can speculate based off of the best available evidence, but speculation has to be couched in the appropriate degree of uncertainty, and honest discourse often requires us to say “I’m not sure”.
All of that can be very dissatisfying to people who just want answers. But impatience in drawing conclusions leads to two potentially costly errors at opposite ends of the spectrum, the premature supposition of the truth of a hypothesis, and the premature abandonment of the prevailing wisdom. I’ll briefly recap why these are errors at the end of the article, but in the meantime just let them stew in your head as you read.
Given the above, science sounds tedious but in some respects simple. You make observations, describe them, make predictions, try to falsify them, and build a coherent framework of thought to explain what you’ve observed based off of whatever survives repeated testing.
Unfortunately, nutrition science is harder again, because human nutrition is incredibly difficult to observe for long terms under controlled conditions while measuring accurately. On top of that, the big outcome measures we tend to look at in nutrition studies are usually enormously multifactorial, and each nutritional influence may only have a very small determining role in them. Make sure to read this article too – it says and demonstrates some really useful things.
And that’s all just a start. I’m going to list, point-form (plus an explanation) a bunch of things that make nutrition science very difficult.
Before I even do that – there are a number of relevant study types that are used in nutrition research. Writing the definitive article explaining all of them is beyond me, but each has strengths and drawbacks – this is itself an area that complicates interpreting research, as different study types examining similar phenomena may not yield the same conclusions for various reasons.
- Longitudinal cohort studies involve following large groups of people over time. Baseline characteristics are measured, then exposure to dietary factors and relevant outcomes are measured over a follow-up period.
- Cross-sectional studies involve measuring dietary exposure and characteristics at one point in time
BOTH of the above study types are useful for hypothesis generation as they allow us to observe associations between individual characteristics and diet, and sometimes also see what other factors mediate these relationships. Whilst the former has a temporal component, it’s still inappropriate to use them as a sole basis to determine whether an observed relationship is causal.
- Case-control studies are used in the instance of rarer conditions/outcome measures, and involve retrospective analysis of the diets of those who experience the condition to see if they differ from otherwise similar controls.
- Randomised controlled trials involve individuals being randomly assigned to either be exposed to a dietary condition or not. These are the “gold-standard” trial for a number of reasons and are useful for determining causality. Because of the difficulty of controlling dietary intake long-term, in nutrition these are usually used for short-term trials and often with the measured outcome being a biological marker that is related to a health condition that researchers are interested in (for example, blood glucose levels when researching type 2 diabetes) which introduces another level of complexity and potential error, as I’ll discuss. Best practice is usually to blind researchers and participants (meaning they don’t know which dietary condition they are exposed to) but in most instances this is impractical or impossible in nutrition.
- In addition to the above, researchers may use randomised controlled trials in animal models, genetic studies or other molecular biology/biochemistry research methods to generate hypotheses or try to reconcile observations made between foods and health.
As I’ve said, there’s a lot more detail to ALL of the above, and a lot that goes into determining what a given study can tell you. There is also a hierarchy of evidence and a commonly used framework for inferring causality from observational/epidemiological evidence where randomised trials are not possible informing how people interpret these studies.
Above and beyond the individual studies above, researchers also conduct reviews and meta-analyses to try to draw together the evidence and make further conclusions. Systematic reviews synthesise all evidence that meets a given inclusion criteria to draw an overall conclusion answering a question, whilst meta analyses pool all of the data from studies that meet a given inclusion criteria to quantify a given effect. These can overcome many of the drawbacks of individual studies and also allow for more detailed analyses (such as subgroup analyses) and can include a description of how homo/heterogeneous the results are across the literature.
Narrative reviews differ from systematic reviews in that they detail the evidence in support/against a given position but do not include/exclude studies on a systematic basis.
ALL of that said, a study isn’t just a study, and if you intend to appraise the literature, having a grasp on HOW information was gathered is important when interpreting the conclusions presented and especially so when you’re trying to reconcile their differences. The degree and conditions of uncertainty with which we appraise the conclusions of a study are tied to its methods.
That out of the way – some reasons nutrition research can be ambiguous (these are particularly relevant for observational studies, which is where a large portion of nutrition evidence comes from):
1 – Many nutritional factors have only small effects on a measured outcome, or the measured outcome is subject to many many factors. Take, for example, heart disease. Heart disease risk is influenced by diet, genetics, smoking, physical activity, age and sex (and probably more). Within diet a variety of factors can contribute, including bodyweight/energy intake, fat composition, fruit and vegetable intake, salt intake and so on. Teasing out the contribution of one small factor is very hard, and controlling for all other factors is also very difficult.
2 – I’ll come back to this, but teasing out the role of individual nutrients is even harder because we don’t eat nutrients, we eat foods. A steak contains complete proteins, fats (including monounsaturated, polyunsaturated and saturated fats), minerals like magnesium and sodium, various vitamins and so on. Exposure to any one nutrient in abundance also entails exposure to a variety of other nutrients.
3 – When researchers use statistical significance cutoffs, they measure the probability that a given relationship has arisen by chance. (It’s actually more complicated than that – a p-value measures the probability that were the null hypothesis true – ie were there no relationship between the variables being compared – an association at least as strong would be observed).
However, the more tests that are run, the more probable that a result will arise from chance within the set of results. A p-value cutoff of 0.05 denotes a 5% (1/20) probability that a given result could be obtained even were there no association between the two variables. Run 20-30 tests with that cutoff and suddenly it’s more likely than not that one of your results IS just random.
This is true in the wider sense too, with a huge number of studies measuring a given phenomenon, there will always be the odd aberrant result.
It is also true that results that reject the status quo are more likely to be reported on. When was the last time you read a headline saying “confirmed – eating your vegetables is healthy”? Yet newspaper articles saying “new study finds opposite of prior theory” come out almost every day. There is also less emphasis given to replication of research (which allows us to strengthen our faith in its conclusions) and in some instances a bias towards publication of novel results as opposed to those that merely reiterate the status quo despite the importance of doing just that in many instances.
Which is to say, given the sheer number of nutritional studies published, periodically there will just be some “noise”, or results that don’t necessary represent a real association, hence the need for multiple streams of evidence and replicated results before we set much stock in a conclusion.
It’s important to remember that recency of research doesn’t necessarily speak to its accuracy, and as I’ll say later again, it’s the body of evidence as a whole that needs be taken into account.
4 – Statistically significant observations may be of limited clinical relevance. For instance, a 1% reduction in risk of a disease that affects 1/100,000 people may be found, but whether it is worth acting on is another story.
Fad diets/dietary practices can arise on the basis of a limited number of studies finding associations between a given food/nutrient and a health condition, and then extrapolating from there without considering the primacy of other nutritional factors. An understanding of the magnitude of effect is necessary before making any type of nutritional prescription, and oftentimes practice involves boiling down complex and interrelated nutritional knowledge to simple, food-based advice.
5 – When associations are made between consumption of foods/food groups and health outcomes based off of observational studies, it can be hard to distinguish between food-specific effects and replacement effects. For instance – diets that are low in processed foods are, by definition, higher in unprocessed foods on a calorie for calorie basis. Similarly, diets that are lower in meat are likely higher in vegetable content etc. Are health effects related to consumption of one food group or the relative absence of another? This is very difficult to control for.
6 – Sometimes research uses proxy measures for health outcomes (remember what I said above re blood glucose levels). This is helpful/practical in shorter term trials, as if you are only doing a 12 week intervention you’re unlikely to see enough actual cases of, say, heart attack to make firm conclusions about its relationship to diet. However you could reasonably expect to see differential effects between two diets and a biological marker such as cholesterol. In certain instances this is probably perfectly fine. For example, fasting blood glucose and HbA1c are diagnostic criteria for type 2 diabetes, so measuring them is akin to measuring disease progression. In other instances, though, extrapolating from a biological marker to a health condition is more fraught. For instance, chronic inflammation may be related to risk of a given disease, but observing a relationship between diet and a biological measure of inflammation is probably only sufficient to generate a hypothesis that said diet increases the risk of that disease, which would then need to be demonstrated over multiple longer-term trials. Additionally, where it matters, you need to be careful to not misconstrue measurements of acute elevation/suppression to be chronic, and this takes a degree of expertise in itself.
The relationships between biological markers and actual pathology can be very complicated, bidirectional and/or mediated by other factors, and so giving them too much weight is probably a mistake. This is especially frustrating given that for practical purposes running long-term randomised trials in nutrition is especially hard, so much of the most rigorously controlled/measure evidence comes from trials that don’t directly assess the outcomes we are most interested in (usually chronic disease).
7 – Even where nutrients are the same, some nutrient-health interactions are mediated by the food complex itself. A couple of examples – the majority of fatty acids in milk/dairy are saturated, with a decent amount of monounsaturated fat and a little polyunsaturated fat. Given the general instruction to reduce saturated fat intake, reducing full fat dairy consumption might be considered healthy, however there is evidence that dairy consumption has neutral or positive effects on CVD risk and may protect against diabetes. One potential avenue to explain dairy’s health benefits is the milk fat globule membrane (MFGM) – a phospholipid and protein membrane that encapsulates milk fats which has shown some anti-inflammatory, blood-pressure and cholesterol-lowering effects. The churning process of creating butter destroys the MFGM, which is why butter more negatively impacts cholesterol than cream. (No, butter isn’t a health food yet. Yes, still having some of it is fine). So for a given nutrient, context CAN still matter.
On a not-entirely dissimilar note, a recent large meta-analysis came to very underwhelming conclusions about the benefits (or lack thereof) of supplementary vitamins and minerals for cardiovascular disease and all-cause mortality. To quote .. “in general the data on the popular supplements… show no consistent benefit for the prevention of CVD, myocardial infarction, or stroke, nor was there a benefit for all-cause mortality to support their continued use”.
This may be surprising, given that most people do not consume an optimal diet, with the majority of Australians failing to meet recommended intakes of our best sources of most micronutrients, vegetables. Additionally, vegetable consumption is quite consistently found to be protective against pretty well everything.
The question then becomes – are there other factors in the food complex of vegetables (phytonutrients etc) mediating or modulating their health effects? Are the benefits of vegetables mostly because they help people control their bodyweight, or because higher vegetable content diets are generally higher quality diets, or (in Western countries, anyway) consumed by those of a higher SES? To be honest, I don’t know and there’s a good chance all of the above are true to some extent, but separating the health effect from the food source itself is very difficult.
So bringing all of that together, making associations between nutrients and health effects is difficult, and to some degree subject to the sources of those nutrients. As with point 2 above – nutrients matter enormously, but first and foremost we eat foods. It also means that some of the disparity in results we may observe between studies examining the relationship between a given nutrient and an outcome are attributable to differences in the actual food composition of the diets.
8 – Many dietary interventions have highly variable results on a person to person basis. Taking a conclusion that only reflects the average/pooled result of a diet may miss the important interindividual differences. Differences in response to a diet may be due to factors such as adherence/personal preference, but also may be due to genetic and lifestyle factors that may not be as of yet entirely understood. Consider that in weight loss individuals vary in the magnitude of their adaptive response to reduced energy intake and propensity to misreport intake etc. What may constitute an optimal approach for one person is not necessarily the same for another, and so drawing conclusions about individuals (and therefore the applicability of dietary approaches) requires a reasonable degree of nuance in itself.
9 – It is really hard to get highly accurate measures of dietary intake, especially long-term. Tracking peoples’ intake in the real world often relies on methods that are burdensome and prone to omission/error (such as weighed food records) or imprecise (such as food frequency questionnaires). Different measurement techniques have different sources of error, and interpreting the information collected requires an understanding of its limitations. This adds another layer of complexity to the interpretation of the information gathered.
10 – There is a large grey-area of optimality in most respects. For instance, it is widely accepted (outside of a few particular echo chambers on the internet) that the ratio of carbohydrates to fats in a weight loss diet has negligible bearing on its efficacy. (There are a variety of reasons why I still think the optimal starting point for most is a low-moderate fat diet, but that’s a separate blog post). Similarly, the consensus view is that the total amount of fat in the diet is less important than the types of fat consumed for determining heart disease risk, and that replacement of saturated fats with mono/polyunsaturated fats is probably helpful, but that replacing saturated fats with carbohydrates has limited effect on risk. In either instance, the first-principles view leaves a very wide range of viable diets, with the choice subject to other factors (tastes, availability, satiety, adherence etc).
This means that on a study by study basis, one might easily conclude that one particular dietary arrangement is favoured over another without appreciating that the differences observed could easily be attributable to something else.
And that brings me neatly to the next section…
How can we possibly know what is true?
Interpreting nutrition research requires a grasp of the TOTALITY of the evidence – that means an appreciation of what the bulk of research investigating a question has found (either through familiarity with individual studies or through reviews and meta analyses), an understanding of the biological/biochemical rationale for a hypothesis or dietary strategy, and an appreciation of dietary and behavioural context. For individual studies, or media reports on individual studies, it is important to be wary of extrapolating too far from the conditions of the study itself.
It also takes an ability to contextualise information – just as many of the health conditions that nutrition has a bearing on are multifactorial, multiple nutritional interventions can have a bearing on a given condition. Understanding which factors are MOST important allows you to make better decisions as to where to invest your energy. As an extremely simplistic example, bodyweight control is of very high importance for prevention/treatment of a large number of conditions, where controlling more finicky factors might only yield a comparatively small benefit.
In general – the basics (bodyweight control, nutritional sufficiency, moderation +/- variety) matter a whole lot more than details, and are also more strongly supported in research. With each step away from the basics we take, the tentativeness with which we draw our conclusions increases. Despite this, the details often get the most attention as they are the most exciting and novel.
Bringing back my two potentially costly errors:
1 – The premature supposition of the truth of a hypothesis – as you can now see, it takes a LOT for us to be really confident in a hypothesis. Many associations we observe turn out to lead nowhere new, and so even promising hypotheses need to be approached with scepticism.
2 – The premature abandonment of the prevailing wisdom – studies that at first glance appear to contradict the status quo are a dime a dozen. However, many such contradictions can be explained by an appreciation of limitations of the measurements of methodology employed by a study, an understanding of the multitude and relative importance of factors that may influence a given outcome, or are simply a blip in a larger body of evidence.
Ironically in both instances, despite the potential for grave errors in thinking/reasoning, it’s still entirely possible for people to follow shonky dietary advice and make improvements to their health, provided it satisfies the higher order principles (see point 10 above). That people then take this is vindication of their dietary ideology is another reason why fads perpetuate.
So if at this point you’ve decided that it’s extremely hard to figure anything out about nutrition, good. It is. That’s why the people who gather, research and interpret this information do so as a full-time job, and why most dietitians specialise and continue to do professional development – learning is an iterative process.
There IS disagreement amongst academics over many things, and that’s both normal and helpful, as disagreement fosters discussion and allows for the eventual exclusion of weaker arguments. As an outside observer it is also easy to confuse disagreement driven by a vocal minority or a few prominent individuals as indication of widespread uncertainty of conclusions that are actually held quite strongly. Again, this means that without a strong grasp of the literature and the theoretical underpinnings of each position, it is not easy to determine which views are best supported.
But amongst all the chaos consensus positions DO eventually emerge, and those that continue to be supported by new research and stand the test of time are the basis of most nutritional advice.
A practical summary in short:
– Put MOST of your confidence and most of your effort into the basics (bodyweight control, varied, largely unprocessed diets, lots of vegetable matter and water)
– Be wary of novel information – it’s unlikely to matter as much as the above, and if it contradicts it in principle, it’s probably rubbish.
– Remember – overturning a large body of evidence requires extraordinarily compelling research. A diet blog (or even a proper study or two) doesn’t cut it, and those in the know might be able to explain the apparent contradictions with other factors.
– If you have a nutritional question, or a health condition that might benefit from nutritional intervention, consult the professionals. Ask your GP for a referral to a dietitian or go to trusted sources (large organisations such as the Cancer Council or (god forbid) the government guidelines) FIRST.
– If somebody says “you can find a study to support anything” any way other than facetiously, they don’t have a clue.
If you LIKED this article, subscribe to my mailing list. I can send you more stuff, direct to your inbox, and it’s free. Click here.
I also have a podcast where I talk about lifting weights and stuff. If you want to hear the sound of my voice, try clicking here.