As ever when there is a diet related TV program on prime time I get a raft of comments, some people genuinely asking for my opinion and others, predicting my opinion, hoping for a bite from an irritated Breed! I am obviously happy to oblige in both situations. Last week saw a 3 part Horizon special on ‘finding the right diet for you’. Given the 3 hours of TV time dedicated to this I felt it was worth addressing it through the blog.
The theory behind the program was that there are certain distinct issues that dieters face and that these can be grouped with a certain diet being optimum for each group. The 75 participants for this program were grouped into ‘constant cravers’ – those that snack all day, ‘feasters’ – those that when they sit down to eat don’t know when to stop and ‘emotional eaters’ – those that over eat in response to emotional situations. The constant cravers were put on a 5:2 intermittent fasting protocol. The feasters were on a calorie controlled high protein, low GI approach and the emotional eaters went to slimming world for group support and a calorie controlled protocol and received CBT to help manage their emotions.
The results were exactly as could be predicted – initial early weight loss and high enthusiasm followed by a slowing of progress and, in some cases, regain of weight. At the end of the ‘study’ which only lasted 3 months average weight loss was 7% of body weight. On the face of it this all seems sensible and promising, however there were, as ever with these sort of programs, some glaring issues which may not be immediately apparent, yet invalidate the entire thing. This piece may seem ‘ranty’ at times yet my aim isn’t purely venting. I hope that with the following explanations I can show you the common issues with pop science programs so you can perform your own analysis and don’t have to take the presenters, often invalid, conclusions at face value.
My first issue with this format is the dramatisation of science. In the first few minutes we were introduced to the ‘nerve centre’ for the research, which was essentially a store cupboard at the University that they had filled with big TVs and had recruited students to stand around in white coats holding clipboards with nothing on them, pretending to have complicated conversations when the camera was pointed at them. If they were filming an episode of Sherlock I could understand but for a real life documentary this is so unnecessary. Then we have the presenters, the engaging Clinical Psychologist Tanya Byron assumed the intelligent role whilst her male Dr Co-presenter bumbled throughout the program acting like he’d been lobotomized, asking the most obvious of questions of the researchers so that us, the presumably thick target audience, could keep up and understand. This dramatisation of science in an effort to make it ‘approachable’ for the lay person has the opposite effect as it makes good science less approachable. People assume that science is for people in white coats and glasses and has to be explained to them in basic English. If the program instead started with a basic explanation of research methods everyone would be good to go and we could actually focus on the findings rather than having everything repeated to us in progressively simple language.
Moving on to the research methods let me start by pointing out that none of the ‘research’ involved in this program could be written up to create a publishable paper. There was a glaring lack of controls throughout which meant the conclusions drawn were erroneous. For example the ‘stress test’ where all the participants were put through a driving test and were then given a buffet from which they could eat ad libitum with the amount of food consumed measured. They then showed how the ‘emotional eaters’ consumed more in response to the stressful situation. This could be a really nice study design, however, they chose to sit all the separate groups together so all the feasters had their own buffet etc. So while the researchers claimed that this task accurately highlighted the ‘emotional eaters’ issue I could just as legitimately claim that this was a great example of social eating patterns and the greater consumption was in response to social pressures of sitting together. Controlling variables is essential to good research yet this is a topic that was never referred to in the program. The most glaring example of this was in the final conclusion drawn in program 3. The presenters excitedly signed off saying that they had proven that creating individualized diets based on certain eating styles was a successful way of dieting. Despite the myriad of smaller issues in this statement, which I will cover later, there is one factor which was again completely overlooked yet invalidates the entire project, which is the lack of a control group.
A control group is used for the purpose of comparison, they go through all the experimental protocols yet never receive the intervention. So in this example they would appear on TV yet never be given a diet to follow, Their weight outcomes would then be used as a comparison point for the intervention groups. The reason a comparison group wasn’t used, I imagine, is that it would have shown that the diets used weren’t all they were built up to be. In fact a more rigorous approach, with tighter controls and more detailed analysis would probably reveal that the best diet for achieving weight loss is to appear on TV in a program about weight loss.
To dig in a little further on some examples of the bad science involved in the show let’s look at another example – the breakfast test. Here some participants were given breakfast whilst others just had water. They were then asked to rate their responses to pictures of food a few hours later. This kind of research is classic University student stuff, simple and cheap and seems to show cause and effect yet in reality is so basic it doesn’t tell us anything. I should know as I had to use a similar method in one of my dissertations. In the Horizon version they showed that the breakfast skippers had a 40% greater response to the food pictures. Sounds impressive, however let’s think back to my post on bad science and representing statistics. What does this 40% actually mean? It means that on a scale of 1-5 the breakfast skippers scored the food pictures on average at 4.2ish whereas the breakfast eaters scored 3.5ish. Hardly jaw dropping is it? So whilst we can pick holes in the methods here that isn’t actually the largest problem. The biggest issue is actually seen in virtually all research on breakfast – the lack of a transitional run in period. What do I mean by this? When you move away from a carb based breakfast your body takes a few days to adjust to running on fat as the dominant fuel. These first few days can be difficult and you may feel sluggish and hungry. However once you’re adapted your energy rockets and your hunger falls. If the researchers had given the breakfast skippers a week of run in on that protocol then it would have been a valid test and the results would have been vastly different. Changing a diet and then testing the same day isn’t a valid research method where a transitional period is going to affect the results.
Now let’s move to the meat of the issue: the prescribed diets. Each diet was based around some form of caloric restriction, using this information alone we can pull apart the approach without even having to start touching on low GI or whole grains. The weight loss experienced by the dieters followed the classic trajectory of quick initial weight loss followed by a slowing and in some cases a regain of weight. This produced furrowed brows from the presenters and explanations were required from the ‘scientists’. With the use of some fancy apparatus (again boosting the image required for it to be real science we were witnessing) the researchers were able to demonstrate how the metabolisms of the dieters had slowed and therefore they needed to eat less calories. This seems a reasonable explanation but when we use some logic we can see where we might run into problems. Following this theory would suggest that we need to keep reducing calories as our metabolism slows. Where is the end point? Not eating at all? If we consider the evolutionary element of this we can see that the slowing metabolism is a response to the body essentially being starved. It is trying to conserve calories by reducing muscle mass. As I explained in my New Year Fads post on slimming clubs this pattern is all too common. Reduced calories produces weight loss - muscle mass is a large part of this weight loss - metabolism slows making weight loss harder - diet is ended - body is primed for fat storage and regains beyond original weight - metabolism is still slow due to reduced muscle mass - further weight loss becomes harder.
If anyone on this program took a step back and used some of the considerable intelligence you would hope they possessed then they could maybe work out that protecting muscle mass and therefore a faster metabolism is fairly integral to the long term success and health of a diet. This is where moving beyond weight loss as an outcome is important, we need to look at body composition changes to truly evaluate the success of a diet.
So all this being said what would I do if, in the unlikely event, I was handed the reigns to such a show?
I would start by stripping the sciency nonsense, to make the subject approachable you need to be honest with the audience. If you don’t need big TVs and clipboards don’t use them. Lay out the facts, carry out real research with real controls and explain why you are doing so. Give people the knowledge to think for themselves, don’t spoon feed. Next I would break out of the thinking that so many prominent researchers and commentators are stuck in. There is a whole wide world beyond low fat, high carb, restricted calorie diets. Whilst there is some mileage in the categories the researchers discussed I would abandon these in favour of one overarching nutritional approach (low carb, high fat). I would pitch this approach against a control, and a mainstream slimming club diet. I would then discuss ways that you at home could tweak and modify the approach to suit you without giving you a big label to live with. I’d look at the benefits of certain kinds of exercise and discuss psychological approaches to dieting. Writing this out it doesn’t seem so hard does it? Anyone know a producer at Channel 4?
Please share your opinions on the program and this piece in the comments or via social media.
The theory behind the program was that there are certain distinct issues that dieters face and that these can be grouped with a certain diet being optimum for each group. The 75 participants for this program were grouped into ‘constant cravers’ – those that snack all day, ‘feasters’ – those that when they sit down to eat don’t know when to stop and ‘emotional eaters’ – those that over eat in response to emotional situations. The constant cravers were put on a 5:2 intermittent fasting protocol. The feasters were on a calorie controlled high protein, low GI approach and the emotional eaters went to slimming world for group support and a calorie controlled protocol and received CBT to help manage their emotions.
The results were exactly as could be predicted – initial early weight loss and high enthusiasm followed by a slowing of progress and, in some cases, regain of weight. At the end of the ‘study’ which only lasted 3 months average weight loss was 7% of body weight. On the face of it this all seems sensible and promising, however there were, as ever with these sort of programs, some glaring issues which may not be immediately apparent, yet invalidate the entire thing. This piece may seem ‘ranty’ at times yet my aim isn’t purely venting. I hope that with the following explanations I can show you the common issues with pop science programs so you can perform your own analysis and don’t have to take the presenters, often invalid, conclusions at face value.
My first issue with this format is the dramatisation of science. In the first few minutes we were introduced to the ‘nerve centre’ for the research, which was essentially a store cupboard at the University that they had filled with big TVs and had recruited students to stand around in white coats holding clipboards with nothing on them, pretending to have complicated conversations when the camera was pointed at them. If they were filming an episode of Sherlock I could understand but for a real life documentary this is so unnecessary. Then we have the presenters, the engaging Clinical Psychologist Tanya Byron assumed the intelligent role whilst her male Dr Co-presenter bumbled throughout the program acting like he’d been lobotomized, asking the most obvious of questions of the researchers so that us, the presumably thick target audience, could keep up and understand. This dramatisation of science in an effort to make it ‘approachable’ for the lay person has the opposite effect as it makes good science less approachable. People assume that science is for people in white coats and glasses and has to be explained to them in basic English. If the program instead started with a basic explanation of research methods everyone would be good to go and we could actually focus on the findings rather than having everything repeated to us in progressively simple language.
Moving on to the research methods let me start by pointing out that none of the ‘research’ involved in this program could be written up to create a publishable paper. There was a glaring lack of controls throughout which meant the conclusions drawn were erroneous. For example the ‘stress test’ where all the participants were put through a driving test and were then given a buffet from which they could eat ad libitum with the amount of food consumed measured. They then showed how the ‘emotional eaters’ consumed more in response to the stressful situation. This could be a really nice study design, however, they chose to sit all the separate groups together so all the feasters had their own buffet etc. So while the researchers claimed that this task accurately highlighted the ‘emotional eaters’ issue I could just as legitimately claim that this was a great example of social eating patterns and the greater consumption was in response to social pressures of sitting together. Controlling variables is essential to good research yet this is a topic that was never referred to in the program. The most glaring example of this was in the final conclusion drawn in program 3. The presenters excitedly signed off saying that they had proven that creating individualized diets based on certain eating styles was a successful way of dieting. Despite the myriad of smaller issues in this statement, which I will cover later, there is one factor which was again completely overlooked yet invalidates the entire project, which is the lack of a control group.
A control group is used for the purpose of comparison, they go through all the experimental protocols yet never receive the intervention. So in this example they would appear on TV yet never be given a diet to follow, Their weight outcomes would then be used as a comparison point for the intervention groups. The reason a comparison group wasn’t used, I imagine, is that it would have shown that the diets used weren’t all they were built up to be. In fact a more rigorous approach, with tighter controls and more detailed analysis would probably reveal that the best diet for achieving weight loss is to appear on TV in a program about weight loss.
To dig in a little further on some examples of the bad science involved in the show let’s look at another example – the breakfast test. Here some participants were given breakfast whilst others just had water. They were then asked to rate their responses to pictures of food a few hours later. This kind of research is classic University student stuff, simple and cheap and seems to show cause and effect yet in reality is so basic it doesn’t tell us anything. I should know as I had to use a similar method in one of my dissertations. In the Horizon version they showed that the breakfast skippers had a 40% greater response to the food pictures. Sounds impressive, however let’s think back to my post on bad science and representing statistics. What does this 40% actually mean? It means that on a scale of 1-5 the breakfast skippers scored the food pictures on average at 4.2ish whereas the breakfast eaters scored 3.5ish. Hardly jaw dropping is it? So whilst we can pick holes in the methods here that isn’t actually the largest problem. The biggest issue is actually seen in virtually all research on breakfast – the lack of a transitional run in period. What do I mean by this? When you move away from a carb based breakfast your body takes a few days to adjust to running on fat as the dominant fuel. These first few days can be difficult and you may feel sluggish and hungry. However once you’re adapted your energy rockets and your hunger falls. If the researchers had given the breakfast skippers a week of run in on that protocol then it would have been a valid test and the results would have been vastly different. Changing a diet and then testing the same day isn’t a valid research method where a transitional period is going to affect the results.
Now let’s move to the meat of the issue: the prescribed diets. Each diet was based around some form of caloric restriction, using this information alone we can pull apart the approach without even having to start touching on low GI or whole grains. The weight loss experienced by the dieters followed the classic trajectory of quick initial weight loss followed by a slowing and in some cases a regain of weight. This produced furrowed brows from the presenters and explanations were required from the ‘scientists’. With the use of some fancy apparatus (again boosting the image required for it to be real science we were witnessing) the researchers were able to demonstrate how the metabolisms of the dieters had slowed and therefore they needed to eat less calories. This seems a reasonable explanation but when we use some logic we can see where we might run into problems. Following this theory would suggest that we need to keep reducing calories as our metabolism slows. Where is the end point? Not eating at all? If we consider the evolutionary element of this we can see that the slowing metabolism is a response to the body essentially being starved. It is trying to conserve calories by reducing muscle mass. As I explained in my New Year Fads post on slimming clubs this pattern is all too common. Reduced calories produces weight loss - muscle mass is a large part of this weight loss - metabolism slows making weight loss harder - diet is ended - body is primed for fat storage and regains beyond original weight - metabolism is still slow due to reduced muscle mass - further weight loss becomes harder.
If anyone on this program took a step back and used some of the considerable intelligence you would hope they possessed then they could maybe work out that protecting muscle mass and therefore a faster metabolism is fairly integral to the long term success and health of a diet. This is where moving beyond weight loss as an outcome is important, we need to look at body composition changes to truly evaluate the success of a diet.
So all this being said what would I do if, in the unlikely event, I was handed the reigns to such a show?
I would start by stripping the sciency nonsense, to make the subject approachable you need to be honest with the audience. If you don’t need big TVs and clipboards don’t use them. Lay out the facts, carry out real research with real controls and explain why you are doing so. Give people the knowledge to think for themselves, don’t spoon feed. Next I would break out of the thinking that so many prominent researchers and commentators are stuck in. There is a whole wide world beyond low fat, high carb, restricted calorie diets. Whilst there is some mileage in the categories the researchers discussed I would abandon these in favour of one overarching nutritional approach (low carb, high fat). I would pitch this approach against a control, and a mainstream slimming club diet. I would then discuss ways that you at home could tweak and modify the approach to suit you without giving you a big label to live with. I’d look at the benefits of certain kinds of exercise and discuss psychological approaches to dieting. Writing this out it doesn’t seem so hard does it? Anyone know a producer at Channel 4?
Please share your opinions on the program and this piece in the comments or via social media.
RSS Feed