We thought we hammered an Israeli study on artificial sweeteners pretty hard in our Sept 18th Dispatch article Israeli study on sugar substitutes is complete bullsweet.
Maybe we did, but we were seriously out-hammered by Matt Raymond s piece Of Mice and Media: A Credulous Response to an Iffy Sweetener Study.
Raymond, the Senior Director of Communications at the International Food Information Council in Washington, is not the person you want to have dissecting your study when it is of questionable quality something that the group at the Weizmann Institute of Science may have found out.
While ACSH s Dr. Josh Bloom focused solely on the questionable chemical premise of the study, Raymond goes considerably further. He scathingly points out both the faults of the study and the bias of the media in covering sensationalistic, but flawed studies.
Diving into the guts of the Israeli study, he pulls no punches in explaining something that we at ACSH harp on constantly faulty studies that lead to faulty conclusions.
Here are some universal study flaws that come up time and time again in studies of all kinds:
- Inadequate study populations: The researchers...divided 20 mice into five groups: One was given plain water, one was given sugar water, and three were given water sweetened by either saccharin, aspartame, or surcalose. If you're keeping score, that's a grand total of four mice in each group from which to discern significance.
- Methods of data collection: consumption patterns were self-reported, which limits the reliability of the data. This is what's known as a "cohort" study, a type whose findings are less reliable than randomized control trials, the gold standard of scientific research
- Confounders and lack of controls: The study corrected only for body mass index, but for no other lifestyle factors such as quality of overall diet, smoking, alcohol consumption, or sedentariness. As you will find out below, when you adjust for these confounding variables, the supposed causation disappears like a wisp of smoke in a hurricane.
- Correlation vs. causation: The study concludes by observing that the increase in [artificial sweetener] consumption coincides with the dramatic increase in the obesity and diabetes epidemics. Coincides is an interesting choice of verbs, because the noun form of that word is coincidence."
Raymond could hardly have chosen a better (or funnier) example of what happens when these are incorrectly linked. He says, If that's what you base your diet on, then you'd better stop eating cheese, because there's an even closer correlation between le fromage and the risk of strangling to death in your bed sheets.
Raymond discusses what happens when a properly designed and conducted trial investigated the relationship between diet soda and weight gain (or loss). The results are quite different.
He says, 641 normal-weight children who were randomly assigned a daily sugar-free or sugar-sweetened beverage for a period of 18 months. The result: [N]oncaloric beverages reduced weight gain and fat accumulation in normal-weight children. It s hard to get more reliable results than a double-blind, placebo-controlled trial like this one.
Raymond also discusses media bias, and how it contributes to the barrage of misinformation that we are routinely bombarded with: Media Coverage of Medical Journals: Do the Best Articles Make the News? The answer is a resounding no. In short, major media organizations (in this case, the five biggest U.S. newspapers) have a systematic bias that leads them to devote heavier coverage to less reliable observational/cohort studies, and far less attention to the more rigorous randomized control trials. Man bites dog sells papers.
Although Raymond s piece is rather long, it is fairly easy to understand, and is an excellent educational tool that demonstrates the shortcomings of bad studies, their negative impact, and why so much published science is junk.
It is well worth the read.
