:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/57498935/shutterstock_139088168.0.0.jpg)
Have you heard? Dark chocolate will do everything from boost your cognition to reduce your cardiovascular disease risk and even help you lose weight! Or so the chocolate science hype machine will tell you.
Several months ago, we got to wondering how chocolate candy had earned such a powerful health halo. So we dove into the science behind these claims about chocolate and cocoa to find out more.
In an original Vox analysis, we discovered that food companies like Nestlé, Mars, Barry Callebaut, and Hershey’s — among the world’s biggest producers of chocolate — have poured millions of dollars into scientific studies and research grants that support cocoa science. Of the 100 Mars-sponsored studies on cocoa, chocolate, and health, 98 had conclusions that were favorable to the candy maker in some way.
That’s an uncannily high number. And it raises questions about the quality of the studies, given that Mars and other chocolate makers can use the positive findings to market their products. Industry-sponsored studies are more likely than independent research to yield conclusions that favor the funder’s products.
:format(webp):no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/9456451/chocolate_graphics3.jpg)
In our review of the research, we found studies that were well-designed, well-executed, and that produced seemingly reliable results. (This was particularly true for the science on cocoa’s effects on blood pressure.) But some of the other claims don’t stand up as well when you look closely at the evidence.
One study in particular about cocoa staving off cognitive decline jumped out at us because it had sparked a small fracas on PubMed Commons, a site where researchers can comment on published studies. Several researchers took the time to critique everything from the study’s design and statistical analysis to how it was reported in the journal where it was published, Nature Neuroscience.
This Mars-sponsored study, led by researchers from Columbia University, was published in 2014. The researchers had wanted to test whether taking cocoa supplements might enhance a region of the brain called the dentate gyrus that deteriorates with age and is associated with age-related memory loss. They concluded that cocoa supplements — particularly the micronutrients called flavanols in them — can indeed boost cognition in older adults.
The research didn’t come out of a vacuum. Previous studies, particularly those focused on aging in rats, suggested flavanols might prevent cognitive decline. But upon closer examination, it became clear that this particular study was very small and preliminary — and that there were several problems with its design that made its results unreliable. That didn’t stop the chocolate hype machine, though. The paper was trumpeted by the Columbia University press office and large media outlets as more evidence that cocoa and chocolate can fight Alzheimer’s.
Ultimately, the study shows how scientists and the media have seized upon the narrative that chocolate is a health food — even when only the thinnest evidence supports the wishful claim.
The cocoa study was short, small, and focused on narrow outcomes that don’t matter to the real world
Before we dive into what made this Nature Neuroscience study suspicious, let’s look at what it was about. The researchers randomly assigned 37 people to one of four groups for a period of three months:
- A group that got a high daily dose (900 mg) of cocoa flavanol supplements as well as one hour of aerobic exercise four times per week
- A group that got the same high dose of cocoa flavanol supplements but without the exercise
- A control group that got a low dose of cocoa flavanols (10mg) with the one hour of aerobic exercise four times per week
- Another control group that got the low cocoa flavanol dose but without the exercise
So basically, the study participants either got a lot of cocoa flavanols or not, and added regular exercise to their lifestyles or not.
The researchers wanted to test whether cocoa flavanol supplements might stave off cognitive decline in the dentate gyrus region of the brain, which is associated with age-related memory loss. They also wanted to see if exercise had any effect on memory, since previous studies had suggested it might.
In the study, they found that exercise had no impact on brain function — but cocoa flavanols did. “Dietary cocoa flavanol consumption enhanced [dentate gyrus] function in the aging human hippocampal circuit,” they concluded. They also made extremely bold statements in the paper, even suggesting that the effects they saw in the high-flavanol group demonstrated that cocoa could reverse age-related memory decline by as many as three decades.
Columbia University’s newsroom touted the research as demonstrating that “dietary flavanols reverse age-related memory decline.” The research was then picked up by media outlets, including the New York Times, which trumpeted chocolate — not just cocoa dietary supplements — as a memory aid.
But here’s the thing: The study never actually proved that cocoa supplements, and especially not chocolate, could prevent memory decline. It was too small, too narrowly focused, and too short-lived to tell us anything important about real memory loss with aging, said Henry Drysdale, a doctor and fellow at Oxford University’s Center for Evidence-Based Medicine.
To track memory decline, the main outcomes the researchers used over a 12-week period were an fMRI test that looked at increases cerebral blood volume, as well as a cognitive function test — the Modified Benton — which was developed at Columbia to measure dentate gyrus function. The researchers who validated the test found that people’s performance on the ModBent worsened with age, so they had reason to believe that this test would be a good marker of whether flavanols could make a difference here.
“Saying if you eat cocoa supplements now you’re going to have better memory in three months is not relevant to real-world [age-related memory decline],” said Drysdale, who co-founded Oxford’s COMPare Trials project which examines the quality of clinical trials.
If you really want to answer that question, you’d run the trial for several years and you’d need a group of study participants that’s bigger than 37 people. Instead of only tracking the study participants’ brain waves in an MRI machine (which is not a measure of cognitive ability), or using an object recognition task (the ModBent) to test memory, you’d also want to measure outcomes that matter in people’s lives, like, whether those taking cocoa could remember what they did that morning or that they had a doctor’s appointment next week better than the people who didn’t take the cocoa, Drysdale added.
This trial only demonstrated that supplements seem to enhance brain function over a period of weeks, and only according to a very specific (and not very widely used) test of cognitive function. That is far from valid proof that cocoa is a memory enhancer.
The researchers did other things that made the results unreliable
Drysdale and other researchers who were not involved with this study also took issue with it for much nerdier reasons. There are problems with how the study was reported that made its results less likely to be reliable — and even less worthy of the hype.
For one thing, the published version of the study looks different from what the researchers originally said they’d set out to do for this trial.
To understand why this matters, let’s step back for a moment.
Before researchers embark on clinical trials, they're supposed to name (or “pre-specify”) which health outcomes they're most interested in on a public database, like ClinicalTrials.gov.
For an antidepressant, these might include people's reports on their mood, or how the drug affects sleep, sexual desire, and even suicidal thoughts. Researchers then group the outcomes into “primary” and “secondary” categories — the primary outcomes being the ones they think are most important — and describe precisely how and when they are going to measure these things.
Scientists are then supposed to broadly stick to this plan when they run their trial, and report on their findings in a journal. If they deviate from their plan, they need to be transparent about it and explain why they did so in the final journal article.
The idea is that researchers won't just change their plans along the way, or publish positive or more favorable outcomes that turn up during the study, while ignoring or hiding important results that don't quite materialize as they were hoping. (That’s a sneaky practice called “outcome switching,” and it’s a big problem in science.) Following these steps also enhances the chances that the findings researchers report on are real, not the result of tweaking a study’s design to get splashier conclusions.
But this didn’t happen in the case of this cocoa study.
ClinicalTrials.gov has a handy version control function that lets you see all the changes that were made to a clinical trials registry over time. It shows that the researchers for this cocoa study changed their outcomes over time, and also failed to clearly pre-specify them before starting the trial. They then didn’t report about the changes they made in their final study, which was published in the prestigious journal Nature Neuroscience.
For example, if you look at the earliest version of their ClinicalTrials.gov report, from 2010, the researchers stated that the primary outcome they were interested in was an fMRI test that measures cerebral blood volume. The secondary outcome they were going to look for was “neurocognitive function” — but they didn’t say which test they’d use to measure neurocognitive function. In the published trial, the ModBent appeared as a second primary outcome along with the fMRI.
“If you don't pre-specify your method of measurement of an outcome — in this case ‘neurocognitive function’ — you are free to choose, consciously or unconsciously, from a range of possible outcomes,” said Drysdale. “You can then pick the outcome that makes your chocolate look good. That’s not to say authors will always do this with vaguely pre-specified outcomes, but the option is there.” In this case, the researchers settled on the ModBent task as their primary outcome (in addition to the fMRI).
I asked the authors on the study why they failed to fully pre-specify their outcomes, and why they didn’t report all the changes they made in their original plan in the final version of the report, like they’re supposed to do. They said they were new to entering clinical trials data on registries, and that they didn’t realize they had to declare changes they had made to their study design in the final study. Whatever the reason, though, these errors in reporting are likely to make their findings less reliable, said Drysdale.
If you look at the most recent version of their clinical trials registry, it was published in January 2015, three months after they published their Nature Neuroscience article. “So they went back after article was published in Nature and changed their clinical trial registry. There is no mention of this in the trial report,” Drysdale added.
To be clear, this cocoa study is not unique. Hype in research is on the rise, and outcome switching is common — as prevalent in industry-sponsored research as it is in independent academic research. But the paper shows how, consciously or unconsciously, studies can be tweaked and exaggerated in ways that can yield misleading conclusions.
“The bigger concern is that people are trying to do a better job of selling the research itself and not just telling what the straight out answer is,” University of Toronto nutrition researcher Richard Bazinet said. This study only showed that over a period of three months, in a small group, according to a very narrow test that taps a very specific region of the brain, cocoa supplements enhanced cognition. That became “chocolate fights memory decline” — a message Mars surely appreciated.
