Journalist Johannes Bohannon tells the story of how he crafted an "experiment" that "proved" that eating chocolate helps weight loss--and sold it to millions of readers in the media world. How timely, especially the part about journalistic laziness. Think of how often the techniques employed here been applied by medical equipment manufacturers and others. Excerpts, with my emphasis:
Could we get something published? Probably. But beyond that? I thought it was sure to fizzle. We science journalists like to think of ourselves as more clever than the average hack. After all, we have to understand arcane scientific research well enough to explain it. And for reporters who don’t have science chops, as soon as they tapped outside sources for their stories—really anyone with a science degree, let alone an actual nutrition scientist—they would discover that the study was laughably flimsy. Not to mention that a Google search yielded no trace of Johannes Bohannon or his alleged institute. Reporters on the health science beat were going to smell this a mile away. But I didn’t want to sound pessimistic. “Let’s see how far we can take this,” I said.
I know what you’re thinking. The study did show accelerated weight loss in the chocolate group—shouldn’t we trust it? Isn’t that how science works?
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
It was time to share our scientific breakthrough with the world. We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers. Since time was tight, I simultaneously submitted our paper—“Chocolate with high cocoa content as a weight-loss accelerator”—to 20 journals. Then we crossed our fingers and waited.
Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose . . . emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”
Could we get something published? Probably. But beyond that? I thought it was sure to fizzle. We science journalists like to think of ourselves as more clever than the average hack. After all, we have to understand arcane scientific research well enough to explain it. And for reporters who don’t have science chops, as soon as they tapped outside sources for their stories—really anyone with a science degree, let alone an actual nutrition scientist—they would discover that the study was laughably flimsy. Not to mention that a Google search yielded no trace of Johannes Bohannon or his alleged institute. Reporters on the health science beat were going to smell this a mile away. But I didn’t want to sound pessimistic. “Let’s see how far we can take this,” I said.
I know what you’re thinking. The study did show accelerated weight loss in the chocolate group—shouldn’t we trust it? Isn’t that how science works?
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
It was time to share our scientific breakthrough with the world. We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers. Since time was tight, I simultaneously submitted our paper—“Chocolate with high cocoa content as a weight-loss accelerator”—to 20 journals. Then we crossed our fingers and waited.
Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose . . . emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”
With
the paper out, it was time to make some noise. I called a friend of a
friend who works in scientific PR. She walked me through some of the
dirty tricks for grabbing headlines. It was eerie to hear the other side
of something I experience every day.
The
key is to exploit journalists’ incredible laziness. If you lay out the
information just right, you can shape the story that emerges in the
media almost like you were writing those stories yourself. In fact,
that’s literally what you’re doing, since many reporters just copied and
pasted our text.
Take a look at the press release
I cooked up. It has everything. In reporter lingo: a sexy lede, a clear
nut graf, some punchy quotes, and a kicker. And there’s no need to even
read the scientific paper because the key details are already boiled
down. I took special care to keep it accurate. Rather than tricking
journalists, the goal was to lure them with a completely typical press
release about a research paper.
I felt a queazy mixture of pride and disgust as our lure zinged out into the world.
When reporters contacted me at all, they asked perfunctory questions.
“Why do you think chocolate accelerates weight loss? Do you have any
advice for our readers?” Almost no one asked how many subjects we
tested, and no one reported that number. Not a single reporter seems to
have contacted an outside researcher. None are quoted.
There was one glint of hope in this tragicomedy. While the reporters
just regurgitated our “findings,” many readers were thoughtful and
skeptical. In the online comments, they posed questions that the
reporters should have asked.
The dirty BIG secret is that almost all scientists run their research hoping to prove their hypothesis and set their studies up to do that. The BIGGEST dirty secret is that the media, who studies have shown are about 70% liberals, push any studies that agree with their thinking to the forefront of the man in the street and convince them it is "proven fact" even though it is unproven theory. The real truth is, given enough time, almost all scientific theories turn out to be disproven or superseded by a new theory that becomes the new "truth."
ReplyDeleteThe scientific method is fundamentally flawed. Isaiah Berlins hedgehogs work relentlessly on linear studies designed around an economic agenda.
ReplyDeleteThe writing style is ritualistic and boring.
Most science writers are well trained parrots This article turns over a large rock.
Coincidence that I finished Richard Feynman's "Surely You're Joking, Mr. Feynman" last night. One chapter contains a rehash of a commencement address he gave at Cal Tech. Scientific rigor and integrity of research was his message. There are many inconvenient truths, especially those that don't fit our preconceptions, hypotheses, and world view.
ReplyDeleteNot much of a secret. I love the article, but R. Barker Bausell wrote a book in 2008 about all the things that are done incorrectly in studies and how they affect results. Although the book was mostly aimed at complementary and alternative medicine, everything he said applies to human studies of all sorts. Every science journalist should read it. But they won't! Until then, we'll have false causality, confounding factors, reverse causality, regression to the mean, selection bias, placebo effects, dropout effects, and a host of other errors confusing reporters and distorting results that masquerade as true.
ReplyDeleteIf I had a nickel for every meaningless headline that says.. X factor linked to X disease in humans.... I'd be rich
ReplyDeleteYes this explains the "proof" that six-sigma, EHR, Press Ganey, ACOs,pay for performance, and the rest of the rent-seekers' boondogles are magic bullets for health care.
ReplyDelete