Junk Science: The News Path of Least Resistance

This reporter fooled millions into thinking chocolate helps weight loss. This is why scientific literacy (including statistics) and critical thinking are such important skills. I mean, getting tricked into eating more chocolate than you normally would "because diet" isn't a bad thing—chocolate fucking rules—but because so many other nutrition news articles and diet fads are based on equally shaky footing.

If you're link-phobic or pressed for time, here's the most important part of the article:

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.

Whenever you hear that phrase, it means that some result has a small p value. The letter p seems to have totemic power, but it’s just a way to gauge the signal-to-noise ratio in the data. The conventional cutoff for being “significant” is 0.05, which means that there is just a 5 percent chance that your result is a random fluctuation. The more lottery tickets, the better your chances of getting a false positive. So how many tickets do you need to buy?

P(winning) = 1 - (1 - p)ⁿ

With our 18 measurements, we had a 60% chance of getting some“significant” result with p < 0.05. (The measurements weren’t independent, so it could be even higher.) The game was stacked in our favor.

And yet, not a single media outlet dug enough to find this out. Bohannon explains:

When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted.

...

Shape magazine’s reporting on our study—turn to page 128 in the June issue—employed the services of a fact-checker, but it was just as lackadaisical. All the checker did was run a couple of sentences by me for accuracy and check the spelling of my name.

So if it sounds too good or bizarre to be true, it probably is.