Monday, 19 January 2009

A new study has found you shouldn’t believe every “new study”

"A new study has found ... "

How many times has that phrase kicked off a new scare story you read over your corn flakes?  Here’s a random selection of the sort of pseudo-academic bullshit you might have read recently: 

  • “A new study has found girls targeted by bullies at primary school are 212 times more likely than boys to remain victims as they get older."
  • "A new study has found that women have better sex with wealthy blokes. "
  • "A new study has found people who sleep less than seven hours a night appear to be almost three times as likely to catch a cold as those who sleep eight hours or more."
  • "A new study has found that men who were programmed in the womb to be the most responsive to testosterone tend to be the most successful financial traders. "
  • “A new study has found women with higher levels of oestradiol, a form of oestrogen, not only look and feel more attractive, they are also more likely to cheat on their partners."
  • "A new study has found Leonardo Di Caprio’s film performance causes cancer in rats."

Yes, grown adults (biologically at least) spend their time and valuable research dollars “studying” this bullshit.  But now a new study says that you shouldn't believe every "new study" that comes out.

    The new study started with an old study that came out last April. British researchers got international publicity when they said they'd found that pregnant women who eat a lot of cereal were more likely to have a boy. The new study took a look at the old study and found that it was faulty.
    "In statistical terms, it's a false positive," said Stanley Young, co-author of the paper in Proceedings of the Royal Society B.
    Young's team had several reasons to question the results of the cereal-eating study, but one of their major concerns was that the researchers asked too many questions of their participants.
    "Put enough variables into a study and meaningless statistical flukes can arise," Young added.
    "The sad thing is what tends to get headlines is the most dramatic or the scarier findings," said CBS4 Denver Medical Editor Dr. Dave Hnida.

The problem is not just too many questions, but too little understanding. Too often, researchers are mistaking correlation for causality.  Where real knowledge uncovers and and explains causal connections, pseudo-knowledge simply discovers accidental coincidences and leaps to unproven conclusions .

Consequently, the “studies” you read over your breakfast cereal are likely to make you less, rather them more knowledgeable.  And that’s a fact.


  1. "And that’s a fact."

    Really? Do you have a study to prove that? :)

  2. It won't be too long before some academics somewhere, published a study of how long to reach an ejaculation during a wanking session.

    I am sure that such study would be very useful.

  3. Most researchers don't actually claim that their studies are definitive. They mostly show that there is a possibly a link, and that further study is warranted.

    If you think that's self serving, then you have to wonder about the media who beat up the research into 'fact' and publish the lurid stories - just to scare you into selling more newspapers.

  4. We have these studies because these people are paid with tax dollars. No sane individual or corporation would ever do or publish studies of these kind (except the New Market Business Association).


1. Commenters are welcome and invited.
2. All comments are moderated. Off-topic grandstanding, spam, and gibberish will be ignored. Tu quoque will be moderated.
3. Read the post before you comment. Challenge facts, but don't simply ignore them.
4. Use a name. If it's important enough to say, it's important enough to put a name to.
5. Above all: Act with honour. Say what you mean, and mean what you say.