Thursday, October 25, 2012

Study Shows Studies Show Nothing [Updated]

Guest post by Nick Hubble of Money Morning Australia

If you’ve ever wondered how a study can show something that just can’t be true, or how studies can completely contradict each other, we’ve figured it out. With a little help of course. After today’s post, I hope you never believe another ‘study.’

Our heartfelt congratulations go out to a computer programme called Mathgen. A mathematics journal provisionally accepted its study for publication.

Wait, ‘its’ study?

Yes, that’s right. These days a computer programme can write an academic paper about mathematics. Then get published in academic journals like‘Advances in Pure Mathematics,’ as this one did.

And you thought those computer programs dominating the stock market were smart!

So what was the paper Mathgen wrote about? Here’s the abstract, which describes it:

‘Let ρ = A. Is it possible to extend isomorphisms? We show that D’ is stochastically orthogonal and trivially affine. In [10], the main result was the construction of p-Cardano, compactly Erdös, Weyl functions. This could shed important light on a conjecture of Conway-d’Alembert.’

If you’re confused, that’s sort of the idea.

Only a mathematics academic could decipher that abstract, because point of fact it’s completely meaningless—and intended to be so. You see, Mathgen creates papers by combining random nouns, verbs, numbers, symbols and the rest of it.  It spits out something that makes grammatical sense, not that you’d know it, but is completely devoid of any meaning.

The formatting is said to be nice, though.

Once the paper is randomly generated and submitted for the academic journal’s review, the academics safeguarding the gates of science and knowledge read the paper and figure it must mean something.

That’s how the paper gets past the peer review process. The same process that keeps climate change science squeaky clean, by the way. Here’s what the anonymous peer reviewer wrote about Mathgen’s bizarre creation:

For the abstract, I consider that the author can’t introduce the main idea and work of this topic specifically.

Maybe that’s because there is no main idea. No ideas at all, in fact.

Anyway, once the academics of the peer review process give the paper a once over and decide it’s fine to publish in their illustrious journal, the valuable and useful knowledge in the paper is disseminated around the academic world. That will probably never happen to Mathgen’s paper because the joke was exposed before the journal was finalised.

If all this makes you chuckle and shrug, consider that it’s the norm in academic publishing. A similar computer program managed to get an article about postmodernism published in a Duke University journal. And even when people run coherent scientific experiments (with real people) the results have a habit of being suspect too.

Many studies can’t seem to be replicated these days. Meaning, if you ran exactly the same experiment, you wouldn’t get results that confirm the study’s findings. According to one science journalist, 47 of the top 53 most important cancer studies can’t be replicated. They might be completely wrong, and yet we base modern research on the assumption they are right.

To be clear, for any sceptics, the Mathgen paper is a true ‘gotcha’ moment. It wasn’t about the fact that a paper can be written by a clever computer program. It wasn’t about anything. It was complete gibberish. But it did show the fact that academic journals are … academic. Let’s hope nobody reads them.

Unfortunately, finance and economics journals actually do get mentioned in the real world. In fact, their conclusions often determine public policy. Politicians hurl studies at each other proving their opinion.

Luckily for economists, it’s very difficult to disprove an economics study. You never know the ‘counterfactual’ — what would have happened. But if maths and science are corrupted, you’d think economics is corrupted twice over.

So the next time you read ‘a study has shown,’ you can disregard the end of the sentence.

Regards,
Nick Hubble
Editor Money Morning Australia

This post first appeared at Money Morning Australia

UPDATE:  The author of the bogus paper blogs the story here.  And compares this to

Alan Sokal’s 1996 hoax, in which Sokal, a physicist, got the cultural studies journal Social Text to accept a parody article which identified physics and physical reality as a social construct.

3 Comments:

Anonymous Falafulu Fisi said...

Nick...
But it did show the fact that academic journals are … academic

Nope! Wrong. Academics is what drive advancement of civilization.

Sure, there some gibberish and nonsense academic publications that have made it to peer reviews, but those slips don't mean that academic researches are nonsense.

Here is why? We all enjoy our modern electronic gadgets of today, as IPad, cell-phone, computers, etc,... Where do you think that the knowledge used by engineers/scientists to build those components & computer-chip used today in modern technology come from? They all come from academic journals.

10/25/2012 01:17:00 pm  
Anonymous Steve D said...

'You see, Mathgen creates papers by combining random nouns, verbs, numbers, symbols and the rest of it.'

That's basically how reviewers think I write my papers as well. Go figure.

But I agree with the previous comment. In my field (biochemistry) about 90% of the studies are useless, bogus or unrepeatable (and biochemistry is the hardest of sciences). That means 10% of them aren't. The trick is figuring out which is which.

10/26/2012 06:34:00 am  
Blogger thor42 said...

I'm sure that the "policy" of the Labour Party is generated by a bot like this, written in some god-awful programming language.

Just like the mentioned paper, Labour "policy" is devoid of content, meaning or sense.

10/27/2012 04:49:00 pm  

Post a Comment

Respond with a polite and intelligent comment. (Both will be applauded.)

Say what you mean, and mean what you say. (Do others the courtesy of being honest.)

Please put a name to your comments. (If you're prepared to give voice, then back it up with a name.)

And don't troll. Please. (Contemplate doing something more productive with your time, and ours.)

<< Home