Monday, 30 June 2008

Peer-review problems

The peer-review process only developed after World War II, but it is now almost universal. Jerry Kirkpatrick suggests that while it has "perhaps improved the accuracy and reliability of conventional [scientific' research", it has done so at the price of innovation.


  1. I don't produce publications for peer review myself, however I do read (consume) some of those materials in certain areas of interest to me.

    I would like to point out some misleading errors in Jerry Kirkpatrick's suggestions:

    Jerry said...
    Peer review is the process by which millions of dollars of government money are handed out to researchers in medicine and the physical sciences.

    This is not what peer review is defined as I understand. I think Jerry is confused between government funded research and the process of selecting articles regarded as high quality to be published by a (private) publisher, ie, the reviewing process is carried out by a selected group of outstanding peers and expertise in that domain, be it Physics, Economics, Medicines, and so on. One such private publisher, ie, a non-government one is Elsevier, which they publish journals in almost every branch of academics, Physics, Economics, Finance, Health/Medicine, Computing, Engineering and so forth. I haven't counted but a rough guess put it at perhaps over thousands different journals (based on the numbers shown on the link cited above for journal names under A only). Any peer reader here on this thread can review my guess and correct me if I am wrong.

    Jerry said...
    Peer review, a “blind” process in which the names of author and evaluator are concealed from each other, requires two or three so-called peers to read a paper or proposal to judge the quality of actual or proposed research before acceptance.

    Now Jerry defines peer review process correctly here, which he didn't in his first attempt (I highlighted that in my first paragraph). Now, I would like to point out that Jerry is wrong to state that reviewers are being concealed. Perhaps in the 1960s , 1970s, etc, but today in the age of internet, as far as I know, the reviewing panels are not concealed at all. Their names are known to potential authors out there who want to submit a paper for possible publication. For example, Elsevier has a journal called Academic Radiology, which listed all the names of the reviewers. Just click on the link and see it for yourself. I believe that all the thousands other Elsevier journals do the same by listing the names of the reviewers.

  2. More errors from Jerry Kirkpatrick:

    Jerry said...
    Medical researcher and long-time critic of peer review, David Horrobin,... has perhaps improved the accuracy and reliability of conventional research published in medicine, but it has done so at the price of innovation.

    No, this is wrong. An author can submit his/her article to multiple publishers in the same domain. For, example, an author whose writing articles in the technology/engineering domain, can in fact submit his/her paper to Elsevier, IEEE (largest in technology/engineering) or other publishers in this domain. A submitted paper might be rejected from all of those private publishers (but this is free-market anyway) , it could be accepted by all, which I have seen this before, ie, a single paper is being published in multiple journals (from different publishers, different journal titles, but the same domain, eg Computing or Physics, etc...). It is vital that the publisher (privately owned) selects only what their reviewers think its worth being published. The quality of what they published must be maintained at a high level or otherwise their readership drops and they lose market (free-market anyone?). So, there is competition out there by private publishers to get the best.

    Jerry said...
    It was the growth of government involvement in education and, especially, the government’s lavishing of money on research that called for the blind-review screening process.

    Jerry is being dishonest here or otherwise being incompetent for not checking his facts before writing his misleading blog post. The government does not dictate to private publishers as the likes of Elsevier, IEEE, APS (American Physical Society), SIAM (Society for Industrial and Applied Mathematics) and many others. Perhaps the government dictates to its own non-popular publishers, but they don't have any influence on private publishers on what they choose to publish.

  3. That's a somewhat misleading paragraph there, PC.

    The article itself concludes that "the real problem of peer review is the severely hampered market in scholarly research" and asks, "What would a truly free market in scholarly research be like?"

    The problem is not peer review or other aspects of 'science', but the deep-reaching involvement of government in all areas of academia and the distorting effect that government funding has on the scientific process.

  4. Robert Winefield1 Jul 2008, 04:33:00

    The peer review system extends past
    journals and into the Grant application review system.

    The US NIH grants are fully peer reviewed and there is an 'art' in getting a proposal past them.

    Basically you need to make sure that your grant is sexy enough in terms of the topic and the techniques you use.

    So when breast cancer research is in, prostate research cancer is generally on the back burner. That is why the survival rate for breast cancer exceeds that for prostate cancer and you may read into that statistic what you may.

    And as Craig Venter will attest, if you happen to piss off the higher ups in the grant application food chain (by coming up with an innovative way to solve a problem) then your research doesn't get funded.

    As for journal peer review. That waxes and wanes depending on the editorial staff and the direction they choose to take the magazine.

    Science got caught out with the Korean human cloning farce because they like to scoop the competition (Nature & Cell) for big results like the Human Genome project and such. That would happen in the market economy too IMHO.

  5. Peer review does have problems. It does tend to act against young researchers with innovative ideas and new approaches. It also tends to inhibit acceptance of new results when they go against accepted doctrine.

    I worked in the R&D field during my last two postings and ended up concluding that publishing was a waste of time and effort. I made it policy that the first time anyone knew what we were doing was when the science was commercialised or a patent was published. Aside from that my rule was STFU, say nothing, publish nothing, do not boast about what you are doing. Some people had trouble with this. They got discarded. Any budget awarded to a task is to undertake the task, not to talk and boast.

    In reviewing literature in several fields I found that the vast majority of papers are derivavtive and of dubious use anyway. Mostly they are written in pompous language that obscures the nature of the work. Sometimes it is nakedly clear that the authors have limited understanding of what they are writing about.

    Best not to get into the publishing buiness unless you are a publisher- in which case what you are not doing is science...



1. Commenters are welcome and invited.
2. All comments are moderated. Off-topic grandstanding, spam, and gibberish will be ignored. Tu quoque will be moderated.
3. Read the post before you comment. Challenge facts, but don't simply ignore them.
4. Use a name. If it's important enough to say, it's important enough to put a name to.
5. Above all: Act with honour. Say what you mean, and mean what you say.