Peering over Peer Review

We all like to flatter ourselves about the deep insights that we publish in 'peer reviewed' journals.  Peer review means that a couple of people who (an Editor thinks) are qualified have looked at a submitted paper and said it's OK to publish in the journal in question.

Peer review has important purposes:  done at its best, it can spot errors, help with clarity of expression, and weed out trivial or irrelevant results.  In part this keeps science at least somewhat honest, and can serve to avoid total Old Boy network exclusion of competing ideas.  If we have to have standards for publication, and to judge scientists for grant support, tenure, promotion and the like, this is a reasonable way to do it.

But like anything, it can become an insider's snob system that either still supports the Old Boys or is exclusive, especially of maverick ideas.  As it grew over the years, peer review like any other system, became vulnerable to gaming by savvy investigators.  And because an ever-growing set of journals have to keep cranking out the pubs, the standards simply can't be kept that unbiased quality uniformly high.  Even with no ill intent, reviewers are far too overloaded to pay close attention to papers they review (and the same goes for grant peer review).  And to end all, perhaps, there is a proliferation of online journals some of which are like vanity press: in the pretense (sometimes justified) of avoiding the creativity stifling of peer review, almost anything submitted gets published....with the pretense of peer review in the ability of readers to comment, blog-style.

Meanwhile, free-standing social networking (like MT and other blogs) has become an important source of views and information, and an outlet for skepticism, circumspection, and different perspectives.  It is not entirely trustworthy, because bloggers like anyone else have their perspectives and agendas (but of course so do the peer reviewers that editors choose -- among whom many are bloggers).  But it's not useless and a careful blog-reader can (and should) judge the reliability.

A paper in Nature reviews the recent storm of complaint about what is increasingly being characterized as over-stated (at best), the story on arsenic based life, shows the state of play.  We immediately jumped on this paper here in MT for both scientific reasons and because this seemed more like NASA advertising than carefully presented science.  We think we were justified  in criticizing the story because it was far from a complete demonstration of what the media hype said it was biologically, and because despite NASA-hype it was irrelevant to any serious question about whether there were Little Green Men out there beyond Hollywood.  So in a sense ours was a meta-criticism: not of the primary work, but of the unjustified extension of it as if it had astrobiological implications.

Many others in the blogosphere, probably the most visible being Carl Zimmer over at The Loom, have seized on the paper, and some have noticed problems with the chemistry itself, that we are not competent to have noticed.  We said the paper itself, in Science, seemed reasoned in terms of overclaiming.  Perhaps even Science was vulnerable to the Big Story self-promotion and failed to obtain adequate peer review.

But there's more because the Nature commentary was as much about the legitimacy of blogosopheric reaction as it was about the science itself.  And the expected reaction, in our culture, is immediately to milk this for more self-gain:  (1) NASA and the authors, who have clammed up in the face of this clamor (contrary to their courting of attention initially), want studies to confirm their results....in other words, please pass out more money.  We would oppose that because the work even if well done would have no real 'astrobiology' research.   (2) Nature noted that they encourage online discussion of such issues, that is, use of their blog site and subscriptions to their journal.

Both Nature and NASA are vested interests.   It's hard not to have self-interested motives these days, and perhaps we shouldn't harp on them.  But they do bring to attention changes in the system by which scientific findings gain credence. The blogosphere that needs nurturing in this context is not one controlled by a commercial journal, but the independent, non-vested, free-for-all of the internet, where there is no control for PR-spinning or profit motive. 

The blogosphere is peer review, of a heterogeneous kind but perhaps not so much worse than stuffy professional expert-controlled peer review.  One need not pee over peer review to ask how more democratic and openly rather than covertly the social wheels are turned to decide what should see the light of day.