Friday, December 17, 2010

Peering over Peer Review

We all like to flatter ourselves about the deep insights that we publish in 'peer reviewed' journals.  Peer review means that a couple of people who (an Editor thinks) are qualified have looked at a submitted paper and said it's OK to publish in the journal in question.

Peer review has important purposes:  done at its best, it can spot errors, help with clarity of expression, and weed out trivial or irrelevant results.  In part this keeps science at least somewhat honest, and can serve to avoid total Old Boy network exclusion of competing ideas.  If we have to have standards for publication, and to judge scientists for grant support, tenure, promotion and the like, this is a reasonable way to do it.

But like anything, it can become an insider's snob system that either still supports the Old Boys or is exclusive, especially of maverick ideas.  As it grew over the years, peer review like any other system, became vulnerable to gaming by savvy investigators.  And because an ever-growing set of journals have to keep cranking out the pubs, the standards simply can't be kept that unbiased quality uniformly high.  Even with no ill intent, reviewers are far too overloaded to pay close attention to papers they review (and the same goes for grant peer review).  And to end all, perhaps, there is a proliferation of online journals some of which are like vanity press: in the pretense (sometimes justified) of avoiding the creativity stifling of peer review, almost anything submitted gets published....with the pretense of peer review in the ability of readers to comment, blog-style.

Meanwhile, free-standing social networking (like MT and other blogs) has become an important source of views and information, and an outlet for skepticism, circumspection, and different perspectives.  It is not entirely trustworthy, because bloggers like anyone else have their perspectives and agendas (but of course so do the peer reviewers that editors choose -- among whom many are bloggers).  But it's not useless and a careful blog-reader can (and should) judge the reliability.

A paper in Nature reviews the recent storm of complaint about what is increasingly being characterized as over-stated (at best), the story on arsenic based life, shows the state of play.  We immediately jumped on this paper here in MT for both scientific reasons and because this seemed more like NASA advertising than carefully presented science.  We think we were justified  in criticizing the story because it was far from a complete demonstration of what the media hype said it was biologically, and because despite NASA-hype it was irrelevant to any serious question about whether there were Little Green Men out there beyond Hollywood.  So in a sense ours was a meta-criticism: not of the primary work, but of the unjustified extension of it as if it had astrobiological implications.

Many others in the blogosphere, probably the most visible being Carl Zimmer over at The Loom, have seized on the paper, and some have noticed problems with the chemistry itself, that we are not competent to have noticed.  We said the paper itself, in Science, seemed reasoned in terms of overclaiming.  Perhaps even Science was vulnerable to the Big Story self-promotion and failed to obtain adequate peer review.

But there's more because the Nature commentary was as much about the legitimacy of blogosopheric reaction as it was about the science itself.  And the expected reaction, in our culture, is immediately to milk this for more self-gain:  (1) NASA and the authors, who have clammed up in the face of this clamor (contrary to their courting of attention initially), want studies to confirm their results....in other words, please pass out more money.  We would oppose that because the work even if well done would have no real 'astrobiology' research.   (2) Nature noted that they encourage online discussion of such issues, that is, use of their blog site and subscriptions to their journal.

Both Nature and NASA are vested interests.   It's hard not to have self-interested motives these days, and perhaps we shouldn't harp on them.  But they do bring to attention changes in the system by which scientific findings gain credence. The blogosphere that needs nurturing in this context is not one controlled by a commercial journal, but the independent, non-vested, free-for-all of the internet, where there is no control for PR-spinning or profit motive. 

The blogosphere is peer review, of a heterogeneous kind but perhaps not so much worse than stuffy professional expert-controlled peer review.  One need not pee over peer review to ask how more democratic and openly rather than covertly the social wheels are turned to decide what should see the light of day.

4 comments:

James Goetz said...

How about "Peer Reviewing Peer Review"?:)

"Testing evolutionary just-so stories about Darwinian adaptive selection in Nature is another instance where theory ideology comes up against the facts, or lack of available facts. Arguments are heated, issues basically never settled."

I support speculation about adaptive selection scenarios, but it needs to be taken with a grain of salt.

occamseraser said...

@jg
c'mon, grow a corpus callosum and go read Losos. Adaptation is alive and well.

Re: peer (sic) review, ANYTHING can get published today in a so-called peer-reviewed journal. Authors shop around: submit, reject, submit, reject, submit, accept, press release. Articles I've rejected from various journals almost always pop up somewhere else in essentially the same hideous state I first saw them. But I keep trying to review mss. fully and honestly, and do my best to offer constructive tips to improve a ms. in a way that might salvage it (from my perspective, to be sure). The number of crappy journals desperate for words has never been higher; add in anything-goes e-journals and ...barf. I also get emails every week to contribute -- for a price, of course -- to "special" editions and newly spawned pseudojournals out to make a quick buck. And the noise-to-signal ratio in most blog comments (MT excepted, of course) is VERY high and full of mis/disinformation. end of rant 1

What I find especially troubling is the collaboration and coordination between hi-profile journals and PR machines. Darwinius (PLoS) and Ardipithecus (Science) spring to mind as cases wherein peer-review was sacrificed on the altar of rushed and breathless scientific promo "journalism". It's hard somedays not to become a grouchy cynic. eor 2

Ken Weiss said...

Occam, you're right on the mark. The idea and oft-invoked phrase 'peer review' is a kind of self-congratulatory attempt at self-justification....even if it is important to have something like that. This is our culture: science as industry.

But on the other hand we're a bourgeois society and we have to work for a living and this is certainly an understandable way for things to go. The x-y-z complex (military-industrial, healthscience-industry, science-media industry, etc) is how we work as a society.

Going public with show-science to lobby for one's ideas or for money is not new. Boyle did it. Pasteur did it. Edison did it. Lander does it. Dawkins does it. And lots of paleo anthropologists do it.

Ken Weiss said...

Some (like us) are bad at this kind of self-promotion, but as anthropologists we need to recognize that it's part of the way our culture works. Some justify it, others bemoan it but play the game anyway (emics vs etics).

But it's also part of our culture to have vocal opposition as one of the ways we titer the misrepresentation launched by self-interest.

There is also the fallibility of science and scientists. Few of us like to, or feel we can have careers and still, acknowledge what we don't know. The rewards are all for claiming what you do know.

And a huge fraction of our society is made up of far worse parasites than university scientists...