Monday, April 11, 2011

What is peer review? What should it be? Does it work?

Scientific fraud
There was a recent discussion on our favorite medium, the BBC (in this case, the World Service on radio) about the problem of scientific fraud.  How common is it?  How harmful?  How does it get through the peer review process?

Science depends more than perhaps any other area of human societal life, on honesty.  That's because we each can only do our own experiments, and can't possible do all that was done before that set the stage for what we attempt to do.  Fortunately, real fraud seems to be very rare in science.  It occurs, but is usually punished when caught, by strong sanctions.  It is very serious indeed, because investigators can spend precious time, effort, and resources pursuing ideas suggested by published reports of important findings or methods.  The findings are widely cited and used in support of the new ideas in, for example, grant applications.  In anthropology, the Piltdown fossil fraud was accepted by many scientists for decades, and built into texts and other frameworks for analyzing human evolution.  The recent Korean cloning fraud misdirected many labs into wasteful cloning experiments.

Major fraud is caught in various ways, perhaps most often by others trying, and eventually failing, to replicate an important result.  Minor fraud may be more common--we know about the major cases because of the problems they cause, but few care about minor results.  And dissembling and exaggeration and self-promotion are rife.  The public may not be aware of it, but scientists usually are (and often justify doing it themselves because 'everybody does it' and 'you have to, to get funding or to be published in Science.')

Puffery has consequences similar to fraud in that skillful hyperbole establishes fads, and most of us, desperate for attention and funding and so on, eagerly jump on band-wagons (especially when a new toy, like fMRI, gene expression arrays, and the like are available).  Overall, it is likely that puffery in fact causes more damage by diverting funds and energies in areas less promising than is believed.

Peer review
Peer review is the system that supposedly keeps up the scientific standard.  An article is submitted to a journal, and if deemed relevant, is sent to 2 or 3 experts to judge.  They are to judge quality and appropriateness, importance or novelty, and find factual errors and so on that may be in the reported work.  But, of course, these peers may also be rivals, or (almost always) too busy to pay close attention to the paper they're sent.  Nowadays, with the huge proliferation in the numbers of frenetically competing investigators, and new journals, and tons of online 'supplemental' information allowed, it is nearly impossible to maintain a standard.

When reviewers find unclear statements, things not well explained, or actual errors, then they do the original authors a huge positive service by improving the paper before it is published. 

Peer reviewers were never asked to find fraud, however.  They are hardly ever in a position to do that anyway.  They, like all of us, must in practice assume the honesty of the authors.  So peer review doesn't generally find fraud.

One thing the system of peer review does that we find increasingly irritating--and try to avoid ourselves--is empower the reviewer to insist that the author(s) modify the paper in ways that, when you get right down to it, would make the paper more like what the reviewer would write if s/he were the author.  This leads to often extensive modification in ways that are forced rather than natural, and not what the authors actually wanted to say.  The authors, anxious to get their work published and move on, will then litter their revision with every kind of cumbersome citation, caveat, and implicit rhetorical bow to the reviewers--so they can tell the Editor they responded to all of the comments.

More and more, we feel, the reviewer should find mistakes and unclear aspects of papers, but just let the authors have their own conclusions and interpretations.  If the authors do a bad job, then readers of the paper will do better work, ignore the paper, or whatever.  But if the authors do a good job--including a good job of presenting their case in a style they wish to use--then the paper they wrote is better off unadulterated. 

Peer review is valuable, but could be improved if reviewers were instructed to stick to the important issues.  Let the poets in science out of the lab!


Holly Dunsworth said...

Thanks for posting this!

Texbrit said...

Maybe scientists can just self-publish their work on this Interweb thing I keep hearing about. Then they get to post it the way they want it, and reviewers have to make THEIR comments in the light of scrutiny as well! A great un-sheather of agendas that might be!

Texbrit said...

Said differently - part of the problem is that reviewers (and writers!) have hidden agendas. But another problem is that there aren't enough "avenues" for getting your work out there. If it has to be "Science" magazine, then Science has too much control of scientific output and content.

Ken Weiss said...

I think that the internet is going to make things more democratic and less constrained. I don't know about the Interweb (never heard of it), but blogs like MT could be one of many forms of less constrained publication--and open to the whole world, not just those who can afford a subscription. But there are some problems.

First, without any sort of peer review it will be even harder to assume any sort of quality check (esp. for errors and proper recognition of the existing literature.

Second, this could proliferate reports even beyond the current overload

Third, there may be even less constraint on length and 'supplemental information'.

Fourth, the web is not as well indexed for searching (though these days perhaps proper keywording will turn up even blogs).

Fifth, departments and institutions will have to start recognizing this form of work dissemination, in tenure and promotion decisions--and to reinstate the idea of passing judgment on quality rather than bean-counting: something bureaucracies (that like to have CYA 'data' to hide behind when lawyers show up).

Sixth, there can be some patent and copyright issues.

But overall, it probably will be healthy for science and academic work in general....maybe also for literature and the arts (though how the creative artists will be rewarded will have to be worked out)

Ken Weiss said...

The prestige journals do have too much control, and there are too many gravitational pulls towards the incremental safe, technologically driven conceptual center.

Actually, there is also a proliferation of new journals, often online-only. But most of them make you pass at least some sort of peer review, and/or pay a fee (vanity press charge?).

There needs to be a settling out, adjustment on how things are indexed or found via keyword searching, and the like.

The best would be a curtailing of the proliferation of the rush to publication, to paper-counting in job evaluations, and to the way in which publications affect grant proposal prospects.

It will take some time, hopefully not too much trauma. But the system is relatively out of control at present.

Arjun said...

Dr. Weiss, do you foresee any "tipping point" which might precipitate methodological revision such as that which you describe? Also, to your point of "bean counting," I have noticed a trend dating over the past 30 years for authors to split papers into multiple publications in order to increase their publication count. Would you consider this a serious issue?

Ken Weiss said...

These are definitely legitimate and important issues. I think that, like economics, there is no way to 'forecast' what will happen (and I say that, reluctantly, as a former meteorologist!).

Systems evolve and we can predict change because of budget issues and overload problems. We are unlikely to get away from a hierarchical, competitive, pyramidal approach, I'd guess, since that is built into our thinking in this country at least.

If, say, China and India start to become wealthier than we, or develop their own technologies and so on, we will have to adapt to that.

Part of the problem is hierarchy in general and the need--I would say the 'middle class' need, since that's who scientists are these days--to have jobs, status, and the like.

A second part of the problem is the entangling and unremitting growth of bureaucracies, which need something to hide behind (to justify their existence), and bean-counting is the safe way for ladder-climbing bureaucrats to do it.

occamseraser said...

Don't worry, if your paper is rejected by a respectable, truly peer-reviewed journal, you can always get it stuffed into PLoS ONE!

As a former editor of a paleoanthro journal, I'm still sincerely grateful for the hard work of many many reviewers, most of whom took on the reviewer burden with few expectations except the hope that their comments might improve a study -- or in some cases save the authors from serious embarrassment had their work not been scrutinized and revised. Sure, sometimes reviewers have agendas, but editors have the responsibility to account for that in final recommendations to authors. For young professionals new to the publishing biz, constructive and creative recommendations on how to recast data and results can be part of a genuine learning process, and that ain't poetry.

If there's a problem, it's the existence of too many journals filled with junk and throw-away studies of trivial significance. The proliferation of web-only "journals" hungry for words of any kind has only served to exacerbate the problem.

With the fed cuts on the horizon to NIH and NSF, there will be less science, but perhaps no fewer poets.

Ken Weiss said...

I agree about reviewing except to the extent that reviewers are too busy to do it carefully, want to rewrite the authors' paper rather than letting them say their peace, or sometimes simply nit pick.

But we are all so overloaded that this very valuable service is being degraded, not diminishing in potential importance.

The proliferation of vanity journals (like PLoSOne?) allows authors to say their piece without the hassle (if they can pay the pub. fee!), but of course also clutters the landscape.

Blog-based or other kinds of idea-dissemination will probably become more legitimately important,faster, and more fluid, too.

John said...

There are a set of interrelated issues here that need to be explicitly separated.

First, the myth of imprimatur: (1) getting a book published by the "important" publishers means the book is worthy. Given the plethora of crap major publishers routinely and persistently publish, this standard is not to be taken seriously. But (2), the same applies to the "major" journals (Nature and Science): they also routinely and unfortunately publish the same crap. Indeed, they seem to be more subject to this problem than are mainstream journals within any given field.

Second, the myth of "peer review". Here the problem is one of quality control of the reviews, not that which is reviewed. Many reviews are short, biased, stupid, and useless. But why is that surprising? There is no academic reward for reviewers (indeed, many will opine as to how it is ridiculous to spend any time on reviews).

Given that print (i.e., the use of desiccated tree products as a dissemination device) is no longer a vehicle (or restraint), my preferences (i.e., everything is published electronically):

(1) All reviews are published with both the original and the emendated article. This approach has many positive consequences. First, reviewers would receive publication credit for their (sometimes, now) hard work, and the vacuous crap that attends too many reviews would be lost.

(2) The journal editor (or editorial team) decides only whether or not a submitted article should be subjected to review. Once sent out to review, *everything* subsequently is published. This approach provides some quality control over editors.

(3) Every author submits using TeX or LaTeX (available free for virtually every operating system) files, with journals posting the requisite .sty files where some peculiar formats are desired. Among the virtually infinite positives associated with this approach is that WORD will be eliminated as a (almost universal) submission format. Indeed, in theoretical physics and math, LaTeX is the default format. The primary advantage here is that the costs associated with type-setting are eliminated. Indeed, journals are now reduced to what we all already do for free (writing, editing, and reviewing). Which is to say, we do not any non-open, non-free (on the web) journals.

Ken Weiss said...

I don't use LaTeX but have heard that idea before. Your ideas would solve some of the problems as you describe, but would generate so very much data (articles, revisions, reviews) that I doubt many would wade through it.

But it would be more open and less rigidly hierarchical.

A main underlying issue is the volume and the various uses publication serves, including profit for journals, career gains for authors, grantsmanship, and (of course!) actual dissemination of scientific results--the main purpose we're all after.