Showing posts with label peer review. Show all posts
Showing posts with label peer review. Show all posts

Monday, February 17, 2014

In search of scientific blunders

The current (March 6,  2014) issue of The New York Review of Books includes a review (paywall) by physicist Freeman Dyson of a new book by Mario Livio called Brilliant Blunders: Colossal Mistakes by Great Scientists that Changed Our Understanding of Life and the Universe. The book provides the story of five famous mistakes by giants of science, Charles Darwin, William Thomson (Lord Kelvin), Linus Pauling, Fred Hoyle, and Albert Einstein, but Dyson writes that after reading the book, he now sees mistakes by great scientists everywhere.  Isaac Newton, James Clerk Maxwell, Gregor Mendel, himself. The thesis is that these great and greatly influential thinkers made major mistakes but essentially because they were thoughtful about it and had good reasons relative to then-current theory, the price in reputation they paid, or that society paid, was slight (unlike military or other real-world blunders).

Darwin
Darwin ignored Mendel and Mendel ignored Darwin.  But who cares?  They did so despite vastly important work that, well, that worked.  In a sense, their mistakes reflect the boldness of their contributions.  Dyson argues that "The greatest scientists are the best losers."

In contrast to these bold thinkers, most modern scientists, caught in the current system, we think, are pressured to play safe and too often that's just what they do.  We have major problems facing us in terms of understanding biological evolution and causation and as we've recently written, we personally think the name of the game today is generally just that: play safe, don't make blunders, because if you do, your continued success in the system is at risk.  No tenure, no grants, no promotions.  (Note, we say 'blunders', that is, mistakes or wrong guesses even if for good reasons, but not fraud or other culpable misdeeds).

We've tried to take the basic statistical basis of modern evolutionary and genomic inference to task for clinging to conservative, established modes of thought rather than addressing what we believe are genuine, deep problems that are not (we feel) amenable to multivariate sampling statistics and their kin.

The proof isn't in the pudding!
The current peer review system is supposed to be a quality control system for grants and publications, so that work is judged fairly on merits alone.  Of course like all large institutional systems it has its flaws.  Some are easy trades for fairness rather than Old Boy-ism, but the system also very strongly and clearly leads to safe conventionalism.  It can accommodate real innovation, but it makes that hard.  But we have at least lived under the illusion that citation counts are an accurate recognition of quality.

Now, a new commentary in Science ("Peering into Peer Review," Jeffrey Mervis) reports that this doesn't seem to be so.  The results of two studies of the citation records of papers reporting the results of citation counts of nearly 1500 studies funded by the National Heart, Lung, and Blood Institute of the NIH between 2001 and 2008.  At least among those that were funded, there is no sense that better priority meant higher impact; the funded projects with the poorest review scores were cited as many times as the studies with the highest scores.  Michael Lauer, head of the division of Cardiovascular Sciences at the NHLBI who carried out the studies, was surprised.
"Peer review should be able to tell us what research projects will have the biggest impacts," Lauer contends.  "In fact, we explicitly tell scientists it's one of the main criteria for review.  But what we found is quite remarkable.  Peer review is not predicting outcomes at all.  And that's quite disconcerting."
In recent years, 8-10% or so of grant applications have been funded, so even grants with the lowest priority scores are in the top tier so it's no surprise if they are all being cited equally.  But, as Mervis points out, the fraction of grants funded was much higher in the early years of these studies, and presumably, if peer review ranking of applications is at all meaningful, there should have been quality differences between the highest and lowest ranked grants.  Another difference, that probably should have made a difference in quality of results, was that the lowest ranked grants generally received less funding.

If citation counts actually reflect on quality, this indicates that a whole lot of good research is not being funded these days -- which of course is no surprise.  However, this study highlights something else about citations, too, if in an oblique way: we're not talking about a lot of citations.  According to the commentary, each publication, whether from a grant that had a top, middle or lower tier priority score, was cited 13 or 14 times.  To us, that's not high impact.  Perhaps the equal paucity of citations should be of more concern than that high and low quality grants are equally cited.  Perhaps this shows the unsurprising arbitrary aspect of grant ratings and reviews, but it also shows something that not many people talk about, which is that most publications are mostly ignored.  This isn't a surprise either, given the number of papers published every year, but it should probably be more sobering than it seems to be.

Or could something beyond these issues be going on?

There is more than bad luck here--much more!
Perhaps if we were really funding scientific quality--innovation, creative thinking, and really bellying-up to core, serious problems, we should expect a strong negative correlation.  The top scores should go to deep, creative proposals that, almost by nature, are likely to fail.  These would not be crank or superficial proposals, but seriously good probing of the most important (and hence truly highest priority) research.  Most of us know very well that being really creative is a sure way to not be funded, and that is widely (if privately) acknowledged: write a safe, incremental proposal, promise major results even if you know that's unlikely, and then use the money to do something you really think is new.

If not a strong negative correlation overall, at least there should be a few really new findings that got lots of citations and recognition, but most top-scoring projects went nowhere. That, at least, could be an indicator that we're funding the best real science.

One wonders how, or if, the Large Hadron Collider would have reported their very large experiment if they had not found evidence for the Higgs Boson--or, are they forcing a positive interpretation out of their statistical data?  We can't judge, but the power of wishful thinking is always there we think a clear negative would have been reported as such.

One can say that much of today's science is going nowhere or littlewhere (GWAS and much else), though the 'negative' results are often safe, highly publishable and lauded results that a big-data study is guaranteed to find. The first few times no real map results were achieved, and we saw that polygenic traits really turned out to be polygenic, we might have had a major positive finding: a theory, even if a very generic one, was confirmed.  Of course, a GWAS study may make a big strike, but more typically, instead of saying 'our GWAS basically found nothing, showing that our idea was wrong and some other idea is needed',  even a big strike-out can be made into a big story as if there is a real lesson learned; minor hits are lauded.  Indeed, empty big data results are usually not viewed as the anathema of a failed theory, but highly published and publicized to justify even bigger studies of the same type.

The kind of busts that regularly result in the safe market that predominates today are not really tests of any theory, and so do not come up with a definitively empty results.  They are not high quality blunders. And even these top-ranked, generally safe studies,  are not reflecting distinctions between priority scores and citations.

If what we want is safe science, in areas where we just need routine work on some new project, even if it's important such as a universal flu vaccine or data on obesity and disease, then we could establish an Engineering Science funding pool and build a firewall between that highly relevant but specific goal-oriented work, and basic science.  Nominally, NSF should be the place for the latter and NIH for the former, but that's not how things are today, because both have become the social welfare net for universities and their faculty.  Good work does get done, but a lot of trivial work is supported even if it's of good technical quality, and the pressure is to do that,while actual innovation is in a sense punished by the conservative nature of the system.

It would be terrific if the very best science tested real ideas that went nowhere.  Then we could encourage development that would lead to more blunders, and the correlated emergence of more Darwins and Einsteins.

Thursday, June 14, 2012

Open access -- its time has come?

The end is nigh?
The rumbling about open access to scientific publications is getting louder.  If you've been out of earshot, here's a brief run-down of some of the recent major events.

A boycott of academic publisher Elsevier began 6 months ago when British mathematician Tim Gower issued a 'call to action' to urge academics to refuse to submit to or review for any Elsevier journal.  He  explained in his call to action why he decided to go public with his anger over the exorbitant profits Elsevier and others are making from his free labor.  To date, 12,000 scientists have joined the boycott. You too can add your name to the list.

The fundamental issue has to do with open access to the results of scientific research, particularly research funded by public money.  Gower's specific objections are that major academic publishers charge usurious prices for subscriptions to single journals, often $1000 or more, that libraries have to buy large bundles of journals, many of which they don't want, in order to get the major journals that an academic library can't do without, and that Elsevier supports attempts to restrict free exchange of information. So, you pay for the research with your taxes, and that includes fees the libraries (and the researchers) pay (again with your taxes) for their subscriptions.  That's double-dipping!

One of many Boycott Elsevier images on the web;
this one's a poster, available here
Three major publishing houses, Elsevier, Springer and Wiley, produce the bulk of the more than 20,000 academic journals published worldwide, and over 40% of the journal articles that appear each year.  This is big business, with a profit margin of 35% or more, according to numerous sources (here or here, e.g.), worth on the order of $12 billion a year.  This profit, more and more academics are coming to feel, depends too heavily on their free labor.  Publishers pay academics nothing to write the papers published in these journals and nothing to review the work that their peers submit, nor come to that, do academics get paid for the work they do to review and screen the grant applications in the first place.  Peer review is the process that is meant to ensure high quality contributions to journals, though it's the highest tier -- and most lucrative -- journals that can get the most well-known academics to do the reviews, and they might be asked to review hundreds of papers a year.

Journal subscriptions already can be prohibitively expensive, and even access to a single paper can cost $30 or more.  Yes, publishing costs money, and if authors and reviewers were paid it would cost even more, but when the work has been funded by public money, and the writing and reviewing is free to the publisher, and then the results sequestered behind a paywall, given the increasingly organized protests, the system is looking more and more unsustainable.

Open access isn't a new issue, of course.  PLoS (Public Library of Science) has been publishing open access journals since 2008, and they quickly became well-respected -- high impact -- places to publish.  One of the journals, PLoS One, will publish essentially anything that is well-written, presents a study done with acceptable methods and that hasn't been published previously; it's the reviewers job to ensure that a paper meets these criteria.  As a result, PLoS One does publish some lesser work, but they also publish some very fine work -- presumably by well-funded researchers who don't want to deal with the hassle and delays of the review system, and its arbitrary nature, or who just want to circumvent the private-gain system.  Someone's got to pay for these journals, either authors or readers: PLoS is a non-profit publisher but they do charge to publish (hmm, where does that money come from?), though to be fair they will waive the fee if necessary.  The other PLoS journals have more stringent standards for publication.

ArXiv is an early adopter of e-print service, online since 1991.  It's a database maintained at Cornell that publishes and provides open access to physics, mathematics and statistics papers, and hosts probably every physics paper published in the last 2 decades.  Open access is old hat to people in the hard sciences.

More recently, a petition from the White House to require open Internet access to taxpayer-funded research reached the magic number of 25,000 signatures and so the idea that publications of publicly funded research results must be freely available online is now officially in play in the upper echelons of government.  Here's evolutionary biologist and co-founder of PLoS, Michael Eisen's reasoning on this issue.  The National Institutes of Health enacted public access policies requiring that scientists who are publishing results of NIH funded research to submit final peer-reviewed journal manuscripts to a publically accessible digital archive (PubMed Central) upon acceptance for publication.  There are currently 2.4 million articles in the archive, and journals can choose to be full participants but most do not or only selectively deposit.  The library, and Faculty Advisory Council, of at least one major university, Harvard, are urging their faculty to submit their work to open access journals, primarily because journals have become prohibitively expensive and this is a way to chip away at the status quo. 

Further alternatives to the traditional publishing model are beginning to appear, as Reuters reported on June 12.  One  is called PeerJ, the founders of which come from PLoS and the research database, Mendeley.  Their website has just come on line.
PeerJ provides academics with two Open Access publication venues: PeerJ (a peer-reviewed academic journal) and PeerJ PrePrints (a 'pre-print server'). Both are focused on the Biological and Medical Sciences, and together they provide an integrated solution for your publishing needs. Submissions open late Summer.
They offer a 3-tier subscription scheme; until September for $99 you purchase a lifetime right to publish a single paper a year in their journals and to freely access any of the papers they publish; for $169 you can publish 2 papers a year and for $259 your publication rights are unlimited.  The fees will go up some at the end of the summer.  The founders hope to hit the ground running, publishing significant papers by well-established researchers immediately. As in PLoS One, papers will be reviewed for scientific validity, not potential impact.

Another open access journal about to come online, eLife, is a joint collaboration of the Howard Hughes Institutes, Wellcome Trust and the Max Planck Institute.  They also hope to be a high-impact journal right off the bat.  Because of the well-heeled backers publication is going to be free, at least initially.

Let's be fair...
Of course, not everyone is a fan of open access, and it's not just the journals that stand to lose who are against it.  Opponents say it's anti-free market, expensive for authors rather than libraries and thus it disadvantages those without grants, as well as scientists in poorer countries, peer review will be scarified and quality will be scarified, or it's just too big of a change, after 350 years of publishing science this way. 

After all, it is also true that until recently there was no way to get your work known to any kind of broad audience other than publishing on paper, individual and library subscriptions.  Even then it took a long time to get the peer review done and the paper in print.  Without centuries of such publication there would hardly be any scientific establishment and much slower progress!

Indeed, everybody has to earn a living, and that includes people who work for Elsevier.  And our pension plans are based on stock which means on profits.  It is not surprising that the status journals work hard to preserve their market-share.  Nor can we be blamed for succumbing to this aspect of the status system, given how rewards are doled out in academia.

But not too fair!
Just because publishers are people with kids to feed, and because they were the only, and an effective, way to disseminate information that made widespread access possible, doesn't mean things can't or shouldn't change.  After all, think of the scribes in monasteries whose work on scrolls and tablets was undermined by Gutenberg and the disrespectful paper inkers.  They had families, too -- well, the Church was their family.

Directly or otherwise, publishers play on your fears: that if you don't publish in their expensive high-visiblity journals, you won't get tenure or a grant.  Making us vulnerable, they make it difficult to resist the game they're playing -- that we let them play be going along with the 'impact' system (maintained by another private, profit-making company) that determines which journals people submit papers to, and how their Deans judge their influence.

Jean MiƩlot, author and scribe
The peer review argument also rings rather hollow.  First, peer review can also mean insider-club exclusion, and we know that innovative ideas have a tough row to hoe.  Second, anyone in academe knows that you can always find someplace to publish your work if you try hard enough (PLoS One is one place).  Peer review may once have served as a guard at the quality gate, but those days were long gone decades ago in terms of serious quality control.  We think there is no serious evidence that e-journals have lower standards -- and indeed, PLoS One, often the whipping boy for this argument, has published some very fine papers, indeed.  And almost anyone who is candid knows that the 'main' journals publish a lot that is weak, sensationalized or over-claiming, and most of the results turn out not to be replicable (as Ionnides said in a paper we've referred to before, published in PLoS Medicine, in fact).  So we need not weep for the poor publishers and their role as Guardians of Quality.

So, given everything, change is hard, always has uncertainties, and we're all in a tangle that keeps costs higher than they need to be. And those not in science are paying the price for it.  They pay for research and expect to reap the benefits.  But that is, to many of us, too indirect and inefficient, with too many in the middle skimming of the top, before the research ever sees the light of day.  But, it does look as though the push for open access is getting stronger, big money is starting to rally behind it, and a consensus is perhaps beginning to build.

Friday, May 25, 2012

You scientist, we want you to get ahead....but not too FAR ahead!

A paper in the June issue of BioEssays is titled "Collegiality and careerism trump critical questions and bold new ideas" and, no, we didn't write it.  The subtitle is "A student’s perspective and solution": the author is Joshua Nicholson, a grad student at Virginia Polytech.  It's a mark of the depth of the problem that it is recognized and addressed by a student, in a very savvy and understanding way.  But of course it's students who will most feel its impact as they begin their careers when money is tight and the old boy network is alive and well. The situation is so critical today that even a student can sense it without the embittering experience of years of trying to build a post-training career.
As students we are taught principles and ideals in classrooms, yet as we advance in age, experience, and career, we learn that such lessons may be more rhetoric than reality.
Nicholson is not the first to notice that the current system of funding and rewards encourages more of the same, not innovation.  Scientists, he notes, are discouraged from having radical, or even new ideas in everything from grant applications to even just expression of ideas.  Indeed, numerous examples exist of brilliant scientists who have said they couldn't have done their work within the system; Darwin, Einstein, and whatever you think of Gaia, its conceptor, but also innovative inventor, James Lovelock, has said the same (and he did so recently on BBC Radio 4's The Life Scientific). Other creative people, in the arts, have felt the same way about universities (e.g., Wordsworth the poet, Goya the painter).

The US National Institutes of Health and National Science Foundation both pay lip service to innovation, yes, but still within the same system of application and decision-making.  Nicholson says that the NIH and NSF in fact admit that these efforts are not encouraging innovation (as those of us who have been on such panels and never seen an original project actually funded--usually the reviewers pat the proposer on the head patronizingly and say make it safe and resubmit).  He blames this, correctly, on the review structure; peer review.  Yes, experts in a field are required to evaluate new ideas, but it is they who are often most unwilling to accept them.

To be fair, usually this is not explicit and reviewers may usually not even be aware of their inertial resistance to novelty.  But Nicholson explains that:
(i) they helped establish the prevailing views and thus believe them to be most correct, (ii) they have made a career doing this and thus have the most to lose, and (iii) because of #1 and #2 they may display hubris [2–4, 9, 10]. If, historically, most new ideas in science have been considered heretical by experts [11], does it make sense to rely upon experts to judge and fund new ideas?
He concludes that a student looking to build a career therefore must choose between getting funding by following the crowd and doing more of the same, or being innovative but without any money... that is, driving a taxi.

He goes on to say that the system not only encourages safe science, but cronyism as well.  We would add that this includes hierarchies, which foster obedience by many to the will of the few.  Because the researcher's affiliation, collaborators, co-authors, publication record and so on are a part of the whole grant package, it's impossible for reviewers to not use this information in their judgments and review a grant impartially. As Nicholson puts it, the whole emphasis is on a scientist "being liked" by the scientific community.  Negative findings are rarely published, which in effect means that scientists can't disagree with each other in print, and peer review ensures that scientists stay within the fold.  Nicholson believes this has all created a culture of mediocrity in science.  We can say from experience that submitting grants anonymously is unlikely to work because, like 'anonymous' manuscripts sent out for review, one can almost always guess the authors.

There is of course a problem.  Most off-center science is going to go nowhere.  Real innovation is a small fraction of ideas that claim it (sincerely or as puffery).   Accepted wisdom has been hard-won and that's a legitimate reason to resist.  So not all those whose ideas are off base are brilliant or right. How one tells in advance is the question that's a problem because there's no good way, and that provides a ready-made excuse for generic resistance.

Nicholson's solution to restructuring "the current scientific funding system, to emphasize new and radical work"?  He proposes that the grant review system change to include non-scientists who don't understand the field, as well as scientists who do. "Indeed," he says, "the participation of uninformed individuals in a group has recently been shown to foster democratic consensus and limit special interests." And, "crowd funding" has been successful in a lot of non-scientific arenas, he notes, and could conceivably be used to fund grants as well.

It will come as no surprise to regular MT readers to know that we endorse Joshua Nicholson's indictment of the current system.  Peer review seems necessary, brilliant and democratic, and it was established largely and explicitly to break up and prevent Old Boy networking, and make public research funding more 'public'.  Indeed, money no longer goes quite so exclusively to the Elite universities.  But politicians promise things to get a crowd of funders, who want the rewards.  And even that, like any system, can be gamed, and a pessimist (or realist?) is likely to argue that after you've relied on it for a while, it produces just the kind of stale, non-democratic, old boy network that Nicholson describes--similar hierarchies even with many of the same hierarchs resurfacing.

It's unlikely that the grant system will undergo radical transformation any time soon, because too many people would have to be dislodged, though, perhaps when the old goats retire and get out of the way, that will smooth the way.   But there are rumblings in the world of scientific publishing, and demands for change, and this makes us hopeful that perhaps these growing challenges to the system can have widespread effects in favor of innovation and a more egalitarian sharing of the wealth (in the form of academic positions, grant money, publications, and so on).  The demands are coming from scientists boycotting Elsevier Publishing because they are profiting handsomely from the scientists' free labor; scientists and others petitioning for open and free access to papers publishing the results of studies paid for by the taxpayer; physicists circumventing the old-boy peer review process by publishing online or first passing their manuscripts through open-ended peer review online.  And, yes, there are open access journals (e.g., PLoS), though generally at high cost.

The system probably can't change too radically so long as science costs money and research money doesn't come along with salary as part of an academic job, as it probably should since research is required for the job!  Instead, the opposite is true: universities hunger for you to come do your science there largely because they expect you to bring in money (they live on the overhead)!  And humans are tribal animals so the fact that who you know is such an intrinsic part of the scientific establishment is not a surprise--but that aspect of the system can and should be changed.  The reasons that science has grown into the lumbering, conservative, money-driven, careerist megalith that it is can be debated, as can the degree to which it is delivering the goods, even if imperfectly.  But it is possible that we're beginning to see glimmers of hope for change.  The best science is at least sometimes unconventional, and there must be rewards for that as well.

Monday, April 11, 2011

What is peer review? What should it be? Does it work?

Scientific fraud
There was a recent discussion on our favorite medium, the BBC (in this case, the World Service on radio) about the problem of scientific fraud.  How common is it?  How harmful?  How does it get through the peer review process?

Science depends more than perhaps any other area of human societal life, on honesty.  That's because we each can only do our own experiments, and can't possible do all that was done before that set the stage for what we attempt to do.  Fortunately, real fraud seems to be very rare in science.  It occurs, but is usually punished when caught, by strong sanctions.  It is very serious indeed, because investigators can spend precious time, effort, and resources pursuing ideas suggested by published reports of important findings or methods.  The findings are widely cited and used in support of the new ideas in, for example, grant applications.  In anthropology, the Piltdown fossil fraud was accepted by many scientists for decades, and built into texts and other frameworks for analyzing human evolution.  The recent Korean cloning fraud misdirected many labs into wasteful cloning experiments.

Major fraud is caught in various ways, perhaps most often by others trying, and eventually failing, to replicate an important result.  Minor fraud may be more common--we know about the major cases because of the problems they cause, but few care about minor results.  And dissembling and exaggeration and self-promotion are rife.  The public may not be aware of it, but scientists usually are (and often justify doing it themselves because 'everybody does it' and 'you have to, to get funding or to be published in Science.')

Puffery has consequences similar to fraud in that skillful hyperbole establishes fads, and most of us, desperate for attention and funding and so on, eagerly jump on band-wagons (especially when a new toy, like fMRI, gene expression arrays, and the like are available).  Overall, it is likely that puffery in fact causes more damage by diverting funds and energies in areas less promising than is believed.

Peer review
Peer review is the system that supposedly keeps up the scientific standard.  An article is submitted to a journal, and if deemed relevant, is sent to 2 or 3 experts to judge.  They are to judge quality and appropriateness, importance or novelty, and find factual errors and so on that may be in the reported work.  But, of course, these peers may also be rivals, or (almost always) too busy to pay close attention to the paper they're sent.  Nowadays, with the huge proliferation in the numbers of frenetically competing investigators, and new journals, and tons of online 'supplemental' information allowed, it is nearly impossible to maintain a standard.

When reviewers find unclear statements, things not well explained, or actual errors, then they do the original authors a huge positive service by improving the paper before it is published. 

Peer reviewers were never asked to find fraud, however.  They are hardly ever in a position to do that anyway.  They, like all of us, must in practice assume the honesty of the authors.  So peer review doesn't generally find fraud.

One thing the system of peer review does that we find increasingly irritating--and try to avoid ourselves--is empower the reviewer to insist that the author(s) modify the paper in ways that, when you get right down to it, would make the paper more like what the reviewer would write if s/he were the author.  This leads to often extensive modification in ways that are forced rather than natural, and not what the authors actually wanted to say.  The authors, anxious to get their work published and move on, will then litter their revision with every kind of cumbersome citation, caveat, and implicit rhetorical bow to the reviewers--so they can tell the Editor they responded to all of the comments.

More and more, we feel, the reviewer should find mistakes and unclear aspects of papers, but just let the authors have their own conclusions and interpretations.  If the authors do a bad job, then readers of the paper will do better work, ignore the paper, or whatever.  But if the authors do a good job--including a good job of presenting their case in a style they wish to use--then the paper they wrote is better off unadulterated. 

Peer review is valuable, but could be improved if reviewers were instructed to stick to the important issues.  Let the poets in science out of the lab!

Friday, December 17, 2010

Peering over Peer Review

We all like to flatter ourselves about the deep insights that we publish in 'peer reviewed' journals.  Peer review means that a couple of people who (an Editor thinks) are qualified have looked at a submitted paper and said it's OK to publish in the journal in question.

Peer review has important purposes:  done at its best, it can spot errors, help with clarity of expression, and weed out trivial or irrelevant results.  In part this keeps science at least somewhat honest, and can serve to avoid total Old Boy network exclusion of competing ideas.  If we have to have standards for publication, and to judge scientists for grant support, tenure, promotion and the like, this is a reasonable way to do it.

But like anything, it can become an insider's snob system that either still supports the Old Boys or is exclusive, especially of maverick ideas.  As it grew over the years, peer review like any other system, became vulnerable to gaming by savvy investigators.  And because an ever-growing set of journals have to keep cranking out the pubs, the standards simply can't be kept that unbiased quality uniformly high.  Even with no ill intent, reviewers are far too overloaded to pay close attention to papers they review (and the same goes for grant peer review).  And to end all, perhaps, there is a proliferation of online journals some of which are like vanity press: in the pretense (sometimes justified) of avoiding the creativity stifling of peer review, almost anything submitted gets published....with the pretense of peer review in the ability of readers to comment, blog-style.

Meanwhile, free-standing social networking (like MT and other blogs) has become an important source of views and information, and an outlet for skepticism, circumspection, and different perspectives.  It is not entirely trustworthy, because bloggers like anyone else have their perspectives and agendas (but of course so do the peer reviewers that editors choose -- among whom many are bloggers).  But it's not useless and a careful blog-reader can (and should) judge the reliability.

A paper in Nature reviews the recent storm of complaint about what is increasingly being characterized as over-stated (at best), the story on arsenic based life, shows the state of play.  We immediately jumped on this paper here in MT for both scientific reasons and because this seemed more like NASA advertising than carefully presented science.  We think we were justified  in criticizing the story because it was far from a complete demonstration of what the media hype said it was biologically, and because despite NASA-hype it was irrelevant to any serious question about whether there were Little Green Men out there beyond Hollywood.  So in a sense ours was a meta-criticism: not of the primary work, but of the unjustified extension of it as if it had astrobiological implications.

Many others in the blogosphere, probably the most visible being Carl Zimmer over at The Loom, have seized on the paper, and some have noticed problems with the chemistry itself, that we are not competent to have noticed.  We said the paper itself, in Science, seemed reasoned in terms of overclaiming.  Perhaps even Science was vulnerable to the Big Story self-promotion and failed to obtain adequate peer review.

But there's more because the Nature commentary was as much about the legitimacy of blogosopheric reaction as it was about the science itself.  And the expected reaction, in our culture, is immediately to milk this for more self-gain:  (1) NASA and the authors, who have clammed up in the face of this clamor (contrary to their courting of attention initially), want studies to confirm their results....in other words, please pass out more money.  We would oppose that because the work even if well done would have no real 'astrobiology' research.   (2) Nature noted that they encourage online discussion of such issues, that is, use of their blog site and subscriptions to their journal.

Both Nature and NASA are vested interests.   It's hard not to have self-interested motives these days, and perhaps we shouldn't harp on them.  But they do bring to attention changes in the system by which scientific findings gain credence. The blogosphere that needs nurturing in this context is not one controlled by a commercial journal, but the independent, non-vested, free-for-all of the internet, where there is no control for PR-spinning or profit motive. 

The blogosphere is peer review, of a heterogeneous kind but perhaps not so much worse than stuffy professional expert-controlled peer review.  One need not pee over peer review to ask how more democratic and openly rather than covertly the social wheels are turned to decide what should see the light of day.

Friday, March 5, 2010

The Bishop of Tenure

Most Bishops promise that if you behave yourself you'll be handed everlasting bliss. But Amy Bishop didn't seem to agree. She had what appears to have been a mediocre record, at best, but a far from mediocre ego and temper. The synod that judged her work decided that it did not merit passage to academic haven (tenure--a lifetime chair in their department) so she sent them to heaven--by shooting them dead in their chairs.

This highlights one of the issues in American academic life that has many aspects, ethical, practical, and cultural, and that also affects science itself. Tenure is awarded if you have done important work, and enough of it (and often if you also hustled a lot of money to support it, and the overhead charges of your institution) that your peers within your department, and externally in the profession, judge you to be a Major Player. Even at the University of Alabama/Huntsville which, no matter its strengths, isn't a nationally top school.

The tenure system leads to all sorts of gaming of the system, as people very naturally try to build a case that they should not (ever) lose their jobs. Some people think nobody should have a lifetime job (unless you own the company?). Others defend the system because it truthfully protects freedom of speech in universities, where intimidation of ideas should not happen (but clearly would without a tenure system). And many like the system because we're beneficiaries. It's definitely a privileged position to be in. Unfortunately, in our security hungry class it becomes treated as something like a civil right, identical to one's ego and sense of self-worth, and so on.

It isn't however, something to kill for!

The Bishop murders (well, we weren't witnesses, so we are reporting what we've seen in the papers) show many aspects of our polarized, tightly wound-up society. They show the nature of academic bureaucratic structures. And they show a lot of pretense and loss of mission, too.

Universities used to be thought of as 'schools'. The strange notion was that we were here to teach students so they could be equipped to go out in the world and be successful. Instead, much of what happens at universities neglects the main students (the undergraduates) and instead is structured increasingly by the faculty to serve the faculty.

The point of the age-old publish-or-perish ethos used to be not score-counting, but that faculty members do enough work of good enough quality to show that they were keeping up to date in their field, so that they could be capable teachers. It was not a pretentious self-congratulatory game. Of course there were always those top-level faculty who published well and often, but that was the exception, not the ritualized necessity.

It isn't however, something to kill for!

Besides the stress on the junior faculty member, the system leads to the kind of gaming by which people split up papers into many rather than fewer, worry about the 'impact factor' of the journals they publish in, do whatever kind of work is fundable and publishable rapidly. This is part and parcel of our too-often incremental 'safe' kind of research. Problems that take a long time to address well are not on the plate. Most of us know this, though not all would acknowledge it (we've posted on this subject before).

Stress over tenure decisions spoils a good bit of university life. We have to have standards and, just like a baseball team, not every one who aspires to a position can have one (much less a lifetime one). But good training of students (not just 'our' own students who man the grant projects in our own labs), and our best effort at understanding the truth of our respective fields of work, not paper-counts or grant budgets, should be the marks of success.

Now the Alabama murders make one raise questions about the tenure system, and they're legitimate questions. But this can mistakenly be interpreted that the reason for such examination is that the system is badly flawed. That somehow Bishop's mayhem was understandable, or even excusable. A cause for self-examination where the questioning finger is pointed at ourselves.

It isn't however, something to kill for!

From the stories we see, Dr Bishop snapped under the pressure. There is no excuse or sympathy for that. She's certainly not the kind of Bishop the Church of Knowledge can tolerate. Her peers were doing their job, and that job involves human judgment. That means to some extent it is frail, fallible, and perhaps even sociopolitical. But that's no secret. The system cannot be allowed to be intimidated by those who snap. Even if it leads us to reflect that perhaps we should return the academic altar of knowledge to some saner semblance of balance.