Showing posts with label grant system. Show all posts
Showing posts with label grant system. Show all posts

Tuesday, April 24, 2018

Throw 'em down the stairs! (making grant review fair)

When I was active in the grant process, including my duty to serve as a panelist for NIH and NSF, I realized that the work overload, and the somewhat arbitrary sense that if any reviewer spoke up against a proposal it got conveniently rejected without much if any discussion, meant that reviews were usually scanty at best.  Applications are assigned to several reviewers to evaluate thoroughly, so the entire panel doesn't have to read every proposal in depth, yet each member must vote on each proposal.  Even with this underwhelming consideration, the panel members simply cannot carefully evaluate the boxes full of applications for which they are responsible.  In my experience, once we got down to business, for those applications not immediately NRF'ed (not recommended for funding), there would be some discussion of the surviving proposals; but even then, with still tens of applications to evaluate, most panelists hadn't read the proposal and it seemed that even some of the secondary or tertiary assignees had only scanned it.  The rest of the panel usually sat quietly and then voted as the purported assigned readers recommended.  Obviously (sssh!), much of the final rankings rested on superficial consideration.

When a panel has a heavy overload of proposals it is hard for things to be otherwise, and one at least hoped that the worst proposals got rejected, those with fixable issues were given some thoughtful suggestions about improvement and resubmission, and at least that the best ones were funded.

But there was always the nagging question as to how true that hopeful view was.  We used to joke that a better, fairer reviewing system was to put the proposals to the Stairway Test: throw them down the stairs and the ones that landed closest to the bottom would be funded!

Well, that was a joke about the apparent fickleness (or, shall we say randomness?) of the funding process, especially when busy people had to read and evaluate far, far too many proposals in our heavily overloaded begging system, in which not just science but careers depend on the one thing that counts: bringing in the bucks.
The Stairway Test (technical criteria)

Or was it a joke?  A recent analysis in PNAS showed that randomness is perhaps a best way to characterize the reviewing process.  One can hope that the really worst proposals are rejected, but about the rest.....the evidence suggests that the Stairway Test would be much fairer.

I'm serious!  Many faculty members' careers literally depend on the grant system.  Those whose grants don't get funded are judged to be doing less worthy work, and loss of jobs can literally be the direct consequence, since many jobs, especially in biomedical schools, depend on bringing in money (in my opinion, a deep sin, but in the context of our venal science support system, one not avoidable).

The Stairway Test would allow those who did not get funding to say, quite correctly, that their 'failure' was not one of quality but of luck.  Deans and Chairs would, properly, be less able to terminate jobs because of failure to secure funding, if they could not claim that the victim did inferior work.  The PNAS paper shows that the real review system is in fact not different from the Stairway Test.

So let's be fair to scientists, and the public, and acknowledge honestly the way the system works.  Either reform the system from the ground up, to make it work honorably and in the best interest of science, or adopt a formal recognition of its broken-nature: the Stairway Test.

Friday, May 25, 2012

You scientist, we want you to get ahead....but not too FAR ahead!

A paper in the June issue of BioEssays is titled "Collegiality and careerism trump critical questions and bold new ideas" and, no, we didn't write it.  The subtitle is "A student’s perspective and solution": the author is Joshua Nicholson, a grad student at Virginia Polytech.  It's a mark of the depth of the problem that it is recognized and addressed by a student, in a very savvy and understanding way.  But of course it's students who will most feel its impact as they begin their careers when money is tight and the old boy network is alive and well. The situation is so critical today that even a student can sense it without the embittering experience of years of trying to build a post-training career.
As students we are taught principles and ideals in classrooms, yet as we advance in age, experience, and career, we learn that such lessons may be more rhetoric than reality.
Nicholson is not the first to notice that the current system of funding and rewards encourages more of the same, not innovation.  Scientists, he notes, are discouraged from having radical, or even new ideas in everything from grant applications to even just expression of ideas.  Indeed, numerous examples exist of brilliant scientists who have said they couldn't have done their work within the system; Darwin, Einstein, and whatever you think of Gaia, its conceptor, but also innovative inventor, James Lovelock, has said the same (and he did so recently on BBC Radio 4's The Life Scientific). Other creative people, in the arts, have felt the same way about universities (e.g., Wordsworth the poet, Goya the painter).

The US National Institutes of Health and National Science Foundation both pay lip service to innovation, yes, but still within the same system of application and decision-making.  Nicholson says that the NIH and NSF in fact admit that these efforts are not encouraging innovation (as those of us who have been on such panels and never seen an original project actually funded--usually the reviewers pat the proposer on the head patronizingly and say make it safe and resubmit).  He blames this, correctly, on the review structure; peer review.  Yes, experts in a field are required to evaluate new ideas, but it is they who are often most unwilling to accept them.

To be fair, usually this is not explicit and reviewers may usually not even be aware of their inertial resistance to novelty.  But Nicholson explains that:
(i) they helped establish the prevailing views and thus believe them to be most correct, (ii) they have made a career doing this and thus have the most to lose, and (iii) because of #1 and #2 they may display hubris [2–4, 9, 10]. If, historically, most new ideas in science have been considered heretical by experts [11], does it make sense to rely upon experts to judge and fund new ideas?
He concludes that a student looking to build a career therefore must choose between getting funding by following the crowd and doing more of the same, or being innovative but without any money... that is, driving a taxi.

He goes on to say that the system not only encourages safe science, but cronyism as well.  We would add that this includes hierarchies, which foster obedience by many to the will of the few.  Because the researcher's affiliation, collaborators, co-authors, publication record and so on are a part of the whole grant package, it's impossible for reviewers to not use this information in their judgments and review a grant impartially. As Nicholson puts it, the whole emphasis is on a scientist "being liked" by the scientific community.  Negative findings are rarely published, which in effect means that scientists can't disagree with each other in print, and peer review ensures that scientists stay within the fold.  Nicholson believes this has all created a culture of mediocrity in science.  We can say from experience that submitting grants anonymously is unlikely to work because, like 'anonymous' manuscripts sent out for review, one can almost always guess the authors.

There is of course a problem.  Most off-center science is going to go nowhere.  Real innovation is a small fraction of ideas that claim it (sincerely or as puffery).   Accepted wisdom has been hard-won and that's a legitimate reason to resist.  So not all those whose ideas are off base are brilliant or right. How one tells in advance is the question that's a problem because there's no good way, and that provides a ready-made excuse for generic resistance.

Nicholson's solution to restructuring "the current scientific funding system, to emphasize new and radical work"?  He proposes that the grant review system change to include non-scientists who don't understand the field, as well as scientists who do. "Indeed," he says, "the participation of uninformed individuals in a group has recently been shown to foster democratic consensus and limit special interests." And, "crowd funding" has been successful in a lot of non-scientific arenas, he notes, and could conceivably be used to fund grants as well.

It will come as no surprise to regular MT readers to know that we endorse Joshua Nicholson's indictment of the current system.  Peer review seems necessary, brilliant and democratic, and it was established largely and explicitly to break up and prevent Old Boy networking, and make public research funding more 'public'.  Indeed, money no longer goes quite so exclusively to the Elite universities.  But politicians promise things to get a crowd of funders, who want the rewards.  And even that, like any system, can be gamed, and a pessimist (or realist?) is likely to argue that after you've relied on it for a while, it produces just the kind of stale, non-democratic, old boy network that Nicholson describes--similar hierarchies even with many of the same hierarchs resurfacing.

It's unlikely that the grant system will undergo radical transformation any time soon, because too many people would have to be dislodged, though, perhaps when the old goats retire and get out of the way, that will smooth the way.   But there are rumblings in the world of scientific publishing, and demands for change, and this makes us hopeful that perhaps these growing challenges to the system can have widespread effects in favor of innovation and a more egalitarian sharing of the wealth (in the form of academic positions, grant money, publications, and so on).  The demands are coming from scientists boycotting Elsevier Publishing because they are profiting handsomely from the scientists' free labor; scientists and others petitioning for open and free access to papers publishing the results of studies paid for by the taxpayer; physicists circumventing the old-boy peer review process by publishing online or first passing their manuscripts through open-ended peer review online.  And, yes, there are open access journals (e.g., PLoS), though generally at high cost.

The system probably can't change too radically so long as science costs money and research money doesn't come along with salary as part of an academic job, as it probably should since research is required for the job!  Instead, the opposite is true: universities hunger for you to come do your science there largely because they expect you to bring in money (they live on the overhead)!  And humans are tribal animals so the fact that who you know is such an intrinsic part of the scientific establishment is not a surprise--but that aspect of the system can and should be changed.  The reasons that science has grown into the lumbering, conservative, money-driven, careerist megalith that it is can be debated, as can the degree to which it is delivering the goods, even if imperfectly.  But it is possible that we're beginning to see glimmers of hope for change.  The best science is at least sometimes unconventional, and there must be rewards for that as well.

Thursday, February 17, 2011

The "Me!" parade, or is there a better way to think about funding realities?

So, proposed new budgets are suggesting a $1.6 billion, or 5% or so cut in NIH research funding.  What we've seen in regard to the NSF budget is somewhat different: it may increase modestly, but with the  funds clearly targeted to investment in science interactions,  infrastructure, and education. These seem reasonable and not parochial, but of course NSF budgets are generally much less than NIH grants, and less often cover faculty salaries the way NIH does.

We knew about the NIH proposal because as soon as the proposed budget cuts were announced, we've been besieged by the "Me!" parade of  'urgent'  messages from professional societies urging us (with nice assistance in the form of convenient links) to write our congressmen in outrage, to protest the very nerve of suggesting that research take a hit!

But why does this instant email lobbying go on?  Millions are homeless or even jobless, or have no health care, or have disorders that don't require exotic research to alleviate, and to keep the economy from total free-fall the feds had to go into debt that they now have to figure out how to get out of, and if cuts have to go across the board.  Given this, why can't we be realistic and even good citizens to boot, and realize that a 5% cut in funds is not a cataclysm for science, somewhat less serious than being homeless, and  something to which we should respond to constructively, and with good grace?

If everyone in our society  feels  it's an automatic given that we'll protest anything disadvantageous to our personal selves, we'll descend into more internecine strife than the current situation is going to cause anyway.  Science has grown fat (and complacent?) on grant largess over recent decades, but have we delivered to society in commensurate terms?  Or have we become self-satisfied and ever willing to ask for more, bigger, longer, grander funds for feathering our own nests, with universities and research institutes living on the overhead that we (as their sales force) bring in?  Are the new data and findings in NIH-funded projects--which are very interesting, to be sure--our private playground, or are they really what the public tax base should be used for?  Are measures of public health improving as a result--if they're improving at all?

Investigators will certainly be able to manage, if we must, on less, and we think this could even be good in several ways.  Rather than lobby for more funds, why not lobby for more grants, even if they're smaller, and of shorter duration than they've become?  That way new investigators, young investigators, and people who actually have clever new ideas can have a better chance!  Why not cap the amounts any given lab can have, or stop projects that have been continued too long, or have grown too large, or have reached diminishing returns--and divide the savings up among people who offer something new?  Why not do more centralizing and sharing of costly hi-tech resources, and be more stringent about funding hi-tech but low-thoughtfulness projects?

Maybe schools that have grown fat on overhead with inflated but unpaid (soft-money) faculty, driven to flood the system with relentless grant applications, will have to develop a new sense of socially responsible mission, even if this means shrinking in size, and paying more attention (heavens!) to teaching.  The reversal of a 30-year trend towards growth for its own sake would not be an entirely bad thing. With the current age distribution, phasing back could be done as people retire and simply aren't replaced.  We don't need as many graduate students in an environment that is not able to grow exponentially--even if students are the trophies we like to wave about to demonstrate our importance.

Maybe departments will have to think about importance rather than dollars, when they hire new faculty.  Maybe as we tighten our belts, it will push the blood back to our brains, and the constraint will force new, creative thinking, and a new day for science that is both innovative and socially responsible.

Sunday, June 28, 2009

Can the grant system be overhauled?

Gina Kolata writes in today's New York Times that $105 billion dollars have been spent on cancer research since 1971 when Richard Nixon declared 'war on cancer', but, measuring progress in death rates, we're not a whole lot better off now than we were then. Indeed, cancer is good business, and cancer specialty clinics are opening or expanding all over the country (and advertising about how good they are, to promote sales). Cancer treatments are long and costly to our health care system, so it's a serious problem, both economically and for anyone who has cancer.

Ms Kolata correctly attributes much of this to the conservative nature of the grant system. Researchers don't even apply for money for new ideas because they know the system doesn't reward innovation, so what does get funded is research that doesn't even attempt to go beyond incremental progress. And researchers' careers, prestige, even salaries depend on grants.

Kolata's article is specifically about cancer, but the conservative nature of the grant system is true for all fields. It's partly because 'peer review'--the judging of grants by people who do similar work--keeps it that way. People can only evaluate what they already know. And it's partly because the system demands that the researcher demonstrate that the work can be done, which requires pilot data. And as with any large establishment, it learns how to protect and perpetuate its own interests.

It is not easy to say what to do about it. What kind of accountability for grant recipients would be appropriate? The research questions being asked are tough, so 'cures' cannot be promised in advance, and the more basic the research the less clear what the criteria for success could be. The idea of accountability is that if your research is paid by a health institute it should make notable contributions to health, not just journal articles or to the researcher's career. A candid observer could quickly eliminate a high fraction of grant applications on the grounds that, even if successful as promised, their contribution would be very minor, as Kolata illustrates. Perhaps there should be a penalty for making promises that aren't kept--at least, that could help make the system more honest.

Limits on the size or length of projects or of an investigator's total grants would help spread funds around. But what about the role, sometimes legitimate and sometimes mainly based on a love of technology, of very expensive equipment and approaches? Is there a way to identify less technically flashy, but perhaps more efficacious work? It's easy to see that this can be true: lifestyle changes could prevent vastly more cancer than, say, identifying genetic susceptibility causes, yet we spend much money on cancer genetics research compared to environmental change.

Speaking of lifestyles, one cannot order up innovations the way one can order a burger with fries. Might there be 'meta' approaches that would increase the odds that someone, somewhere will make a key finding or have a penetrating idea? Would that more likely come from someone in a big lab working on long-term projects, or someone in a small lab working in relative obscurity?

Or is it OK to perpetuate the system, assuming good will come of it here and there, meanwhile a lot of people are employed to manage big labs, run the experiments, collect data, make machinery and lab equipment, and sweep the floors in large lab buildings?

These reflections apply to much that is happening in the life (and other) sciences today. They drive the system in particular directions, including fads and technologically rather than conceptually based approaches, and in that sense some things are studied while other approaches may not be considered (or funded because they're out of the mainstream). An example relevant to our blog and work is the way that genetic determinism and, more broadly, a genome-centered focus, drives so much of life and health sciences. By no means irrelevant or all bad! But it is a gravitational force that pulls resources away from other areas that might be equally important.

Clearly major findings are generated by this way of doing science, even if false promises go unsanctioned (indeed, those making them usually continue to do that and continue to be funded with major grants). Life sciences certainly do increase our knowledge, in many clearly important ways. Yet disease rates are not dropping in proportionate ways relative to grandiose promises.

Is there a solution to all this? Could the system be dramatically overhauled, with, say, research money being parceled out equally to anyone a university has deemed worthy of employment? Could the peer review system be changed, so that some non-experts are on review panels, ensuring that the system doesn't simply perpetuate the insider network? Or would they not know enough to act independently? Universities encourage and reward grant success not because it allows important work to be done by their brilliant professors but because it brings prestige, score-counting, and 'overhead' money to campus--can university dependence on overhead or faculty's on salary be lessened? Is there a way to encourage and reward innovation?