When I was active in the grant process, including my duty to serve as a panelist for NIH and NSF, I realized that the work overload, and the somewhat arbitrary sense that if any reviewer spoke up against a proposal it got conveniently rejected without much if any discussion, meant that reviews were usually scanty at best. Applications are assigned to several reviewers to evaluate thoroughly, so the entire panel doesn't have to read every proposal in depth, yet each member must vote on each proposal. Even with this underwhelming consideration, the panel members simply cannot carefully evaluate the boxes full of applications for which they are responsible. In my experience, once we got down to business, for those applications not immediately NRF'ed (not recommended for funding), there would be some discussion of the surviving proposals; but even then, with still tens of applications to evaluate, most panelists hadn't read the proposal and it seemed that even some of the secondary or tertiary assignees had only scanned it. The rest of the panel usually sat quietly and then voted as the purported assigned readers recommended. Obviously (sssh!), much of the final rankings rested on superficial consideration.
When a panel has a heavy overload of proposals it is hard for things to be otherwise, and one at least hoped that the worst proposals got rejected, those with fixable issues were given some thoughtful suggestions about improvement and resubmission, and at least that the best ones were funded.
But there was always the nagging question as to how true that hopeful view was. We used to joke that a better, fairer reviewing system was to put the proposals to the Stairway Test: throw them down the stairs and the ones that landed closest to the bottom would be funded!
Well, that was a joke about the apparent fickleness (or, shall we say randomness?) of the funding process, especially when busy people had to read and evaluate far, far too many proposals in our heavily overloaded begging system, in which not just science but careers depend on the one thing that counts: bringing in the bucks.
The Stairway Test (technical criteria)
Or was it a joke? A recent analysis in PNAS showed that randomness is perhaps a best way to characterize the reviewing process. One can hope that the really worst proposals are rejected, but about the rest.....the evidence suggests that the Stairway Test would be much fairer.
I'm serious! Many faculty members' careers literally depend on the grant system. Those whose grants don't get funded are judged to be doing less worthy work, and loss of jobs can literally be the direct consequence, since many jobs, especially in biomedical schools, depend on bringing in money (in my opinion, a deep sin, but in the context of our venal science support system, one not avoidable).
The Stairway Test would allow those who did not get funding to say, quite correctly, that their 'failure' was not one of quality but of luck. Deans and Chairs would, properly, be less able to terminate jobs because of failure to secure funding, if they could not claim that the victim did inferior work. The PNAS paper shows that the real review system is in fact not different from the Stairway Test.
So let's be fair to scientists, and the public, and acknowledge honestly the way the system works. Either reform the system from the ground up, to make it work honorably and in the best interest of science, or adopt a formal recognition of its broken-nature: the Stairway Test.
Any aspect of society needs to be examined on a continual basis to see how it could be improved. University research, such as that which depends on grants from the National Institutes of Health, is one area that needs reform. It has gradually become an enormous, money-directed, and largely self-serving industry, and its need for external grant funding turns science into a factory-like industry, which undermines what science should be about, advancing knowledge for the benefit of society. The Trump policy, if there is one, is unclear, as with much of what he says on the spur of the moment. He's threatened to reduce the NIH budget, but he's also said to favor an increase, so it's hard to know whether this represents whims du jour or policy. But regardless of what comes from on high, it is clear to many of us with experience in the system that health and other science research has become very costly relative to its promise and too largely mechanical rather than inspired. For these reasons, it is worth considering what reforms could be taken--knowing that changing the direction of a dependency behemoth like NIH research funding has to be slow because too many people's self-interests will be threatened--if we were to deliver in a more targeted and cost-efficient way on what researchers promise. Here's a list of some changes that are long overdue. In what follows, I have a few FYI asides for readers who are unfamiliar with the issues.
1. Reduce grant overhead amounts
[ FYI: Federal grants come with direct and indirect costs. Direct costs pay the research staff, the supplies and equipment, travel and collecting data and so on. Indirect costs are worked out for each university, and are awarded on top of the direct costs--and given to the university administrators. If I get $100,000 on a grant, my university will get $50,000 or more, sometimes even more than $100K. Their claim to this money is that they have to provide the labs, libraries, electricity, water, administrative support and so on, for the project, and that without the project they'd not have these expenses. Indeed, an indicator of the fat that is in overhead is that as an 'incentive' or 'reward', some overhead is returned as extra cash to the investigator who generated it.]
University administrations have notoriously been ballooning. Administrators and their often fancy offices depend on individual grant overhead, which naturally puts intense pressure on faculty members to 'deliver'. Educational institutions should be lean and efficient. Universities should pay for their own buildings and libraries and pare back bureaucracy. Some combination of state support, donations, and bloc grants could be developed to cover infrastructure, if not tied to individual projects or investigators' grants.
2. No faculty salaries on grants
[ FYI:Federal grants, from NIH at least, allow faculty investigators' salaries to be paid from grant funds. That means that in many health-science universities, the university itself is paying only a fraction, often tiny and perhaps sometimes none, of their faculty's salaries. Faculty without salary-paying grants will be paid some fraction of their purported salaries and often for a limited time only. And salaries generate overhead, sothey're now well paid: higher pay, higher overhead for administrators! Duh, a no-brainer!]
Universities should pay their faculty's salaries from their own resources. Originally, grant reimbursement for faculty investigators' salaries were, in my understanding, paid on grants so the University could hire temporary faculty to do the PI's teaching and administrative obligations while s/he was doing the research. Otherwise, if they're already paid to do research, what's the need? Faculty salaries paid on grants should only be allowed to be used in this way, not just as a source of cash. Faculty should not be paid on soft money, because the need to hustle one's salary steadily is an obvious corrupting force on scientific originality and creativity.
3. Limit on how much external funding any faculty member or lab could have
There is far too much reward for empire-builders. Some do, or at least started out doing, really good work, but that's not always the case and diminishing returns for expanding cost is typical. One consequence is that new faculty are getting reduced teaching and administrative duties so they can (must!) write grant applications. Research empires are typically too large to be effective and often have absentee PIs off hustling, and are under pressure to keep the factory running. That understandably generates intense pressure to play it safe (though claiming to be innovative); but good science is not a predictable factory product.
4. A unified national health database
We need health care reform, and if we had a single national health database it would reduce medical costs and could be anonymized so research could be done, by any qualified person, without additional grants. One can question the research value of such huge databases, as is true even of the current ad hoc database systems we pay for, but they would at least be cost-effective.
5. Temper the growth ethic
We are over-producing PhDs, and this is largely to satisfy the game of the current faculty by which status is gained by large labs. There are too many graduate students and post-docs for the long-term job market. This is taking a heavy personal toll on aspiring scientists. Meanwhile, there is inertia at the top, where we have been prevented from imposing mandatory retirement ages. Amicably changing this system will be hard and will require creative thinking; but it won't be as cruel as the system we have now.
6. An end to deceptive publication characteristics
We routinely see papers listing more authors than there are residents in the NY phone book. This is pure careerism in our factory-production mode. As once was the standard, every author should in principle be able to explain his/her paper on short notice. I've heard 15 minutes. Those who helped on a paper such as by providing some DNA samples, should be acknowledged, but not listed as authors. Dividing papers into least-publishable-units isn't new, but with the proliferation of journals, it's out of hand. Limiting CV lengths (and not including grants on them) when it comes to promotion and tenure could focus researchers' attention on doing what's really important rather than chaff-building. Chairs and Deans would have to recognize this, and move away from safe but gameable bean-counting.
[ FYI:We've moved towards judging people internally, and sometimes externally in grant applications, on the quantity of their publications rather than the quality, or on supposedly 'objective' (computer-tallied) citation counts. This is play-it-safe bureaucracy and obviously encourages CV padding, which is reinforced by the proliferation of for-profit publishing. Of course some people are both highly successful in the real scientific sense of making a major discovery, as well as in publishing their work. But it is naive not to realize that many, often the big players grant-wise, manipulate any counting-based system. For example, they can cite their own work in ways that increase the 'citation count' that Deans see. Papers with very many authors also lead to red-claiming that is highly exaggerated relative to the actual scientific contribution. Scientists quickly learn how to manipulate such 'objective' evaluation systems.]
7. No more too-big-and-too-long-to-kill projects
The Manhattan Project and many others taught us that if we propose huge, open-ended projects we can have funding for life. That's what the 'omics era and other epidemiological projects reflect today. But projects that are so big they become politically invulnerable rarely continue to deliver the goods. Of course, the PIs, the founders and subsequent generations, naturally cry that stopping their important project after having invested so much money will be wasteful! But it's not as wasteful as continuing to invest in diminishing returns. Project duration should be limited and known to all from the beginning.
8. A re-recognition that science addressing focal questions is the best science
Really good science is risky because serious new findings can't be ordered up like hamburgers at McD's. We have to allow scientists to try things. Most ideas won't go anywhere. But we don't have to allow open-ended 'projects' to scale up interminably as has been the case in the 'Big Data' era, where despite often-forced claims and PR spin, most of those projects don't go very far, either, though by their size alone they generate a blizzard of results.
9. Stopping rules need to be in place
For many multi-year or large-scale projects, an honest assessment part-way through would show that the original question or hypothesis was wrong or won't be answered. Such a project (and its funds) should have to be ended when it is clear that its promise will not be met. It should be a credit to an investigator who acknowledges that an idea just isn't working out, and those who don't should be barred for some years from further federal funding. This is not a radical new idea: it is precedented in the drug trial area, and we should do the same in research. It should be routine for universities to provide continuity funding for productive investigators so they don't have to cling to go-nowhere projects. Faculty investigators should always have an operating budget so that they can do research without an active external grant. Right now, they have to piggy-back their next idea by using funds in their current grant, and without internal continuity funding, this is naturally leads to safe 'fundable' projects, rather than really innovative ones. The reality is that truly innovative projects typically are not funded, because it's easy for grant review panels to fault-find and move on the safer proposals.
10. Research funding should not be a university welfare program
Universities are important to society and need support. Universities as well as scientists become entrenched. It's natural. But society deserves something for its funding generosity, and one of the facts of funding life could be that funds move. Scientists shouldn't have a lock on funding any more than anybody else. Universities should be structured so they are not addicted to external funding on grants. Will this threaten jobs? Most people in society have to deal with that, and scientists are generally very skilled people, so if one area of research shrinks others will expand.
11. Rein in costly science publishing
Science publishing has become what one might call a greedy racket. There are far too many journals, rushing out half-way reviewed papers for pay-as-you-go authors. Papers are typically paid for on grant budgets (though one can ask how often young investigators shell out their own personal money to keep their careers). Profiteering journals are proliferating to serve the CV-padding hyper-hasty bean-counting science industry that we have established. Yet the vast majority of papers have basically no impact. That money should go to actual research.
12. Other ways to trim budgets without harming the science
Budgets could be trimmed in many other ways, too: no buying journal subscriptions on a grant (universities have subscriptions),less travel to meetings (we have Skype and Hangout!), shared costly equipment rather than a sequencer in every lab. Grants should be smaller but of longer duration, so investigators can spend their time on research rather than hustling new grants. Junk the use of 'impact' factors and other bean-counting ways of judging faculty. It had a point once--to reduce discrimination and be more objective, but it's long been strategized and manipulated, substituting quantity for quality. Better evaluation means are needed.
These suggestions are perhaps rather radical, but to the extent that they can somehow be implemented, it would have to be done humanely. After all, people playing the game today are only doing what they were taught they must do. Real reform is hard because science is now an entrenched part of society. Nonetheless, a fair-minded (but determined!) phase-out of the abuses that have gradually developed would be good for science, and hence for the society that pays for it. ***NOTES: As this was being edited, NY state has apparently just made its universities tuition-free for those whose families are not wealthy. If true, what a step back towards sanity and public good! The more states can get off the grant and other grant and strings-attached private donation hooks, the more independent they should be able to be. Also, the Apr 12 Wall St Journal has a story (paywall, unless you search for it on Twitter) showing the faults of an over-stressed health research system, including some of the points made here. The article points out problems of non-replicability and other technical mistakes that are characteristic of our heavily over-burdened system. But it doesn't go after the System as such, the bureaucracy and wastefulness and the pressure for 'big data' studies rather than focused research, and the need to be hasty and 'productive' in order to survive.
The problem An important commentary appeared in the September PLoS Biology, though we have only just stumbled across it. It has already been viewed close to 30,000 times but seems to have generated little discussion, either on the PLoS website or in the blogosphere. Indeed, we wonder why so few have wanted to comment, since the article points to serious problems that we all know about, and describes them accurately. We think the subject deserves more attention.
In his paper, with a title that says it all, "Real Lives and White Lies in the Funding of Scientific Research: the Granting System Turns Young Scientists into Bureaucrats and then Betrays Them", Peter Lawrence, Department of Zoology, University of Cambridge and Medical Research Council Laboratory of Molecular Biology, Cambridge, United Kingdom makes a strong case for the need for drastic change in the way science is funded. Lawrence is a highly regarded developmental biologist, with a long and distinguished research record in patterning in general and the genetics of development of the fruit fly in particular. He also has many times confronted problems in the politics of science head on. He has no qualms about telling it as he sees it; his is a very welcome and needed voice. Since he has not been a research failure, his views can be taken seriously: they are not just sour grapes.
In the paper, he argues that the status quo is not good for young scientists. The incessant need to apply for research money takes far too much time, encourages conventional thinking, and even lies. As he says,
To expect a young scientist to recruit and train students and postdocs as well as producing and publishing new and original work within two years (in order to fuel the next grant application) is preposterous. It is neither right nor sensible to ask scientists to become astrologists and predict precisely the path their research will follow—and then to judge them on how persuasively they can put over this fiction. It takes far too long to write a grant because the requirements are so complex and demanding. Applications have become so detailed and so technical that trying to select the best proposals has become a dark art. For postdoctoral fellowships, there are so many arcane and restrictive rules that applicants frequently find themselves to be of the wrong nationality, in the wrong lab, too young, or too old.
And, he tells his own story:
After more than 40 years of full-time research in developmental biology and genetics, I wrote my first grant and showed it to those experienced in grantsmanship. They advised me my application would not succeed. I had explained that we didn't know what experiments might deliver, and had acknowledged the technical problems that beset research and the possibility that competitors might solve problems before we did. My advisors said these admissions made the project look precarious and would sink the application. I was counselled to produce a detailed, but straightforward, program that seemed realistic—no matter if it were science fiction. I had not mentioned any direct application of our work: we were told a plausible application should be found or created. I was also advised not to put our very best ideas into the application as it would be seen by competitors—it would be safer to keep those ideas secret.
The implications This will resonate with anyone who has written a grant proposal--or taken a class on 'grantsmanship'. The process is less about good ideas than about gaming the system, and this takes more and more time. Science, we are proud to boast publicly, rests on truth and trust--that's why plagiarism or fudged experiments are treated so harshly.
But what about the routine kinds of dishonesty that our system fosters? People rarely admit it publicly, but it is absolutely routinely acknowledged in private that, along with the dissembling Peter describes, proposals are submitted for large amounts of funds for work that has already been done, or that the investigator knows won't deliver what is promised (and, often, simultaneously hyped in the public media). What about dissembling by the manipulation of data to present it in a technically honest way that nonetheless biases the impression of the importance of the data and of the authors' conclusions?
These issues are but the tip of a potentially destructive iceberg. The current system builds big empires with large, long-term and hence unstoppable entrenched budgets. The system encourages large teams of workers, hires large numbers of post-docs as its cogs, and actively lobbies publicly and privately to secure its funding base. Peter discusses how large groups can cover for the low-yield of many of their members.
Now, there's nothing wrong with wanting funds for your research. But when it becomes institutionalized, is based heavily on competition, and careers (and university budgets) depend on a steady stream, the pressures unite in the direction of grantsmanship: gaming the system first to secure funding and then, if there's time, to do something innovative. But when you're preoccupied with securing the funds, there's much less time or even incentive to think about the real science questions.
Fundability often means safety and that means predictability which in turn often means incremental rather than major advances. Research in some areas of biology is very expensive, to be sure, and new technologies definitely do help reveal facts we could not otherwise obtain. But sequestering of resources in a few hands, or for a few technologies, deprives other avenues of resources. The safety-first system encourages (forces?) most investigators to use the technology for various understandable reasons: being fashionable, hiding a lack of ideas behind the predictability of at least some descriptive results if new technology is applied, and equating large-scale with importance. This systematically rewards investigators and of course pleases their Deans who get the overhead.
To be sure, a lot of good science is being done! No system can guarantee that more than a fraction of science will have lasting value. Most papers are hardly ever cited other than by their own authors, and the shelf-life of most research in these overheated days is very short. The distribution of quality has probably always been skewed towards a majority of trivia. And it is reasonable that we have some ways to weed out sluggish or useless yet costly research. This is especially true when there are more claimants than funds, and this is one direct consequence of the system we have now.
But when the rewards of successful grantsmanship are great, they lead to manipulation of the system, as we have seen over the past few decades, and everyone's research becomes "paradigm shifting" on the proverbial cutting edge.
Peer review, designed as one means to reduce the clubbiness of the OldBoy system, has done some of that, but people are hierarchy builders and have long ago figured out to build new kinds of OldBoy systems. Bureaucrats, too, want their portfolios of funded clients, as that helps them (the bureaucrats) build their own careers. But portfolios are jeopardized if there isn't continuity, so investigators have relatively clear paths to continued funding....whether or not they have generated really solid ideas.
We also have a crazily proliferating number of journals, and they allow piles of 'supplemental information' (often sloppily written) to be included. This leads to a tsunami of content that we simply can't keep up with, even within specialty areas--we've commented on this before on this blog. And the pressures mitigate against teaching, because rewards are for funded research. Scientists aren't stupid: we go where the rewards are!
The solution? We undoubtedly must live with inefficiency in science. If you're really exploring the unknown, you can't know what you'll learn. Most ideas don't pan out. But a kind of evolutionary ecology perspective is worth taking: an ecosystem is most robust to environmental change when it is most diverse. A larger number of funded scientists, perhaps all with less funding, with high-end resources housed in technology service centers, could foster a higher probability and faster flow of really new ideas.
The problems are deeper and more subtle even than all of this, though. It's become a positive-feedback system. Status depends on having students, and the more the merrier. Rather than being sane, and mutually paring back to, say, generational replacement levels in which we each train only one or two students in our careers, the system encourages us to take more graduate students, to help us write more papers and get more grants, and then to do the work on the grants. That means more competition for fewer jobs and grants. We can't taper back because everyone would have to agree to that, and we're not in an altruistic mode these days. So, the squeeze is on--mainly on the young aspirants to science careers. Our university, like your university, wants more!
There is no easy solution for a positive-feedback system, particularly because universities have become so fundamentally dependent on overhead money from grants. Professor Lawrence proposes the shortening of grant applications. But daily life out here in the field immediately reveals that such changes simply encourage many more applications per person, since it's less work to do each one and they can be parceled, packaged, slightly modified and so on, in many ways. The overall probability of funding probably will go down, if anything. The US Stimulus grants showed that, when some 20,000 or so applications were submitted--because the applications were relatively easy--for around 200-300 grants. And the administrative overhead, of preparing and submitting grants can't change much per grant, so will go up and up and up, eating further into the useful amount of funds.
One can say that this is a harsh system but that, like natural selection, it screens out the worst and favors the best. That's true certainly to some extent. But who says that human life must be made harsh, to feed the self-interest of a few?
The burden will fall on the new people who are entering, or hope to enter, the fields of science. We owe it to them to resist a system that systematically grinds the spirit, or even the careers, out of so many.
We may all have dreams, and may all seek dream jobs. But not every dream is fulfilled, and not every dream job turns out to be a dream. Some, even as students, look at their research professors' lives and say "not for me!" Others are lured by the status system into high-pressure, grant-dependent careers that turn out to be relentlessly tense, by which time it may be to late for the person to taper back and get a less-intense job.
But the system as it now exists is structured to make you feel like a failure if you don't have grants, don't publish frequently in 'High Impact' journals, or --heaven forbid -- like to teach! Nobody should feel disappointed, disillusioned or like a failure because they didn't live up to somebody else's -- to the System's -- notion of success, a notion that is in their, but perhaps not your, self interest. But it's very hard to resist the allure of illusion.
Beyond shorter grant applications, what other solutions does Lawrence propose? Smaller, less costly labs, longer lasting grants (5 years minimum), the option of being judged on past research rather than future plans, less reliance on citation counts. Others have proposed more radical changes, such as all researchers being given a research allowance, that automatically gets renewed for those doing good work -- the obvious problem with this is that it's readily gameable too.
These issues are deep and many, and need to be discussed. The system needs to reward good science again, and young researchers need to be able to expect to retain their love of science long into their careers.
Peter Lawrence lays out the problems in an honest and straightforward way. If you're a scientist, particularly if you are just embarking on your career, you owe it to yourself to read, and talk about, his paper. We'll write more about this in our next post, but meanwhile, your thoughts and comments are most welcome.
