Crossing the academic integrity line

Every so often, perhaps now ever more often, we sees stories of university faculty members, or their research groups, fabricating data or publishing distorted, or culpably misleading presentations of their work in science journals.  Presumably the same is happening in grant applications.  (I refer here to the life sciences, since I know little about other areas of academic publishing).

I think that far more often, indeed, perhaps almost regularly, the science community publishes papers that are configured to put the authors' best feet forward.  Research results can easily be manipulated, within the confines of strict truthfulness.  For example, many transgenic mice are produced with some dysfunctional result introduced to, for example, understand some human disease by elucidating how, say, a gene mis-functions when mutated.  Mice can't object and humans can't be experimented on, so we need 'model' systems, and we naturally want to both discover important things in their use and to get credit for our work.

But we are only human.  Too often, what is published in a main paper are the best-foot-forward results, with caveats clearly stated.....in 'supplemental information'.  Unfortunately, the latter can be as long as the New York phone book, while the actual paper is only a few pages.  And burying uncomfortable aspects of results, in length-unlimited 'supplemental information' is a well-honed skill.  Some 'supplementals' are quite extensive and of course inconvenient truths can be hidden--indeed, probably only a small number of readers ever bother to check the supplements beyond looking for some point or other.

Anyone who thinks that this is not a way for best feet to be put forward is naive.  Indeed, in most cases, it's not cheating, since the article itself simply cannot convey the entire experimental system.  BUT.....

If supplemental information can function to allow investigators to hide their worse feet, that can verge on outright dishonesty.  For example, if I make, say, ten transgenic mice with Mutant Gene X replacing the normal gene, the common result is that the animals are all different, for many reasons including reasons unknown.  Some victims (er, mice) may show no effect at all, others may die as a side effect of the experiments (poor mice!).  No one can be blamed for this sort of thing (except, perhaps, for doing what they do to innocent mice in the first place).  But, often and  I think unarguably typically, the mouse showing the clearest effects, assuming they are effects, is the one published and often denoted 'representative'.  Is this fraud?

It is strategic, but one can explain it away by saying (hopefully) that the mice with lesser effects didn't entirely receive the transgene properly and so on.  The 'representative' mouse must be the one with the complete incorporation of the gene.  If you want to see all those with the transgene, even the Supplemental tome may not (I think typically does not) show the other mice, the ones with lesser (or no) effect.

There can be legitimate reasons for this variation, of course: the transgene may in fact not be properly inserted, or not in all cells, or the receiving mouse may have somehow detected or compensated for it.  So--shouldn't the paper reporting the experiment show all the recipient mice?  Obviously it should--but if Nature or Science only give you 5 pages, and you want to show that you've found what the transgene does, you have motivation for only showing the 'representative' mouse in your paper.

Is this a form of scientific misinformation or even 'fraud'?  You have to make your own judgment.  Even given legitimate reasons why some recipient mice don't show the result, there can be lots of reasons why you don't want to present it in your paper--not least being that if only one of your transgene victims shows the result, maybe your interpretation or claims are simply wrong!  Gulp--then no Nature paper!

How often do you think that, perhaps uncomfortably like advertizing agencies, papers even in prestigious journals are doing something similar?  How can one tell?  If the problem is more than trivial, is there anything that can be done to stop or at least minimize it?

After all, the point of scientific reports is to lead others to build on them.  The building is only as strong as its foundation.

When the cat brings home a mouse

To our daughter's distress, she needs to find a new home for her beloved cats, so overnight we've gone from no cats to three cats, while we try to find them someplace new.  I haven't lived with cats since I was a kid really, because I was always allergic.  When I visited my daughter, I'd get hives if Max, her old black cat, sadly now gone, rubbed against my legs, and I always at least sneezed even when untouched by felines.  But now with three cats in the house, I'm allergy-free and Ken, never allergic to cats before, is starting to sneeze -- loudly.

Old Max

Casey

Oliver upside-down

But the mystery of the immune system is just one of the mysteries we're confronting -- or that's confronting us -- this week.  Here's another.  The other day my daughter brought over a large bag of dry cat food.  I put it in a closet, but the cats could smell it, and it drove them nuts, so I moved it into the garage.  A few days later I noticed that the cats were all making it clear that they really, really wanted to go into the garage, but we were discouraging that given the dangers of spending time in a location with vehicles that come and go unpredictably. I just assumed they could smell the kibbles, or were bored and wanted to explore new horizons.

But two nights ago I went out to the garage myself to get pellets for our pellet stove, and Mu managed to squeeze out ahead of me.  He made a mad dash for the kibbles.  Oliver was desperate to follow, but I squeezed out past him and quickly closed the door.  At which point, Mu came prancing back, squeaking.  Oh wait, he wasn't squeaking, it was the mouse he was carrying in his mouth that was squeaking!  He was now just as eager to get back in the house as he'd been to get out.  After a few minutes he realized that wasn't going to happen, so he dropped the now defunct mouse, and I let him back in.

Mu, the Hunter

So, that 'tear' in the kibbles bag that I'd noticed a few days before?  Clearly made by a gnawing mouse (mice?).  And the cats obviously had known about this long before I did.  But how did Mu know exactly where to make a beeline to to catch the mouse?  He'd never seen where I put the bag, nor the mouse nibbling at it!  And I have to assume the other cats would have been equally able hunters had they been given the chance.

Amazing.  A whole undercurrent of sensory awareness and activity going on right at our feet, and we hadn't clued in on any of it.  I'd made unwarranted assumptions about holes in the bag, but the cats knew better.  Yes, I could have looked more closely at the kibble that had spilled out of the bag and noticed the mouse droppings.  But I didn't, because, well, because it didn't occur to me.

Though, now that I'm clued in, I believe we've got another mouse...

Mu and Ollie at the door to the garage yesterday afternoon

And?
I might even have been able to detect the mouse without seeing any of the evidence, just like the cats, if I'd tuned in more attentively, but I'm pretty sure it would have required better hearing.  In any case, other bits of evidence more suited to my perceptive powers were available, but I didn't notice.  I take this as yet another cautionary tale about how we know what we know, and I will claim it applies as well to politics, economics, psychology, forensics, religion, science, and more.  We build our case on preconceived notions, beliefs, assumptions, what we think is true, rarely re-evaluating those beliefs -- unless we're forced to, when, say, Helicobacter pylori is found to cause stomach ulcers, or our college roommate challenges our belief in God, or economic austerity does more harm than good.

As Holly often says, scientists shouldn't fall in love with their hypothesis.  Hypotheses are made to be tested; stretched, pounded, dropped on the floor and kicked, and afterwards, and continually, examined from every possible angle, not defended to the death.  But we often get too attached, and don't notice when the cat brings home a mouse.

An illustrative blog post in The Guardian by Alberto Nardelli and George Arnett last October tells a similar tale (h/t Amos Zeeberg on Twitter).  "Today’s key fact: you are probably wrong about almost everything."  Based on a survey by Ipsos Mori, Nardelli and Arnett report disconnects between what people around the world believe is true about the demographics of their country, and what's actually true.

So, people in the US overestimate the percentage of Muslims in the country, thinking it's 15% when it's actually 1%.  Japanese think the percentage of Muslims is 4% when it's actually 0.4%, and the French think it's 31% while it's actually 8%.

In the US, we think immigrants make up 32% of the population, but in fact they are 13%.  And so on.  We think we know, but very often we're wrong.  We're uninformed, ill-informed, or under informed, even while we think we're perfectly well informed.

Source: The Guardian

The Guardian piece oozes political overtones, sure.  But I think it is still a good example of how we go about our days, thinking we're making informed decisions, based on facts, but it's not always so.  A minority of Americans accept evolution, despite the evidence; you made up your mind about whether Adnan is guilty or innocent if you listened to Serial, even though you weren't a witness to the murder, and the evidence is largely circumstantial.  And so on.  And this all has consequences.

In a sense, even if we are right about what we think, or its consequences, based on what we know, it's hard to know if we are missing relevant points because we simply don't have the data, or haven't thought to evaluate it correctly, as me in regard to Mu and the mouse.  We have little choice but to act on what we know, but we do have a choice about how much confidence, or hubris, we attribute to what we know, to consider that what we know may not be all there is to know.

This is sobering when it comes to science, because the evidence for a novel or alternative interpretation might be there to be seen in our data, but our brains aren't making the connections, because we're not primed to or because we're unaware of aspects of the data.  We think we know what we're seeing, and it's hard to draw different conclusions.

Fortunately, occasionally an Einstein or a Darwin or some other grand synthesizer comes along and looks at the evidence in a different way, and pushes us forward.  Until then, it's science as usual; incremental gains based on accepted wisdom.  Indeed, even when such a great synthesizer provides us with dramatically better explanations of things, there is a tendency to assume that now, finally, we know what's up, and to place too much stock in the new theory......repeating the same cycle again.

The Genome Institute and its role

The NIH-based Human Genome Research Institute (NHGRI) has for a long time been funding the Big Data kinds of science that is growing like mushrooms on the funding landscape.  Even if overall funding is constrained, and even if this also applies to the NHGRI (I don't happen to know), the sequestration of funds in too-big-to-stop projects is clear. Even Francis Collins and some NIH efforts to reinvigorate individual-investigator RO1 awards don't really seem to have stopped the grab for Big Data funds.

That's quite natural.  If your career, status, or lab depends on how much money you bring into your institution, or how many papers you publish, or how many post-docs you have in your stable, or your salary and space depend on that, you will have to respond in ways that generate those score-counting coups.  You'll naturally exaggerate the importance of your findings, run quickly to the public news media, and do whatever other manipulations you can to further your career.  If you have a big lab and the prestige and local or even broader influence that goes with that, you won't give that up easily so that others, your juniors or even competitors can have smaller projects instead.  In our culture, who could blame you?

But some bloggers, Tweeters, and Commenters have been asking if there is a solution to this kind of fund sequestration, largely reserved (even if informally) for the big usually private universities.  The arguments have ranged from asking if the NHGRI should be shut down (e.g., here) to just groping for suggestions.  Since many of these questions have been addressed to me, I thought I would chime in briefly.

First, a bit of history or perspective, as informally seen over the years from my own perspective (that is, not documented or intended to be precise, but a broad view as I saw things):

The NHGRI was located administratively where it was for reasons I don’t know.  Several federal institutes were supporting scientific research.  NIH was about health, and health 'sells', and understandably a lot of fund is committed to health research.  It was natural to think that genome sequences and sciences would have major health implications, if the theory that genes are the fundamental causal elements of life was in fact true.  Initially James Watson, discoverer of DNA's structure, and perhaps others advocated the effort.  He was succeeded by Francis Collins who is a physician and clever politician.

However, there was competition for the genome ‘territory’, at least with the Atomic Energy Commission.  I don’t know if NSF was ever in the ‘race’ to fund genomic research, but one driving force at the time was the fear of mutations that atomic radiation (therapeutic, from wars, diagnostic tests, and weapons fallout) generated.  There was also a race with the private sector, notably Celera as a commercial competitor that would privatize the genome sequence.  Dr Collins prominently, successfully, and fortunately defended the idea of open and free public access.  The effort was seen as important for many reasons, including commercial ones, and there were international claimants in Japan, the UK, and perhaps elsewhere, that wanted to be in on the act.  So the politics were rife as well as the science, understandably.

It is possible that only with the health-related promises was enough funding going to be available, although nuclear fears about mutations and the Cold War probably contributed, along with the usual less savory for self-interest, to AEC's interests.

Once a basic human genome sequence was available, there was no slowing the train. Technology, including public and private innovation promised much quicker sequencing in the future, that was quickly to become available even to ordinary labs (like mine, at the time!).  And once the Genome Institute (and other places such as the Sanger Centre in Britain and centers in Japan, China, and elsewhere) were established, they weren't going to close down!  So other sequences entered the picture--microbes, other species, and so on.  

It became a fad and an internecine competition within NIH.  I know from personal experiences at the time that program managers felt the need to do 'genomics' so they would be in on the act and keep their budgets.  They had to contribute funds, in some way I don't recall, to the NHGRI's projects or in other ways keep their portfolios by having genomics as part of this.  -Omics sprung up like weeds, and new fields such as nutrigenomics, cancer genomics, microbiomics and many more began to pull in funding, and institutes (and the investigators across the country) hopped aboard.  Imitation, especially when funds and current fashion are involved, is not at all a surprise, and efficiency or relative payoff in results took the inevitable back seat: promises rather than deliveries naturally triumphed.

In many ways this has led to the current of exhaustively enumerative Big Data: a return to 17th century induction.  This has to do not just with competition for resources, but a changed belief system also spurred by computing power: Just sample everything and pattern will emerge!

Over the decades the biomedical (and to some lesser extent biological) university establishment grew on the back of the external funding which was so generous for so long.  But it has led to a dependency.  Along with exponential growth in the number of competitors, hierarchies of elite research groups developed--another natural human tendency.  We all know the career limitations that are resulting from this.  And competition has meant that deans and chairs expect investigators always to be funded, in part because there aren't internal funds to keep labs running in the absence of grants. It's been a vicious self-reinforcing circle over the past 50 years.

As hierarchies built, private donors were convinced (conned?) into believing that their largesse would lead to the elimination of target diseases ('target' often meaning those in the rich donors' families). Big Data today is the grandchild of the major projects, like the Manhattan Project in WWII, that showed that some kinds of science could be done on a large scale.  Many, many projects during past decades showed something else: Fund a big project, and you can't pull the plug on it!  It becomes too entrenched politically.  

The precedents were not lost on investigators!  Plead for bigger, longer studies, with very large investments, and you have a safe bet for decades, perhaps your whole career. Once started, cost-benefit analysis has a hard time paring back, much less stopping such projects. There are many examples, and I won't single any of them out.  But after some early splash, by and large they have got to diminishing returns but not got to any real sense of termination: too big to kill.

This is to some extent the same story with the NHGRI.  The NIH has got too enamored of Big Data to keep the NHGRI as limited or focused as perhaps it should have been (or should be). In a sense it became an openly anti-focused-research sugar daddy (Dr Collins said, perhaps officially, that NHGRI didn’t fund ‘hypothesis-based research”) based on pure inductionism and reductionism, so it did not have to have well-posed questions.  It basically bragged about not being focused.

This could be a change in the nature of science, driven by technology, that is obsolescing the nature of science that was set in motion in the Enlightenment era, by the likes of Galileo, Newton, Bacon, Descartes and others.  We'll see.  But the socioeconomic, political sides of things are part of the process, and that may not be a good thing.

Will focused, hypothesis-based research make a comeback?  Not if Big Data yields great results, but decades of it, no matter how fancy, have not shown the major payoff that has been promised.  Indeed, historians of science often write that the rationale, that if you collect enough data its patterns (that is, a theory) will emerge, has rarely been realized.  Selective retrospective examples don't carry the weight often given them.

There is also our cultural love affair with science.  We know very clearly that many things we might do at very low cost would yield health benefits far exceeding even the rosy promises of the genomic lobby.  Most are lifestyle changes.  For example, even geneticists would (privately, at least) acknowledge that if every 'diabetes' gene variant were fixed, only a small fraction of diabetes cases would be eliminated. The recent claim that much of cancer is due just to bad mutational luck has raised lots of objections--in large part because Big Data researchers' business would be curtailed. Everyone knows these things.

What would it take to kill the Big Data era, given the huge array of commercial, technological, and professional commitments we have built, if it doesn't actually pay off on its promises?  Is focused science a nostalgic illusion? No matter what, we have a major vested interest on a huge scale in the NHGRI and other similar institutes elsewhere, and grantees in medical schools are a privileged, very well-heeled lot, regardless of whether their research is yielding what it promises.

Or, put another way, where are the areas in which Big Data of the genomic sort might actually pay, and where is this just funding-related institutional and cultural momentum?  How would we decide?

So what do to?  It won't happen, but in my view the NHGRI does not, and never did, belong properly in NIH. It should have been in NSF, where basic science is done.  Only when clearly relevant to disease should genomics be funded for that purpose (and by NIH, not NSF).  It should be focused on soluble problems in that context.

NIH funds the greedy maw of medical schools.  The faculty don't work for the university, but for NIH.  Their idea of 'teaching' often means giving 5-10 lectures a year that mainly consist of self-promoting reports about their labs, perhaps the talks they've just given at some meeting somewhere. Salaries are much higher than at non-medical universities--but in my view grants simply should not pay faculty salaries.  Universities should.  If research is part of your job's requirements, its their job to pay you.  Grants should cover research staff, supplies and so on.

Much of this could happen (in principle) if the NHGRI were transferred to NSF and had to fund on an NSF-level budget policy.  Smaller amounts, to more people, on focussed basic research.  The same total budget would go a lot farther, and if it were restricted to non-medical school investigators there would be the additional payoff that most of them actually teach, so that they disseminate the knowledge to large numbers of students who can then go out into the private sector and apply what they've learned.  That's an old-fashioned, perhaps nostalgic(?) view of what being a 'professor' should mean.  

Major pare-backs of grant size and duration could be quite salubrious for science, making it more focused and in that sense accountable.  The employment problem for scientists could also be ameliorated.  Of course, in a transition phase, universities would have to learn how to actually pay their employees.

Of course, it won't happen, even if it would work, because it's so against the current power structure of science.  And although Dr Collins has threatened to fund more small RO1 grants it isn't clear how or whether that will really happen.  That's because there doesn't seem to be any real will to change among enough people with the leverage to make it happen, and the newcomers who would benefit are, like all such grass-roots elements, not unified enough.

These are just some thoughts, or assertions, or day-dreams about the evolution of science in the developed world over the last 50 years or so.  Clearly there is widespread discontent, clearly there is large funding going on with proportionately little results.  Major results in biomedical areas can't be expected over night.   But we might expect that research had more accountability.

Sloppy, over-sold research: is it a new problem? Is there a solution?

In our previous posts on epistemology (e.g., here)--the question of how we know or infer things about Nature--we listed several criteria that are widely used; induction, deduction, falsifiability, and so on.  Sometimes they are invoked explicitly, other times they are just used implicitly.

A regular MT commenter pointed out a paper on which he himself is an author, showing serious flaws in an earlier paper (Fredrickson et al.) published in PNAS, a prominent journal.  Fredrickson et al., is a report of a study of the genetics of well-being  The critique, also published in PNAS, points out fundamental flaws in the original paper. ("We show that not only is Fredrickson et al.’s article conceptually deficient, but more crucially, that their statistical analyses are fatally flawed, to the point that their claimed results are in fact essentially meaningless.") We can't judge the issues ourselves, as the paper is out of our area, but the critique seems to be rather broad, comprehensive, and cogent.  So, how could such a flawed paper make it into such a journal?

Our answer is that journals have always had their good and less-good papers, and there have always been scientists (and those who claim to be scientists) who trumpeted their knowledge and/or wares.  When there are credit, jobs, fame and so on to be had, one cannot be surprised at this.

Science has become a market, with industry and university welfare systems, a way for the middle class to get societal recognition (which is an important middle-class bauble), and journals proliferate, many avenues for profit blossom, and university administrators stop thinking and become bean-counters.  Solid science isn't always the first priority.

Science was never a pure quest for knowledge, but it is now to a considerable extent more than before, we think, a business with these various forms of material and symbolic 'profit' as coins of the realm, and the faux aspect can be expected to grow.  There isn't any easy fix, because raising standards to become better policed usually leads to becoming more elite, closed, and exclusive, and that is itself a form of opportunity-abuse.

Our commenter did add that he can no longer trust research sponsored by the US government, and here we would differ.  Much good work is done under government sponsorship, as well as industry sponsorship (which can have its own problems).  The government is a loaded, inertial bureaucracy with its armada of career-builders, and that is predictably stifling.  But the general idea is to do things right, to benefit society (not just professors, or funders, or university administrators).  The problem is how to improve the standard.

The issue is not epistemological
Actually we think the comment was misplaced in a sense, because our post was about epistemological criteria--how do we know how to design studies and make inferences?  The comment was about the way the results are reported, accepted, exaggerated, and the like.  This is certainly related to inference, but rather indirectly we'd say.  Reviewers and editors are too lax, have too many pages to fill, too many submissions to read and the like, so that judgment is not always exercised (or, often, authors bury their weak points in a dense tangle of 'supplementary information').

That is, one can do the most careful study, following the rules, but use bad judgment in its design, be too-accepting of test results (such as statistical tests), use inappropriate measures or tests.  And then, often in haste or desperation to get something published from one's work (an understandable pressure!) submit a paper that's less than even half baked.

What is needed is to tighten up the standards, the education and training, reduce the pressure for continual grant funding and publication streams to please Deans or reviewers, and give scientists time to think, make them accountable for their promises, and slow down.  In a phrase, reward and recognize quality more than quantity.

This is very hard to do.  Our commenter's points are very well taken, in that the journals (and news media) are now heavily populated by low- or sub-standard work whose importance is routinely and systematically exaggerated to feed the insatiable institutional maw that is contemporary science.