I've just read a new book that MT readers would benefit from reading as well. It's Rigor Mortis, by Richard Harris (2017: Basic Books). His subtitle is How sloppy science creates worthless cures, crushes hope, and wastes billions. One might suspect that this title is stridently overstated, but while it is quite forthright--and its argument well-supported--I think the case is actually understated, for reasons I'll explain below.
Harris, science reporter for National Public Radio, goes over many different problems that plague biomedical research. At the core is the reproducibility problem, that is, the numbers of claims by research papers that are not reproducible by subsequent studies. This particular problem made the news within the last couple of years in regard to using statistical criteria like p-values (significance cutoffs), and because of the major effort in psychology to replicate published studies, with a lot of failure to do so. But there are other issues.
The typical scientific method assumes that there is a truth out there, and a good study should detect its features. But if it's a truth, then some other study should get similar results. But many many times in biomedical research, despite huge media ballyhoo with cheerleading by the investigators as well as the media, studies' breakthrough!! findings can't be supported by further examination.
As Harris extensively documents, this phenomenon is seen in claims of treatments or cures, or use of animal models (e.g., lab mice), or antibodies, or cell lines, or statistical 'significance' values. It isn't a long book, so you can quickly see the examples for yourself. Harris also accounts for the problems, quite properly I think, by documenting sloppy science but also the careerist pressures on investigators to find things they can publish in 'major' journals, so they can get jobs, promotions, high 'impact factor' pubs, and grants. In our obviously over-crowded market, it can be no surprise to anyone that there is shading of the truth, a tad of downright dishonesty, conveniently imprecise work, and so on.
Since scientists feed at the public trough (or depend on profits and sales for biomedical products to grant-funded investigators), they naturally have to compete and don't want to be shown up, and they have to work fast to keep the funds flowing in. Rigor Mortis properly homes in on an important fact, that if our jobs depend on 'productivity' and bringing in grants, we will do what it takes, shading the truth or whatever else (even the occasional outright cheating) to stay in the game.
Why share data with your potential competitors who might, after all, find fault with your work or use it to get the jump on you for the next stage? For that matter, why describe what you did in enough actual detail that someone (a rival or enemy!) might attempt to replicate your work.....or fail to do so? Why wait to publish until you've got a really adequate explanation of what you suggest is going on, with all the i's dotted and t's crossed? Haste makes credit! Harris very clearly shows these issues in the all-too human arena of our science research establishment today. He calls what we have now, appropriately enough, a "broken culture" of science.
Part of that I think is a 'Malthusian' problem. We are credited, in score-counting ways, by chairs and deans, for how many graduate students we turn (or churn) out. Is our lab 'productive' in that way? Of course, we need that army of what often are treated as drones because real faculty members are too busy writing grants or traveling to present their (students') latest research to waste--er, spend--much time in their labs themselves. The result is the cruel excess of PhDs who can't find good jobs, wandering from post-doc to post-doc (another form of labor pool), or to instructorships rather than tenure-track jobs, or who simply drop out of the system after their PhD and post-docs. We know of many who are in that boat; don't you? A recent report showed that the mean age of first grant from NIH was about 45: enough said.
A reproducibility mirage
If there were one central technical problem that Harris stresses, it is the number of results that fail to be reproducible in other studies. Irreproducible results leave us in limbo-land: how are we to interpret them? What are we supposed to believe? Which study--if any of them--is correct? Why are so many studies proudly claiming dramatic findings that can't be reproduced, and/or why are the news media and university PR offices so loudly proclaiming these reported results? What's wrong with our practices and standards?
Rigor Mortis goes through many of these issues, forthrightly and convincingly--showing that there is a problem. But a solution is not so easy to come by, because it would require major shifting of and reform in research funding. Naturally, that would be greatly resisted by hungry universities and those who they employ to set up a shopping-mall on their campus (i.e., faculty).
One purpose of this post is to draw attention to the wealth of reasons Harris presents for why we should be concerned about the state of play in biomedical research (and, indeed, in science more generally). I do have some caveats, that I'll discuss below, but that is in no way intended to diminish the points Harris makes in his book. What I want to add is a reason why I think that, if anything, Harris' presentation, strong and clear as it is, understates the problem. I say this because to me, there is a deeper issue, beyond the many Harris enumerates: a deeper scientific problem.
Reproducibility is only the tip of the iceberg!
Harris stresses or even focuses on the problem of irreproducible results. He suggests that if we were to hold far higher evidentiary standards, our work would be reproducible, and the next study down the line wouldn't routinely disagree with its predecessors. From the point of view of careful science and proper inferential methods and the like, this is clearly true. Many kinds of studies in biomedical and psychological sciences should have a standard of reporting that leads to at least some level of reproducibility.
However, I think that the situation is far more problematic than sloppy and hasty standards, or questionable statistics, even if they are clearly a prominent ones. My view is that no matter how high our methodological standards are, the expectation of reproducibility flies in the face of what we know about life. That is because life is not a reproducible phenomenon in the way physics and chemistry are!
Life is the product of evolution. Nobody with open eyes can fail to understand that, and this applies to biological, biomedical, psychological and social scientists. Evolution is at its very core a phenomenon that rests essentially on variation--on not being reproducible. Each organism, indeed each cell, is different. Not even 'identical' twins are identical.
One reason for this is that genetic mutations are always occurring, even among the cells within our bodies. Another reason is that no two organisms are experiencing the same environment, and environmental factors affect and interact with the genomes of each individual organism of any species. Organisms affect their environments in turn. These are dynamic phenomena and are not replicable!
This means that, in general, we should not be expecting reproducibility of results. But one shouldn't overstate this because while obviously the fact that two humans are different doesn't mean they are entirely different. Similarity is correlated with kinship, from first-degree relatives to members of populations, species, and different species. The problem is not that there is similarity, it is that we have no formal theory about how much similarity. We know two samples of people will differ both among those in each sample and between samples. And, even the same people sampled at separate times will be different, due to aging, exposure to different environments and so on. Proper statistical criteria and so on can answer questions about whether differences seem only due to sampling from variation or from causal differences. But that is a traditional assumption from the origin of statistics and probability, and isn't entirely apt for biology: since we cannot assume identity of individuals, much less of samples or populations (or species, as in using mouse models for human disease), our work requires some understanding of how much difference, or what sort of difference, we should expect--and build into our models and tests etc.
Evolution is by its very nature an ad hoc phenomenon in both time and place, meaning that there are no fixed rules about this, as there are laws of gravity or of chemical reactions. That means that reproducibility is not, in itself, even a valid criterion for judging scientific results. Some reproducibility should be expected, but we have no rule for how much and, indeed, evolution tells us that there is no real rule for that.
One obvious and not speculative exemplar of the problem is the redundancy in our systems. Genomewide mapping has documented this exquisitely well: if variation at tens, hundreds, or sometimes even thousands of genome sites' affects a trait, like blood pressure, stature, or 'intelligence' and no two people have the same genotype, then no two people, even with the same trait measure have that measure for the same reason. And as is very well known, mapping only accounts for a fraction of the estimated heritability of the studied traits, meaning that much or usually most of the contributing genetic variation is unidentified. And then there's the environment. . . . .
It's a major problem. It's an inconvenient truth. The sausage-grinder system of science 'productivity' cannot deal with it. We need reform. Where can that come from?
Showing posts with label scientific method. Show all posts
Showing posts with label scientific method. Show all posts
Monday, August 14, 2017
Tuesday, July 5, 2016
When scientific theory constrains
It's good from time to time to reflect on how we know what we think we know. And to remember that, as it has been in any time in history, much of what we now think is true will sooner or later be found to be false or, often, only inaccurately or partially correct. Some of this is because values change -- not so long ago homosexuality was considered to be an illness, e.g. Some is because of new discoveries -- when archaea were first discovered they were thought to be exotic microbes that inhabited extreme environments but now they're known to live in all environments, even in and on us. And of course these are just two of countless examples.
But what we think we know can be influenced by our assumptions about what we think is true, too. It's all too easy to look at data and interpret it in a way that makes sense to us, even if there are multiple possible interpretations. This can be a particular problem in social science, when we've got a favorite theory and the data can be seen to confirm it; this is perhaps easiest to notice if you yourself aren't wedded to any of the theories. But it's also true in biology. It is understandable that we want to assert that we now know something, and are rewarded for insight and discoveries, rather than more humbly hesitating to make claims.
Charitable giving
The other day I was listening to the BBC Radio 4 program Analysis on the charitable impulse. Why do people give to charity? It turns out that a lot of psychological research has been done on this, to the point that charities are now able to manipulate us into giving. If you call your favorite NPR station to donate during a fund drive, e.g., if you're told that the caller just before you gave a lot of money, you're more likely to make a larger donation than if you're told the previous caller pledged a small amount.
Or, if an advertisement pictures one child, and tells us the story of that one child, we're more likely to donate than if we're told about 30,000 needy children. This works even if we're told the story of two children, one after the other. But, according to one of the researchers, if we're shown two children at once, and told that if we give, the money will randomly go to just one of the children, we're less likely to give. This researcher interpreted this to mean that two is too many.
But there seem to me to be other possible interpretations given that the experiment changes more than one variable. Perhaps it's that we don't like the idea that someone else will choose who gets our money. Or that we feel uncomfortable knowing that we've helped only one child when two are needy. But surely something other than that two is too many, given that in 2004 so many people around the world donated so much money to organizations helping tsunami victims that many had to start turning down donations. These were anonymous victims, in great numbers. Though, as the program noted, people weren't nearly as generous to the great number of victims of the earthquake in Nepal in 2015, with no obvious explanation.
The researcher did seem to be wedded to his one vs too many interpretation, despite the contradictory data. In fact, I would suggest that the methods, given what were presented, don't allow him to legitimately draw any conclusion. Yet he readily did.
Thinness microbes?
The Food Programme on BBC Radio 4 is on to the microbiome in a big way. Two recent episodes (here and here) explore the connection between gut microbes, food, and health and the program promises to update us as new understanding develops. As we all know by now, the microbiome, the bug intimates that accompany us through life, in and on our body, may affect our health, our weight, our behavior, and perhaps much more. Or not.
Obesity, asthma, atopy, periodontal health, rheumatoid arthritis, Parkinson's, Alzheimer's, autism, and many many more conditions have been linked with, or are suggested to be linked with, in one way or another, our microbiome. Perhaps we're hosting the wrong microbes, or not a diverse enough set of microbes, or we wipe the good ones out with antibiotics along with the bad, or with alcohol, and what we eat may have a lot to do with this.
One of the researchers interviewed for the program was experimenting with a set of identical twins in Scotland. He varied their diets having them eat, for example, lots of junk food and alcohol, or a very fibrous diet, and documented changes in their gut microbiomes which apparently can change pretty quickly with changes in diet. The most diverse microbiome was associated with the high fiber diet. Researchers seem to feel that diversity is good.
Along with a lot of enthusiasm and hype, though, mostly what we've got in microbiome research so far is correlations. Thin people tend to have a different set of microbes than obese people, and people with a given neurological disease might statistically share a specific subset of microbes. But this tells us nothing about cause and effect -- which came first, the microbiome or the condition? And because the microbiome can change quickly and often, how long and how consistently would an organism have to reside in our gut before it causes a disease?
There was some discussion of probiotics in the second program, the assumption being that controlling our microbiome affects our health. Perhaps we'll soon have probiotic yogurt or kefir or even a pill that keeps us thin, or prevents Alzheimer's disease. Indeed, this was the logical conclusion from all the preceding discussion.
But one of the researchers, inadvertently I think, suggested that perhaps this reductionist conclusion was unwarranted. He cautioned that thinking about probiotic pills rather than lifestyle might be counterproductive. But except for factors with large effects such as smoking, the effect of "lifestyle" on health is rarely obvious. We know that poverty, for example, is associated with ill health, but it's not so easy to tease out how and why. And, if the microbiome really does directly influence our health, as so many are promising, the only interesting relevant thing about lifestyle would be how it changes our microbiomic makeup. Otherwise, we're talking about complexity, multiple factors with small effects -- genes, environmental factors, diet, and so on, and all bets about probiotics and "the thinness microbiome" are off. But, the caution was, to my mind, an important warning about the problem of assuming we know what we think we know; in this case, that the microbiome is the ultimate cause of disease.
The problem of theory
These are just two examples of the problem of assumption-driven science. They are fairly trivial, but if you are primed to notice, you'll see it all around you. Social science research is essentially the interpretation of observational data from within a theoretical framework. Psychologists might interpret observations from the perspective of behavioral, or cognitive, or biological psychology, e.g., and anthropologists, at least historically, from, say, a functionalist or materialist or biological or post-modernist perspective. Even physicists interpret data based on whether they are string theorists or particle physicists.
And biologists' theoretical framework? I would suggest that two big assumptions that biologists make are reductionism and let's call it biological uniformitarianism. We believe we can reduce causation to a single factor, and we assume that we can extrapolate our findings from the mouse or zebrafish we're working on to other mice, fish and species, or from one or some people to all people. That is, we assume invariance rather than that what we can expect is variation. There is plenty of evidence to show that by now we should know better.
True, most biologists would probably say that evolutionary theory is their theoretical framework, and many would add that traits are here because they're adaptive, because of natural selection. Evolution does connect people to each other and people to other species, it has done so by working on differences, not replicated identity, and there is no rule for the nature or number of those differences or for extrapolating from one species or individual to another. We know nothing to contradict evolutionary theory, but that every trait is adaptive is an assumption, and a pervasive one.
But what we think we know can be influenced by our assumptions about what we think is true, too. It's all too easy to look at data and interpret it in a way that makes sense to us, even if there are multiple possible interpretations. This can be a particular problem in social science, when we've got a favorite theory and the data can be seen to confirm it; this is perhaps easiest to notice if you yourself aren't wedded to any of the theories. But it's also true in biology. It is understandable that we want to assert that we now know something, and are rewarded for insight and discoveries, rather than more humbly hesitating to make claims.
Charitable giving
The other day I was listening to the BBC Radio 4 program Analysis on the charitable impulse. Why do people give to charity? It turns out that a lot of psychological research has been done on this, to the point that charities are now able to manipulate us into giving. If you call your favorite NPR station to donate during a fund drive, e.g., if you're told that the caller just before you gave a lot of money, you're more likely to make a larger donation than if you're told the previous caller pledged a small amount.
 |
| A 1931 advertisement for the British charity, Barnardo's Homes; Wikipedia |
Or, if an advertisement pictures one child, and tells us the story of that one child, we're more likely to donate than if we're told about 30,000 needy children. This works even if we're told the story of two children, one after the other. But, according to one of the researchers, if we're shown two children at once, and told that if we give, the money will randomly go to just one of the children, we're less likely to give. This researcher interpreted this to mean that two is too many.
But there seem to me to be other possible interpretations given that the experiment changes more than one variable. Perhaps it's that we don't like the idea that someone else will choose who gets our money. Or that we feel uncomfortable knowing that we've helped only one child when two are needy. But surely something other than that two is too many, given that in 2004 so many people around the world donated so much money to organizations helping tsunami victims that many had to start turning down donations. These were anonymous victims, in great numbers. Though, as the program noted, people weren't nearly as generous to the great number of victims of the earthquake in Nepal in 2015, with no obvious explanation.
The researcher did seem to be wedded to his one vs too many interpretation, despite the contradictory data. In fact, I would suggest that the methods, given what were presented, don't allow him to legitimately draw any conclusion. Yet he readily did.
Thinness microbes?
The Food Programme on BBC Radio 4 is on to the microbiome in a big way. Two recent episodes (here and here) explore the connection between gut microbes, food, and health and the program promises to update us as new understanding develops. As we all know by now, the microbiome, the bug intimates that accompany us through life, in and on our body, may affect our health, our weight, our behavior, and perhaps much more. Or not.
 |
| Pseudomonas aeruginosa, Enterococcus faecalis and Staphylococcus aureus on Tryptic Soy Agar. Wikipedia |
Obesity, asthma, atopy, periodontal health, rheumatoid arthritis, Parkinson's, Alzheimer's, autism, and many many more conditions have been linked with, or are suggested to be linked with, in one way or another, our microbiome. Perhaps we're hosting the wrong microbes, or not a diverse enough set of microbes, or we wipe the good ones out with antibiotics along with the bad, or with alcohol, and what we eat may have a lot to do with this.
One of the researchers interviewed for the program was experimenting with a set of identical twins in Scotland. He varied their diets having them eat, for example, lots of junk food and alcohol, or a very fibrous diet, and documented changes in their gut microbiomes which apparently can change pretty quickly with changes in diet. The most diverse microbiome was associated with the high fiber diet. Researchers seem to feel that diversity is good.
Along with a lot of enthusiasm and hype, though, mostly what we've got in microbiome research so far is correlations. Thin people tend to have a different set of microbes than obese people, and people with a given neurological disease might statistically share a specific subset of microbes. But this tells us nothing about cause and effect -- which came first, the microbiome or the condition? And because the microbiome can change quickly and often, how long and how consistently would an organism have to reside in our gut before it causes a disease?
There was some discussion of probiotics in the second program, the assumption being that controlling our microbiome affects our health. Perhaps we'll soon have probiotic yogurt or kefir or even a pill that keeps us thin, or prevents Alzheimer's disease. Indeed, this was the logical conclusion from all the preceding discussion.
But one of the researchers, inadvertently I think, suggested that perhaps this reductionist conclusion was unwarranted. He cautioned that thinking about probiotic pills rather than lifestyle might be counterproductive. But except for factors with large effects such as smoking, the effect of "lifestyle" on health is rarely obvious. We know that poverty, for example, is associated with ill health, but it's not so easy to tease out how and why. And, if the microbiome really does directly influence our health, as so many are promising, the only interesting relevant thing about lifestyle would be how it changes our microbiomic makeup. Otherwise, we're talking about complexity, multiple factors with small effects -- genes, environmental factors, diet, and so on, and all bets about probiotics and "the thinness microbiome" are off. But, the caution was, to my mind, an important warning about the problem of assuming we know what we think we know; in this case, that the microbiome is the ultimate cause of disease.
The problem of theory
These are just two examples of the problem of assumption-driven science. They are fairly trivial, but if you are primed to notice, you'll see it all around you. Social science research is essentially the interpretation of observational data from within a theoretical framework. Psychologists might interpret observations from the perspective of behavioral, or cognitive, or biological psychology, e.g., and anthropologists, at least historically, from, say, a functionalist or materialist or biological or post-modernist perspective. Even physicists interpret data based on whether they are string theorists or particle physicists.
And biologists' theoretical framework? I would suggest that two big assumptions that biologists make are reductionism and let's call it biological uniformitarianism. We believe we can reduce causation to a single factor, and we assume that we can extrapolate our findings from the mouse or zebrafish we're working on to other mice, fish and species, or from one or some people to all people. That is, we assume invariance rather than that what we can expect is variation. There is plenty of evidence to show that by now we should know better.
True, most biologists would probably say that evolutionary theory is their theoretical framework, and many would add that traits are here because they're adaptive, because of natural selection. Evolution does connect people to each other and people to other species, it has done so by working on differences, not replicated identity, and there is no rule for the nature or number of those differences or for extrapolating from one species or individual to another. We know nothing to contradict evolutionary theory, but that every trait is adaptive is an assumption, and a pervasive one.
Theory and assumption can guide us, but they can also improperly constrain how we think about our data, which is why it's good to remind ourselves from time to time to think about how we know what we think we know. As scientists we should always be challenging and testing our assumptions and theories, not depending on them to tell us that we're right.
Tuesday, October 6, 2015
The Blind Men and the Elephant -- a post-modern parable
It's an ancient parable; a group of blind men are lead to an elephant and asked to describe what they feel. One feels a tusk, another a foot, a third the tail, and so on, and of course they disagree entirely about what it is they are feeling. This tale is usually used as an illustration of the subjectivity of our view of reality, but I think it's more than that.
I heard a talk by Anthropologist Agustin Fuentes here at Penn State the other day, on blurring the boundaries between science and the humanities. He used the parable to illustrate why science needs the humanities and vice versa; each restricted view of the world is enhanced by the other to become complete.
But, this assumes that the tales that science tells, and the tales that the humanities tell are separate but equally true -- scientists feel the tail, humanities feel the tusk and accurately report what they feel. Once they listen to each other's tales, they can describe the whole elephant.
But I don't think so. I don't think that all that scientists are missing is a humanities perspective, and vice versa. I think in a very real sense we're all blind all of the time, and there's no way to know what we're missing and when it matters. You feel the tusk, and you might be able to describe it, but you have no clue what it's made of. Or, you feel the tail but you have no idea what the elephant uses it for, if anything.
Here's my own personal version of the same parable -- some years ago we purchased a new landline with answering machine. Oddly, we have a lot of power outages here, and it seemed that every time I set the time and day on the answering machine, we'd have another outage and the time and day would disappear, having to be set once again. I decided that was a nuisance, and I stopped setting time and day.
The next time the machine said we had a message, I listened to it, but it was blank. There was no message! Naturally enough (I thought), I concluded that the time and day had to be set for the machine to record a message. Unhappy consumers, we contacted the maker, and they said no, the machine should record the message anyway. Which of course it would have if the caller had left a message, as was proven the next time someone called on unknown day at unknown time and ... left a message.
My conclusion was reasonable enough for the data I had, right? It just happened not to be based on adequate data (aka reality). But, we always think we've got enough data to draw a conclusion, no matter how much we're in fact missing. This is true in epidemiology, genetics, medical testing, the humanities, interpersonal relationships; we think we know enough about our partner to commit to marrying him or her, but half of us turn out to be wrong. Indeed, if all you've seen are white swans, you'll conclude that all swans are white -- until you see your first black one.
No, you say, we did power tests and we know we've got enough subjects to conclude that gene X causes disease Y. But, it's possible that all your subjects are from western Europe, or even better, England, say, and what you've done is identify a gene everyone shares because they share a demographic history. You won't know that until you look at people with the same disease from a different part of the world -- until you collect more data. Until you see your first black swan.
But, you say, no one would make such an elementary mistake now -- you've drawn you controls from the same population, and they will share the same population-specific allele, so differences between cases and controls will be disease-specific. But, western Europe is a big area, and even England is heterogeneous, and it's possible that everyone with your disease is more closely related than people without. So, you really might have identified population structure rather than a disease allele but you can't know, until you collect more data -- you look at additional populations, or more people in the same population.
Even then, say you look at additional populations and you don't find the same supposedly causal allele. You can't know why -- is it causal in one population and not another? Is it not causal in any population, and your initial finding merely an artifact of ill-conceived study design?
Without belaboring this particular example any further, I hope the point is clear. You feel the tail, but that doesn't tell you everything about the tail. But you can't know what you're missing until you ask more questions, and gather more data.
Darwin explained inheritance with his idea of gemmules. He was wrong, of course, but he had no way to know how or why, and it wasn't until Mendel's work was rediscovered in 1900 that people could move on. Everything we know about genetics we've learned since then, but that doesn't mean we know everything about genetics. But theories of inheritance (and much else) don't include acknowledgement of glaring holes: "My theory is obviously inadequate because, as always, there is a lot we don't yet understand but we don't know what that is so I'm leaving gaps, but I don't know how big or how many." And, in a related issue that we write about frequently here, it's also true that instead of coming clean, we often claim more than we know (and often we know what we're doing in doing so).
Even very sophisticated theories just 15 or 20 years ago had no way to include, say, epigenetics, or the importance of transcribed but untranslated RNAs (that is, RNA not coding for genes but doing a variety of other things, some of them still unknown), or interfering RNAs, and so on, and we have no idea today what we'll learn tomorrow. But, like the blind men, we act as though we can draw adequate conclusions from the data we've got.
Science is about pushing into the unknown. But, because it's unknown, we have no idea how far we need to push. I think in most cases, there's always further, we're never done, but we often labor under the illusion that we are. Or, that we're close.
But, should ductal cancer in situ, a form of breast cancer, be treated? And how will we know for sure? Systems biology sounds like a great idea, but how will we ever know we've taken enough of a given system into account to explain what we're trying to explain? Will physicists ever know whether the multiverse, or the symmetry theory is correct (whatever those elusive ideas actually mean!)?
Phlogiston was once real, as were miasma and phrenology, the four humors, and the health benefits of smoking. It's not that we don't make progress -- we do now know that smoking is bad for our health (even if only 10% of smokers get lung cancer; ok, smoking is associated with a lot of other diseases as well, so better not to smoke) -- but we've always got the modern equivalent of phlogiston and phrenology. We just don't know which they are. We're still groping the elephant in the dark.
I heard a talk by Anthropologist Agustin Fuentes here at Penn State the other day, on blurring the boundaries between science and the humanities. He used the parable to illustrate why science needs the humanities and vice versa; each restricted view of the world is enhanced by the other to become complete.
But, this assumes that the tales that science tells, and the tales that the humanities tell are separate but equally true -- scientists feel the tail, humanities feel the tusk and accurately report what they feel. Once they listen to each other's tales, they can describe the whole elephant.
 |
| "Blind monks examining an elephant" by Hanabusa ItchÅ (Wikipedia) |
But I don't think so. I don't think that all that scientists are missing is a humanities perspective, and vice versa. I think in a very real sense we're all blind all of the time, and there's no way to know what we're missing and when it matters. You feel the tusk, and you might be able to describe it, but you have no clue what it's made of. Or, you feel the tail but you have no idea what the elephant uses it for, if anything.
Here's my own personal version of the same parable -- some years ago we purchased a new landline with answering machine. Oddly, we have a lot of power outages here, and it seemed that every time I set the time and day on the answering machine, we'd have another outage and the time and day would disappear, having to be set once again. I decided that was a nuisance, and I stopped setting time and day.
The next time the machine said we had a message, I listened to it, but it was blank. There was no message! Naturally enough (I thought), I concluded that the time and day had to be set for the machine to record a message. Unhappy consumers, we contacted the maker, and they said no, the machine should record the message anyway. Which of course it would have if the caller had left a message, as was proven the next time someone called on unknown day at unknown time and ... left a message.
My conclusion was reasonable enough for the data I had, right? It just happened not to be based on adequate data (aka reality). But, we always think we've got enough data to draw a conclusion, no matter how much we're in fact missing. This is true in epidemiology, genetics, medical testing, the humanities, interpersonal relationships; we think we know enough about our partner to commit to marrying him or her, but half of us turn out to be wrong. Indeed, if all you've seen are white swans, you'll conclude that all swans are white -- until you see your first black one.
No, you say, we did power tests and we know we've got enough subjects to conclude that gene X causes disease Y. But, it's possible that all your subjects are from western Europe, or even better, England, say, and what you've done is identify a gene everyone shares because they share a demographic history. You won't know that until you look at people with the same disease from a different part of the world -- until you collect more data. Until you see your first black swan.
But, you say, no one would make such an elementary mistake now -- you've drawn you controls from the same population, and they will share the same population-specific allele, so differences between cases and controls will be disease-specific. But, western Europe is a big area, and even England is heterogeneous, and it's possible that everyone with your disease is more closely related than people without. So, you really might have identified population structure rather than a disease allele but you can't know, until you collect more data -- you look at additional populations, or more people in the same population.
Even then, say you look at additional populations and you don't find the same supposedly causal allele. You can't know why -- is it causal in one population and not another? Is it not causal in any population, and your initial finding merely an artifact of ill-conceived study design?
Without belaboring this particular example any further, I hope the point is clear. You feel the tail, but that doesn't tell you everything about the tail. But you can't know what you're missing until you ask more questions, and gather more data.
Darwin explained inheritance with his idea of gemmules. He was wrong, of course, but he had no way to know how or why, and it wasn't until Mendel's work was rediscovered in 1900 that people could move on. Everything we know about genetics we've learned since then, but that doesn't mean we know everything about genetics. But theories of inheritance (and much else) don't include acknowledgement of glaring holes: "My theory is obviously inadequate because, as always, there is a lot we don't yet understand but we don't know what that is so I'm leaving gaps, but I don't know how big or how many." And, in a related issue that we write about frequently here, it's also true that instead of coming clean, we often claim more than we know (and often we know what we're doing in doing so).
Even very sophisticated theories just 15 or 20 years ago had no way to include, say, epigenetics, or the importance of transcribed but untranslated RNAs (that is, RNA not coding for genes but doing a variety of other things, some of them still unknown), or interfering RNAs, and so on, and we have no idea today what we'll learn tomorrow. But, like the blind men, we act as though we can draw adequate conclusions from the data we've got.
Science is about pushing into the unknown. But, because it's unknown, we have no idea how far we need to push. I think in most cases, there's always further, we're never done, but we often labor under the illusion that we are. Or, that we're close.
But, should ductal cancer in situ, a form of breast cancer, be treated? And how will we know for sure? Systems biology sounds like a great idea, but how will we ever know we've taken enough of a given system into account to explain what we're trying to explain? Will physicists ever know whether the multiverse, or the symmetry theory is correct (whatever those elusive ideas actually mean!)?
Phlogiston was once real, as were miasma and phrenology, the four humors, and the health benefits of smoking. It's not that we don't make progress -- we do now know that smoking is bad for our health (even if only 10% of smokers get lung cancer; ok, smoking is associated with a lot of other diseases as well, so better not to smoke) -- but we've always got the modern equivalent of phlogiston and phrenology. We just don't know which they are. We're still groping the elephant in the dark.
Monday, September 7, 2015
How do we know what we think we know?
Two stories collided yesterday to make me wonder, yet again, how we know what we think we know. The first was from the latest BBC Radio 4 program The Inquiry, an episode called "Can we learn to live with nuclear power?" which discusses the repercussions of the 2011 disaster in the Fukushima nuclear power plant in Japan. It seems that some of us can live with nuclear power and some of us can't, even when we're looking at the same events and the same facts. So, for example, Germans were convinced by the disaster that nuclear power isn't reliably safe and so they are abandoning it, but in France, nuclear power is still an acceptable option. Indeed most of the electricity in France comes from nuclear power.
Why didn't the disaster convince everyone that nuclear power is unsafe? Indeed, some saw the fact that there were no confirmed deaths attributable to the disaster as proof that nuclear power is safe, while others saw the whole event as confirmation that nuclear power is a disaster waiting to happen. According to The Inquiry, a nation's history has a lot to do with how it reads the facts. Germany's history is one of division and war, and nuclear power associated with bombs, but French researchers and engineers have long been involved in the development of nuclear power, so there's a certain amount of national pride in this form of energy. It may not be an unrelated point that therefore many people in France have vested interests in nuclear power. Still, same picture, different reading of it.
Reading ability is entirely genetic
And, I was alerted to yet another paper reporting that intelligence is genetic (h/t Mel Bartley); this time it's reading ability, for which no environmental effect was found (or acknowledged). (This idea of little to no environmental effect is an interesting one, though, given that the authors, who are Dutch, report that heritability of dyslexia and reading fluency is higher among Dutch readers -- 80% compared with 45-70% elsewhere -- they suggest because Dutch orthography is simpler than that of English. This sounds like an environmental effect to me.)
The authors assessed reading scores for twins, parents and siblings, and used these to evaluate additive and non-additive genetic effects, and family environmental factors. As far as I can tell, subjects were asked to read aloud from a list of Dutch words, and the number they read correctly within a minute constituted their score. And again, as far as I can tell, they did not test for nor select for children or parents with dyslexia, but they seem to be reporting results as though they apply to dyslexia.
The authors report a high correlation in reading ability between monozygotic twins, a lower correlation between dizygotic twins, and between twins and siblings, and a higher correlation between spouses, which to the authors is evidence of assortative mating (choice of mate based on traits associated with reading ability). They conclude:
The ideas about additivity are technical and subtle. Dominant effects, that is, non-additive interactions among alleles within a gene in the diploid copies of an individual, are not inherited as additive ones are (if you are a Dd and that determines your trait, only one of those alleles, and hence not enough to determine the trait, is transmitted to any of your offspring). Likewise, interactions (between loci), called epistasis, is also not directly transmitted.
There are many practical as well as political reasons to believe that interactions can be ignored. In a practical sense, even multiple 2-way interactions make impossible sample size and structure demands. But in a political sense, additive effects mean that traits can be reliably predicted from genotype data (meaning, even at birth): you estimate the effects of each allele at each place in the genome, and add them to get the predicted phenotype. There is money to be made by that, so to speak. But it doesn't really work with complex interactions. Strong incentives, indeed, to report additive effects and very understandable!
Secondly, all these various effects are estimated from samples, not derived from basic theory about molecular-level physiology, and often they are hardly informed by the latter at all. This means that replication is not to be expected in any rigorous sense. For example, dominance is estimated by the deviation of average traits in AA, Aa, and aa individuals from being in 0, 1, 2 proportions if (say) the 'a' allele contributed 1-unit of trait measure. Dominance deviations are thoroughly sample-dependent. It is not easy to interpret those results when samples cannot be replicated (the concepts are very useful in agricultural and experimental breeding contexts, but far less so in natural human populations). And this conveniently overlooks the environmental effects.
This study is of a small sample, especially since for many traits it now seems de rigueur to have samples of hundreds of thousands to get reliable mapping results, not to mention a confusingly defined trait, so it's difficult, at least for me, to make sense of the results. In theory, it wouldn't be terribly surprising to find a genetic component to risk of reading disability, but it would be surprising, particularly since disability is defined only by test score in this study, if none of that ability was substantially affected by environment. In the extreme, if a child hasn't been to school or otherwise learned to read, that inability would be largely determined by environmental factors, right? Even if an entire family couldn't read, it's not possible to know whether it's because no one ever had the chance to learn, or they share some genetic risk allele.
In people, unlike in other animals, assortative mating has a huge cultural component, so, again, it wouldn't be surprising if two illiterate adults married, or if they then had no books in the house, and didn't teach their children that reading was valuable. But this doesn't mean either reading or their mate-choice necessarily has any genetic component.
So, again, same data, different interpretations
But why? Indeed, what makes some Americans hear Donald Trump and resonate with his message, while others cringe? Why do we need 9 Supreme Court justices if the idea is that evidence for determination of the constitutionality of a law is to be found in the Constitution? Why doesn't just one justice suffice? And, why do they look at the same evidence and reliably and predictably vote along political lines?
Or, more uncomfortably for scientists, why did some people consider it good news when it was announced that only 34% of replicated psychology experiments agreed with the original results, while others considered this unfortunate? Again, same facts, different conclusions.
Why do our beliefs determine our opinions, even in science, which is supposed to be based on the scientific method, and sober, unbiased assessment of the data? Statistics, like anything, can be manipulated, but done properly they at least don't lie. But, is IQ real or isn't it? Are behavioral traits genetically determined or aren't they? Have genome wide association studies been successful or not?
As Ken often writes, much of how we view these things is certainly determined by vested interest and careerism, not to mention the emotional positions we inevitably take on human affairs. If your lab spends its time and money on GWAS, you're more likely to see them as successful. That's undeniable if you are candid. But, I think it's more than that. I think we're too often prisoners of induction, based on our experience, training, predilections of what observations we make or count as significant; our conclusions are often underdetermined, but we don't know it. Underdetermined systems are those that are accounted for with not enough evidence. It's the all-swans-are-white problem; they're all white until we see a black one. At which point we either conclude we were wrong, or give the black swan a different species name. But, we never know if or when we're going to see a black one. Or a purple one.
John Snow determined to his own satisfaction during the cholera epidemic in London in 1854 that cholera was transmitted by a contagion in the water. But in fact he didn't prove it. The miasmatists, who believed cholera was caused by bad air, had stacks of evidence of their own -- e.g., infection was more common in smoggy, smelly cities, and in fact in the dirtier sections of cities. But both Snow and the miasmatists had only circumstantial evidence, correlations, not enough data to definitively prove their were right. Both arguments were underdetermined. As it happened, John Snow was right, but that wasn't to be widely known for another few decades when vibrio cholerae was identified under Robert Koch's microscope.
Both sides strongly (emotionally!) believed they were right, believed they had the evidence to support their argument. They weren't cherry-picking the data to better support their side, they were looking at the same data and drawing different conclusions. They based their conclusions on the data they had, but they had no idea it wasn't enough.
But it's not just that, either. It's also that we're predisposed by our beliefs to form our opinions. And that's when we're likely to cherry pick the evidence that supports our beliefs. Who's right about immigrants to the US, Donald Trump or Bernie Sanders? Who's right about whether corporations are people or not? Who's right about genetically modified organisms? Or climate change? Who's right about behavior and genetic determinism?
And it's even more than that! If genetics and evolutionary biology have taught us anything, they've taught us about complexity. Even simple traits turn out to be complex. There are multiple pathways to most traits, most traits are due to interacting polygenes and environmental factors, and so on. Simple explanations are less likely to be correct than explanations that acknowledge complexity, and that's because evolution doesn't follow rules, except that what works works, and to an important degree that's what is here to be examined today.
Simplistic explanations are probably wrong. But they are so appealing.
Why didn't the disaster convince everyone that nuclear power is unsafe? Indeed, some saw the fact that there were no confirmed deaths attributable to the disaster as proof that nuclear power is safe, while others saw the whole event as confirmation that nuclear power is a disaster waiting to happen. According to The Inquiry, a nation's history has a lot to do with how it reads the facts. Germany's history is one of division and war, and nuclear power associated with bombs, but French researchers and engineers have long been involved in the development of nuclear power, so there's a certain amount of national pride in this form of energy. It may not be an unrelated point that therefore many people in France have vested interests in nuclear power. Still, same picture, different reading of it.
 |
| Cattenom nuclear power plant, France; Wikipedia |
Reading ability is entirely genetic
And, I was alerted to yet another paper reporting that intelligence is genetic (h/t Mel Bartley); this time it's reading ability, for which no environmental effect was found (or acknowledged). (This idea of little to no environmental effect is an interesting one, though, given that the authors, who are Dutch, report that heritability of dyslexia and reading fluency is higher among Dutch readers -- 80% compared with 45-70% elsewhere -- they suggest because Dutch orthography is simpler than that of English. This sounds like an environmental effect to me.)
The authors assessed reading scores for twins, parents and siblings, and used these to evaluate additive and non-additive genetic effects, and family environmental factors. As far as I can tell, subjects were asked to read aloud from a list of Dutch words, and the number they read correctly within a minute constituted their score. And again, as far as I can tell, they did not test for nor select for children or parents with dyslexia, but they seem to be reporting results as though they apply to dyslexia.
The authors report a high correlation in reading ability between monozygotic twins, a lower correlation between dizygotic twins, and between twins and siblings, and a higher correlation between spouses, which to the authors is evidence of assortative mating (choice of mate based on traits associated with reading ability). They conclude:
Such a pattern of correlation among family members is consistent with a model that attributes resemblance to additive genetic factors, these are the factors that contribute to resemblance among all biological relatives, and to non-additive genetic factors. Non-additive genetic factors, or genetic dominance, contributes to resemblance among siblings, but not to the resemblance of parents and offspring. Maximum likelihood estimates for the additive genetic factors were 28% (CI: 0–43%) and for dominant genetic factors 36% (CI: 18–65%), resulting in a broad-sense heritability estimate of 64%. The remainder of the variance is attributed to unique environmental factors and measurement error (35%, CI: 29–44%).Despite this evidence for environmental effect (right?), the authors conclude, "Our results suggest that the precursors for reading disability observed in familial risk studies are caused by genetic, not environ- mental, liability from parents. That is, having family risk does not reflect experiencing a less favorable literacy environment, but receiving less favorable genetic variants."
The ideas about additivity are technical and subtle. Dominant effects, that is, non-additive interactions among alleles within a gene in the diploid copies of an individual, are not inherited as additive ones are (if you are a Dd and that determines your trait, only one of those alleles, and hence not enough to determine the trait, is transmitted to any of your offspring). Likewise, interactions (between loci), called epistasis, is also not directly transmitted.
There are many practical as well as political reasons to believe that interactions can be ignored. In a practical sense, even multiple 2-way interactions make impossible sample size and structure demands. But in a political sense, additive effects mean that traits can be reliably predicted from genotype data (meaning, even at birth): you estimate the effects of each allele at each place in the genome, and add them to get the predicted phenotype. There is money to be made by that, so to speak. But it doesn't really work with complex interactions. Strong incentives, indeed, to report additive effects and very understandable!
Secondly, all these various effects are estimated from samples, not derived from basic theory about molecular-level physiology, and often they are hardly informed by the latter at all. This means that replication is not to be expected in any rigorous sense. For example, dominance is estimated by the deviation of average traits in AA, Aa, and aa individuals from being in 0, 1, 2 proportions if (say) the 'a' allele contributed 1-unit of trait measure. Dominance deviations are thoroughly sample-dependent. It is not easy to interpret those results when samples cannot be replicated (the concepts are very useful in agricultural and experimental breeding contexts, but far less so in natural human populations). And this conveniently overlooks the environmental effects.
This study is of a small sample, especially since for many traits it now seems de rigueur to have samples of hundreds of thousands to get reliable mapping results, not to mention a confusingly defined trait, so it's difficult, at least for me, to make sense of the results. In theory, it wouldn't be terribly surprising to find a genetic component to risk of reading disability, but it would be surprising, particularly since disability is defined only by test score in this study, if none of that ability was substantially affected by environment. In the extreme, if a child hasn't been to school or otherwise learned to read, that inability would be largely determined by environmental factors, right? Even if an entire family couldn't read, it's not possible to know whether it's because no one ever had the chance to learn, or they share some genetic risk allele.
In people, unlike in other animals, assortative mating has a huge cultural component, so, again, it wouldn't be surprising if two illiterate adults married, or if they then had no books in the house, and didn't teach their children that reading was valuable. But this doesn't mean either reading or their mate-choice necessarily has any genetic component.
So, again, same data, different interpretations
But why? Indeed, what makes some Americans hear Donald Trump and resonate with his message, while others cringe? Why do we need 9 Supreme Court justices if the idea is that evidence for determination of the constitutionality of a law is to be found in the Constitution? Why doesn't just one justice suffice? And, why do they look at the same evidence and reliably and predictably vote along political lines?
Or, more uncomfortably for scientists, why did some people consider it good news when it was announced that only 34% of replicated psychology experiments agreed with the original results, while others considered this unfortunate? Again, same facts, different conclusions.
Why do our beliefs determine our opinions, even in science, which is supposed to be based on the scientific method, and sober, unbiased assessment of the data? Statistics, like anything, can be manipulated, but done properly they at least don't lie. But, is IQ real or isn't it? Are behavioral traits genetically determined or aren't they? Have genome wide association studies been successful or not?
As Ken often writes, much of how we view these things is certainly determined by vested interest and careerism, not to mention the emotional positions we inevitably take on human affairs. If your lab spends its time and money on GWAS, you're more likely to see them as successful. That's undeniable if you are candid. But, I think it's more than that. I think we're too often prisoners of induction, based on our experience, training, predilections of what observations we make or count as significant; our conclusions are often underdetermined, but we don't know it. Underdetermined systems are those that are accounted for with not enough evidence. It's the all-swans-are-white problem; they're all white until we see a black one. At which point we either conclude we were wrong, or give the black swan a different species name. But, we never know if or when we're going to see a black one. Or a purple one.
John Snow determined to his own satisfaction during the cholera epidemic in London in 1854 that cholera was transmitted by a contagion in the water. But in fact he didn't prove it. The miasmatists, who believed cholera was caused by bad air, had stacks of evidence of their own -- e.g., infection was more common in smoggy, smelly cities, and in fact in the dirtier sections of cities. But both Snow and the miasmatists had only circumstantial evidence, correlations, not enough data to definitively prove their were right. Both arguments were underdetermined. As it happened, John Snow was right, but that wasn't to be widely known for another few decades when vibrio cholerae was identified under Robert Koch's microscope.
 |
| "The scent lies strong here; do you see anything?"; Wikipedia |
Both sides strongly (emotionally!) believed they were right, believed they had the evidence to support their argument. They weren't cherry-picking the data to better support their side, they were looking at the same data and drawing different conclusions. They based their conclusions on the data they had, but they had no idea it wasn't enough.
But it's not just that, either. It's also that we're predisposed by our beliefs to form our opinions. And that's when we're likely to cherry pick the evidence that supports our beliefs. Who's right about immigrants to the US, Donald Trump or Bernie Sanders? Who's right about whether corporations are people or not? Who's right about genetically modified organisms? Or climate change? Who's right about behavior and genetic determinism?
And it's even more than that! If genetics and evolutionary biology have taught us anything, they've taught us about complexity. Even simple traits turn out to be complex. There are multiple pathways to most traits, most traits are due to interacting polygenes and environmental factors, and so on. Simple explanations are less likely to be correct than explanations that acknowledge complexity, and that's because evolution doesn't follow rules, except that what works works, and to an important degree that's what is here to be examined today.
Simplistic explanations are probably wrong. But they are so appealing.
Monday, August 31, 2015
It's not just about psych studies; it's about core aspects of inference.
A paper just published in Science by the "Open Science Collaboration" reports the results of a multi-year multi-institution effort to replicate 100 psychology studies published in three top psychology journals in 2008. This effort has often been discussed since it began in 2011, in large part because the importance of replicability in confirming scientific results is integral to the 'scientific method,' but replicability studies aren't a terribly creative use of a researcher's time, and they're difficult to publish so they aren't often on researchers' To-Do lists. So, this was unusual.
There are many reasons a study can't be replicated. Sometimes the study was poorly conceived or carried out (assumptions and biases not taken into account), sometimes the results pertain only to the particular sample reported (a single family or population), sometimes the methods in an original study aren't described well enough to be replicated, sometimes random or even systematic error (instrument behaving badly) skews the results.
Because there's no such thing as a perfect study, replication studies can be victims of any of the same issues, so interpreting lack of replication isn't necessarily straightforward, and certainly doesn't always mean that the original study was flawed.
The Open Science Collaboration was scrupulous in its efforts to replicate original studies as carefully and faithfully as possible. Still, the results weren't pretty. The authors write:
This effort has been the subject of much discussion, naturally enough. In a piece published last week in The Atlantic, Ed Yong quotes several psychologists, including the project's lead author, saying that this project has been a welcome learning experience for the field. There are plans afoot to change how things are done, including pre-registration of hypotheses so that the reported results can't be cherry-picked, or increasing the size of studies to increase their power, as has been done in the field of genetics.
We'll see whether this is just a predictable wagon-circling welcome, or really means something. One has every reason to be skeptical, and wonder if these fields really are sciences in the proper sense of the term. Indeed, it's quite interesting to see genetics held up as an exemplar of good and reliable study design. After billions of dollars being spent on studies large and small of the genetics of asthma, heart disease, type 2 diabetes, obesity, hypertension, stroke, and so on, we've got not only a lot of contradictory findings, but most of what has been found are genes with small effects. And epidemiology, many of the 'omics fields, evolutionary biology, and others haven't done any better.
Why? The vagueness of the social and behavioral sciences is only part of the problem (unlike, say, force, outcome variables such as stress, aggression, crime, or intelligence are hard to consistently define, and can vary according to the instrument with which they are measured). Biomedical outcomes can be vague and hard to define as well (autism, schizophrenia, high blood pressure). We don't understand enough about how genes interact with each other or with the environment to understand complex causality.
Statistics and science
The problem may be much deeper than any of this discussion of non-replicable results suggests. First, from an evolutionary point of view, we expect organisms to be different, not replicates. This is because mutational changes (and recombination) are always making each individual organism's genotype unique and, second, the need to adapt--Darwin's central claim or observation--means that organisms have to be different so that their 'struggle for life' can occur.
We have only a general theory for this, since life is an ad hoc adaptive/evolutionary phenomenon. Far more broadly than just the behavioral or social sciences, our investigative methods are based on 'internal' comparisons (e.g., cases vs controls, various levels of blood pressure and stroke, fitness relative to different trait values) to evaluate samples against each other, rather than as representations of an externally derived, a priori theory. When we rely on statistics and p-value significance tests and probabilities and so on, we are implicitly confessing that we don't in fact really know what's going on, and all we can get are a kind of shadow of the underlying process that is cast by the differences we detect, and we detect them with generic (not to mention subjective) rather than specific criteria. We've written about these things several times in the past here.
The issue is not just weakly defined terms and study designs. As Freeman Dyson (in "A meeting with Enrico Fermi") wrote in 2004:
In a sense we are using Ptolemaic tinkering to fit a theory that doesn't really fit, in the absence of a better (e.g., Copernican or Newtonian) theoretical understanding. Social and behavioral sciences are far behind the at least considerably more rigorous genetic and evolutionary sciences when the latter are done at their best (which isn't always). Like the shadows of reality seen in Plato's cave, statistical inference reflects the shadows of the reality we want to understand, but for many cultural and practical reasons we don't recognize that, or don't want to acknowledge it. The weaknesses and frangibility of our predictive 'powers' could, if properly understood by the general public, be a threat to our business and our culture doesn't reward candor when it comes to that business. The pressures (including from the pubic media, with their own agenda and interests) necessarily lead to reducing complexity to simpler models and claims far beyond what has legitimately been understood.
The problem is not just with weakly measured variables or poorly defined terms of, for example, outcomes. Nor is the problem if, when, or that people use methods wrongly. The problem is that statistical inference is based on a sample and is often retrospective, or mainly empirical and based on only rather generic theory. No matter how well chosen and rigorously defined, in these various areas (unlike much of physics and chemistry) the estimates of parameters and the like fitted to data that is necessarily about the subjects past, such as their culture or upbringing or lifestyles, but in the absence of adequate formal theory, these findings cannot be used to predict the future with knowable accuracy. That is because the same conditions can't be repeated, say, decades from now, and we don't know what future conditions will be, and so on.
Rather alarmingly, we were recently discussing this with a colleague who works in very physics- and chemistry-rigorous material science. She immediately told us that they, too, face problems in data evaluation with the number of variables they have to deal with, even under what the rest of us would enviously say were very well-controlled conditions where the assumptions of statistics--basically amounting to replicability of some underlying mathematical process--should really apply well.
So the social and related sciences may be far weaker than other fields, and should acknowledge that. But the rest of us, in various purportedly 'harder' biological, biomedical, and epidemiological sciences, are often not so much better off. Statistical methods and theory work wonderfully well when their assumptions are closely met. But there is too much out-of-the-box analytic toolware, that lures us into thinking that quick and definitive answers are possible. Those methods never promise that because what statistics does is account for repeated phenomena following the same rules, and the rule of many sciences is that, in their essence they are not following such rules.
But the lure of easy-answer statistics, and the understandable lack of deeply better ideas, perpetuates the expensive and misleading games that we are playing in many areas of science.
 |
| Les Twins; Wikipedia |
Because there's no such thing as a perfect study, replication studies can be victims of any of the same issues, so interpreting lack of replication isn't necessarily straightforward, and certainly doesn't always mean that the original study was flawed.
The Open Science Collaboration was scrupulous in its efforts to replicate original studies as carefully and faithfully as possible. Still, the results weren't pretty. The authors write:
Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects.Interestingly enough, the authors aren't quite sure what any of this means. First, as they point out, direct replication doesn't verify the theoretical interpretation of the result, which could have been flawed originally, and remain flawed. And it's impossible to know when a study is not replicated why that is, whether the original was flawed or the replication effort was flawed, or even both were flawed.
This effort has been the subject of much discussion, naturally enough. In a piece published last week in The Atlantic, Ed Yong quotes several psychologists, including the project's lead author, saying that this project has been a welcome learning experience for the field. There are plans afoot to change how things are done, including pre-registration of hypotheses so that the reported results can't be cherry-picked, or increasing the size of studies to increase their power, as has been done in the field of genetics.
We'll see whether this is just a predictable wagon-circling welcome, or really means something. One has every reason to be skeptical, and wonder if these fields really are sciences in the proper sense of the term. Indeed, it's quite interesting to see genetics held up as an exemplar of good and reliable study design. After billions of dollars being spent on studies large and small of the genetics of asthma, heart disease, type 2 diabetes, obesity, hypertension, stroke, and so on, we've got not only a lot of contradictory findings, but most of what has been found are genes with small effects. And epidemiology, many of the 'omics fields, evolutionary biology, and others haven't done any better.
Why? The vagueness of the social and behavioral sciences is only part of the problem (unlike, say, force, outcome variables such as stress, aggression, crime, or intelligence are hard to consistently define, and can vary according to the instrument with which they are measured). Biomedical outcomes can be vague and hard to define as well (autism, schizophrenia, high blood pressure). We don't understand enough about how genes interact with each other or with the environment to understand complex causality.
Statistics and science
The problem may be much deeper than any of this discussion of non-replicable results suggests. First, from an evolutionary point of view, we expect organisms to be different, not replicates. This is because mutational changes (and recombination) are always making each individual organism's genotype unique and, second, the need to adapt--Darwin's central claim or observation--means that organisms have to be different so that their 'struggle for life' can occur.
We have only a general theory for this, since life is an ad hoc adaptive/evolutionary phenomenon. Far more broadly than just the behavioral or social sciences, our investigative methods are based on 'internal' comparisons (e.g., cases vs controls, various levels of blood pressure and stroke, fitness relative to different trait values) to evaluate samples against each other, rather than as representations of an externally derived, a priori theory. When we rely on statistics and p-value significance tests and probabilities and so on, we are implicitly confessing that we don't in fact really know what's going on, and all we can get are a kind of shadow of the underlying process that is cast by the differences we detect, and we detect them with generic (not to mention subjective) rather than specific criteria. We've written about these things several times in the past here.
The issue is not just weakly defined terms and study designs. As Freeman Dyson (in "A meeting with Enrico Fermi") wrote in 2004:
In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, "How many arbitrary parameters did you use for your calculations?" I thought for a moment about our cut-off procedures and said, "Four." He said, "I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk." With that, the conversation was over. . . .Here, parameters refers to what are more properly called 'free parameters', that is, ones not fixed in advance, but that are estimated from data. By contrast, for example, in physics the speed of light and gravitational constant are known, fixed values a priori, not estimated from data (though data were used to establish those values). We just lack such understanding in many areas of science, not just behavioral sciences.
In a sense we are using Ptolemaic tinkering to fit a theory that doesn't really fit, in the absence of a better (e.g., Copernican or Newtonian) theoretical understanding. Social and behavioral sciences are far behind the at least considerably more rigorous genetic and evolutionary sciences when the latter are done at their best (which isn't always). Like the shadows of reality seen in Plato's cave, statistical inference reflects the shadows of the reality we want to understand, but for many cultural and practical reasons we don't recognize that, or don't want to acknowledge it. The weaknesses and frangibility of our predictive 'powers' could, if properly understood by the general public, be a threat to our business and our culture doesn't reward candor when it comes to that business. The pressures (including from the pubic media, with their own agenda and interests) necessarily lead to reducing complexity to simpler models and claims far beyond what has legitimately been understood.
The problem is not just with weakly measured variables or poorly defined terms of, for example, outcomes. Nor is the problem if, when, or that people use methods wrongly. The problem is that statistical inference is based on a sample and is often retrospective, or mainly empirical and based on only rather generic theory. No matter how well chosen and rigorously defined, in these various areas (unlike much of physics and chemistry) the estimates of parameters and the like fitted to data that is necessarily about the subjects past, such as their culture or upbringing or lifestyles, but in the absence of adequate formal theory, these findings cannot be used to predict the future with knowable accuracy. That is because the same conditions can't be repeated, say, decades from now, and we don't know what future conditions will be, and so on.
Rather alarmingly, we were recently discussing this with a colleague who works in very physics- and chemistry-rigorous material science. She immediately told us that they, too, face problems in data evaluation with the number of variables they have to deal with, even under what the rest of us would enviously say were very well-controlled conditions where the assumptions of statistics--basically amounting to replicability of some underlying mathematical process--should really apply well.
So the social and related sciences may be far weaker than other fields, and should acknowledge that. But the rest of us, in various purportedly 'harder' biological, biomedical, and epidemiological sciences, are often not so much better off. Statistical methods and theory work wonderfully well when their assumptions are closely met. But there is too much out-of-the-box analytic toolware, that lures us into thinking that quick and definitive answers are possible. Those methods never promise that because what statistics does is account for repeated phenomena following the same rules, and the rule of many sciences is that, in their essence they are not following such rules.
But the lure of easy-answer statistics, and the understandable lack of deeply better ideas, perpetuates the expensive and misleading games that we are playing in many areas of science.
Monday, October 6, 2014
Big Data and its rationale - why are we still at it?
We're in the era of Big Data projects. This is the result of a fervor of belief that more data, essentially comprehensive and completely enumerative, will lead to deeper or even complete understanding. It is an extension of the idea of reductionism and induction that was a major part of the Age of Science, ushered in about 400 years ago with the likes of Bacon, Newton, Galileo and other iconic figures. Examples in physics include huge collider studies and very costly space activities, and of course the Big Data drive is hugely prevalent in genomics and other biological and biomedical sciences. But in our age, several centuries later, why are we still at it in this way?
The story isn't completely simple. The Age of Science also led to the so-called 'scientific method', a systematic way to increase knowledge through highly focused hypothesis-testing. Many philosophers of science have argued that, or tried to show how, the approach enabled our understanding of truth refined in this self-disciplined way, even if ultimate truth may always elude our meagre brainpower. But why then a return to raw induction?
One reason, and we think a predominant reason, is the pragmatic competition for research resources. As technological abilities rise (pushed by corporate interests for their own reasons), we have become able to collect ever more detailed data. The Age of Science was itself ushered in by technology in many ways, the iconic examples being optics (telescopes and microscopes). But sociopolitical reasons also exist. Long, large projects lock up large amounts of funds for years or even decades, guaranteeing jobs and status for people who thus can avoid the draining, relentless pursuit of multiple, small 3- or even 5-year grants. As the science establishment has grown, driven by universities for good as well as greedy reasons, funding inevitably became more competitive.
Careerism and enumerative ways to judge careers by administrators (paper and grant counting) are driving this system, but the funding agencies, too, have become populated with officials whose careers involve holding and building portfolios, using public relations to tout their achievements, and so on. And once you've got to the top of the Big Data pile, it's a high that's hard to come down from!
The history of a worldview
From the Manhattan Project in WWII, and several open-ended research efforts that followed, the idea became obvious that if you can state some generic problem and get funding to study it, and justify why it requires large scale, expensive technology, and long term, well, you snared yourself a career's worth of secure funding and all the status and perks--and, of course, actual research--that go with it. It's only human to understand those reasons.
However, there are also some good, scientifically legitimate reasons for the growth of Big Data. When I have queried investigators in the past--and here, this means over about 20 years--about why they were advocating genome mapping approaches to diseases of concern to them, they often said, rather desperately, that they were taking a 'hypothesis-free' approach because nothing else worked! If biology is genetics, genes must be involved, and mapping may show us what genes or systems are involved. For example, psychiatrists said that they simply had no idea what was going on in the brain to cause traits like schizophrenia, no candidate genes or physiological processes, so they were taking a mapping approach to get at least some clues they could then follow with focused research.
But to a great extent, what started out as a plausible justification for arch induction approaches, has become an excuse and a habit, a convenience or strategy rather than a legitimate rationale. The reason is not because their reasoning in the past was wrong at the time. The reason is because the Big Data approach has in a sense worked successfully: it has by and large proven not to provide the kind of results that initially justified it. Instead of identifying causes that couldn't have been expected, exhaustive Big-Data studies, in genomics and other areas of epidemiology and biomedical science, have identified countless minor or even trivial 'causes' of traits (and the psychiatric traits are good examples), showing they are not well explained by enumerative approaches, genetic or otherwise. For example, if hundreds of genes, each varying in and among populations, contribute to a trait, every occurrence of a disease, or everyone's blood pressure etc. is due to a different combination of causes. Big Data epidemiology has found the same for environmental and life-style factors.
What we should now do is to realize this successful set of findings from mapping studies. Rather than flood the media with hyperbole about the supposed successes of current approaches, we should adapt our approach to what we've learned, and take a time-out somehow, to reflect on what other conceptual approaches might actually work, given what we now know rather clearly. We may even have to substantially reform the types of questions we ask.
The reason for that sort of new approach is that once we, or as we, plunge into ever more too-big-to-terminate studies, with their likely minimal cost-benefit payoff, we lock up resources that clever thinkers might find better ways to use. And unless we do something of that sort, the message to scientists is to be more strongly driven to think in Big Data terms---because they'll know that's where the money is and how to keep their hold on it. This is exactly what's happening today.
Unfortunately, even many fundamental things in physics, the archetype of rigorous science, are being questioned by recent findings. Life sciences are in many relevant ways a century behind even that level. But this seems not to give us, or our funders, pause. Changing gears seems to go against the grain of how our industrialized society works.
In times of somewhat crimped resources from traditional funders, it's no wonder that universities and investigators are frantically turning to any possible source they can find. As we know from our own experience and that of colleagues, so much time is spent hustling and so relatively little in doing actual research, that the latter is becoming a side-light of the job. But it doesn't really seem to be changing how people think, and the push for Big Data is an understandable part of the strategy. The way to think about science itself is not changing under this pressure. At least not yet.
We keep harping on this message because it involves both the nature of knowledge and the societal aspects of how we acquire it. Even if there were no material interests in terms of allocation of resources, to cling to Big Data, we face a scientific or even epistemological problem that few seem interested in facing. There is simply too little pressure to force people to think differently.
Perhaps, if the message is said enough times, and read by enough people, sooner or later, somewhere, someone might get it, and show the rest of us a better way to fathom the causal complexities of the world.
The story isn't completely simple. The Age of Science also led to the so-called 'scientific method', a systematic way to increase knowledge through highly focused hypothesis-testing. Many philosophers of science have argued that, or tried to show how, the approach enabled our understanding of truth refined in this self-disciplined way, even if ultimate truth may always elude our meagre brainpower. But why then a return to raw induction?
One reason, and we think a predominant reason, is the pragmatic competition for research resources. As technological abilities rise (pushed by corporate interests for their own reasons), we have become able to collect ever more detailed data. The Age of Science was itself ushered in by technology in many ways, the iconic examples being optics (telescopes and microscopes). But sociopolitical reasons also exist. Long, large projects lock up large amounts of funds for years or even decades, guaranteeing jobs and status for people who thus can avoid the draining, relentless pursuit of multiple, small 3- or even 5-year grants. As the science establishment has grown, driven by universities for good as well as greedy reasons, funding inevitably became more competitive.
Careerism and enumerative ways to judge careers by administrators (paper and grant counting) are driving this system, but the funding agencies, too, have become populated with officials whose careers involve holding and building portfolios, using public relations to tout their achievements, and so on. And once you've got to the top of the Big Data pile, it's a high that's hard to come down from!
 |
| "It takes Big Data to make it Big--But I did it! (Drawn by the author) |
From the Manhattan Project in WWII, and several open-ended research efforts that followed, the idea became obvious that if you can state some generic problem and get funding to study it, and justify why it requires large scale, expensive technology, and long term, well, you snared yourself a career's worth of secure funding and all the status and perks--and, of course, actual research--that go with it. It's only human to understand those reasons.
However, there are also some good, scientifically legitimate reasons for the growth of Big Data. When I have queried investigators in the past--and here, this means over about 20 years--about why they were advocating genome mapping approaches to diseases of concern to them, they often said, rather desperately, that they were taking a 'hypothesis-free' approach because nothing else worked! If biology is genetics, genes must be involved, and mapping may show us what genes or systems are involved. For example, psychiatrists said that they simply had no idea what was going on in the brain to cause traits like schizophrenia, no candidate genes or physiological processes, so they were taking a mapping approach to get at least some clues they could then follow with focused research.
But to a great extent, what started out as a plausible justification for arch induction approaches, has become an excuse and a habit, a convenience or strategy rather than a legitimate rationale. The reason is not because their reasoning in the past was wrong at the time. The reason is because the Big Data approach has in a sense worked successfully: it has by and large proven not to provide the kind of results that initially justified it. Instead of identifying causes that couldn't have been expected, exhaustive Big-Data studies, in genomics and other areas of epidemiology and biomedical science, have identified countless minor or even trivial 'causes' of traits (and the psychiatric traits are good examples), showing they are not well explained by enumerative approaches, genetic or otherwise. For example, if hundreds of genes, each varying in and among populations, contribute to a trait, every occurrence of a disease, or everyone's blood pressure etc. is due to a different combination of causes. Big Data epidemiology has found the same for environmental and life-style factors.
What we should now do is to realize this successful set of findings from mapping studies. Rather than flood the media with hyperbole about the supposed successes of current approaches, we should adapt our approach to what we've learned, and take a time-out somehow, to reflect on what other conceptual approaches might actually work, given what we now know rather clearly. We may even have to substantially reform the types of questions we ask.
The reason for that sort of new approach is that once we, or as we, plunge into ever more too-big-to-terminate studies, with their likely minimal cost-benefit payoff, we lock up resources that clever thinkers might find better ways to use. And unless we do something of that sort, the message to scientists is to be more strongly driven to think in Big Data terms---because they'll know that's where the money is and how to keep their hold on it. This is exactly what's happening today.
Unfortunately, even many fundamental things in physics, the archetype of rigorous science, are being questioned by recent findings. Life sciences are in many relevant ways a century behind even that level. But this seems not to give us, or our funders, pause. Changing gears seems to go against the grain of how our industrialized society works.
In times of somewhat crimped resources from traditional funders, it's no wonder that universities and investigators are frantically turning to any possible source they can find. As we know from our own experience and that of colleagues, so much time is spent hustling and so relatively little in doing actual research, that the latter is becoming a side-light of the job. But it doesn't really seem to be changing how people think, and the push for Big Data is an understandable part of the strategy. The way to think about science itself is not changing under this pressure. At least not yet.
We keep harping on this message because it involves both the nature of knowledge and the societal aspects of how we acquire it. Even if there were no material interests in terms of allocation of resources, to cling to Big Data, we face a scientific or even epistemological problem that few seem interested in facing. There is simply too little pressure to force people to think differently.
Perhaps, if the message is said enough times, and read by enough people, sooner or later, somewhere, someone might get it, and show the rest of us a better way to fathom the causal complexities of the world.
Friday, September 5, 2014
When do you believe research?
At the end of the mini-course we taught in Helsinki, after a week of discussion of many essentially philosophy-of-science issues including how to make decisions about cause and effect, or how to determine whether a trait is 'genetic', or if it can be predicted from genes, a student asked how we decide which studies to believe. That is, responding to our questioning nature, he wanted to know how we decide which research reports to be skeptical about and which to believe. I've been thinking a lot about that. I don't really have answers because it's a fundamental question, but here are a few thoughts.
The class, called Logical Reasoning in Human Genetics, is meant to get students thinking about how they know what they think they know. Ken gave a lecture on the first day in which he talked about epistemological issues, including the scientific method. We're all taught from childhood that knowledge advances by the scientific method. There are multiple definitions, but let's just go with what's cited in the Wikipedia entry on the subject, in turn taken from the Oxford English Dictionary. It's "a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." But most definitions go further, to say one adjusts the hypothesis until there is no discrepancy between it and the latest results. This is how many web pages and books portray the process.
But this is awfully vague and not terribly helpful (if it's even true: for example, when is there no discrepancy between hypothesis and actual data?). Who decides what is systematic observation, how and what we measure, how we conduct experiments, and formulate, test and modify hypotheses? And even if we do agree on all this, it wouldn't give any hint as to which results should be believed. Any that follow the method? Any that lend evidence to our hypotheses? There was plenty of evidence for the sun revolving around the Earth, and spontaneous generation, and the miasma theory of disease, all based on systematic observation and hypotheses, after all. Clearly, empiricism isn't enough.
In his first lecture, Ken showed this slide:
The essential tenets of the scientific method. Most of us would include at least some of these criteria in a list of essentials, right? Ken discussed them all, and then showed why each of them in turn may be useful but cannot in fact be a solid basis for inferring causation. One may hypothesize that all swans are white, and it may seem to stand up to observation -- but observing a single black swan does in that theory. Figuratively speaking, when can we ever be sure that we'll never see a non-white swan? So induction is not a perfectly reliable criterion for forming general inferences. Prediction is an oft-cited criterion for scientific validity, but in areas of biology depends on knowing future environments, which is impossible in principle. Scientists claim that theories may never be provable but can always be falsified, which leads to better theory. But scientists rarely, if ever, actually work to falsify their own theories. And one can falsify an idea by a bad experiment even if the idea is correct. P-values for statistical significance are subjective choices: P = 0.05 was not decreed by God. And so on.
So, then Ken added the following criterion:
This is probably a better description of how scientists actually do science. And I'm writing this in Austria, so I'll mention that if you've read Austrian philosopher of science, Paul Feyerabend's "Against Method", this will sound familiar. Feyerabend believed that strict adherence to the scientific method would inhibit progress, and that a bit of anarchy is essential to good science. Further, the usual criteria, e.g. consistency and falsification, are antithetical to progress. Indeed, as a philosopher who took a hard long look at the history of scientific advances, Feyerabend concluded that the best description of good science is "anything goes," a phrase for which he is famous, and often condemned. But he didn't mean it as a principle, rather it was a description of how science is actually done. It is a social and even political process.
However, even an anarchic bent doesn't help us decide which results to believe, even if it does mean that we shouldn't consider that sticklers for method have an advantage.
How do we decide?
A few weeks ago we wrote about a paper that claimed that tick bites are causing an epidemic of red meat allergies in the US and Europe. Curious. Curious enough to lead me to read 3 or 4 papers on the subject, all of which suggested a pattern of exposure and symptoms consistent with the habitat of the tick, as well as a mechanism that explained how the tick bite could cause this often severe allergy. Seemed convincing to us.
But, someone on Twitter wasn't convinced:
Or what about this piece, posted last week by the New York Times, on the effects of bisphenol on ovarian health? Evidence seems to be mounting, but even people in the field are cautioning that it's hard to tell cause and effect. Or, what about the causes of asthma? Environmental epidemiology has found that breast feeding is a cause, but also bottle feeding, excessive hygiene, or pollution. Same methods -- "systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses" -- contradictory results.
Or, what about climate change? How do we decide what to believe? Few of us are expert enough in meteorology, geology, or climate history to make a decision based on the data, so essentially we must decide based on whether we believe -- yes, believe -- that the science is being rigorously conducted. But, how would we know? Do we count the number of peer-reviewed papers reporting that the climate is changing? If so, that's just a belief that peer-review adds weight to findings, rather than is simply evidence of a current fad in thinking about climate, or circling of wagons, or some other sociological quirk of science. Do we count the number of papers or op/ed pieces written by US National Academy members, or Nobel prize winners? In which case, we're even further from actual scientific evidence.
We can list one criterion that, today, must be true. The results must be evolutionarily sound. Evolution is probably as close as biology comes to 'theory'; descent with modification from a common ancestor. If results don't fit within that theory, they are probably wrong. But not definitively -- we should always be testing theory.
Here's another one, that must be true when considering causation -- the cause must precede the effect. (This is one in a list of nine criteria sometimes relied upon in epidemiology, the rest of which aren't necessarily true, recognized even by Bradford Hill who devised the list.) But this isn't terribly helpful. Many things can precede an effect, not just one, and many things that precede the event are unrelated to it. Which such preceeder do we accept?
Several criteria that might help are replication and consistency, but for many reasons they can't be considered sufficient or necessary. They might confirm what we think we know -- but consistent and replicated findings of disease due to bad air prior to the germ theory of disease confirmed miasma as a cause. Life is about diversity, and that is how it evolves, so replication is not a necessary criterion for something about, say, genetic causation, to be true under some circumstances but not all.
Science is done by scientists in (and these days supported by) society. We need jobs and we try to seek truth. But one proverbial truth is that science should always be based on doubt and skepticism: rarely do we know everything perfectly. Once we stop questioning -- and the hardest person to question is oneself -- then we become dogmatists, and our science is not that different from received truth in religion.
Scientists may rarely think seriously or critically about their criteria for truth. We believe that there is truth, but it's elusive much of the time, especially in complex areas like evolutionary biology, genetics, and biomedical causation. A major frustration is that we have no formal criteria for inference that always work. Inference is a kind of collective, social decision process, based on faith, yes, faith in whatever a given scientist believes or is pressured by his/her peers to believe. The history of science shows that this year's 'facts' are next year's discards. So which study do we believe when there are important implications for that decision? If it's not true that you can "use whatever criteria you want", for various pragmatic reasons, then what is true about scientific inference in these areas of knowledge?
The class, called Logical Reasoning in Human Genetics, is meant to get students thinking about how they know what they think they know. Ken gave a lecture on the first day in which he talked about epistemological issues, including the scientific method. We're all taught from childhood that knowledge advances by the scientific method. There are multiple definitions, but let's just go with what's cited in the Wikipedia entry on the subject, in turn taken from the Oxford English Dictionary. It's "a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." But most definitions go further, to say one adjusts the hypothesis until there is no discrepancy between it and the latest results. This is how many web pages and books portray the process.
But this is awfully vague and not terribly helpful (if it's even true: for example, when is there no discrepancy between hypothesis and actual data?). Who decides what is systematic observation, how and what we measure, how we conduct experiments, and formulate, test and modify hypotheses? And even if we do agree on all this, it wouldn't give any hint as to which results should be believed. Any that follow the method? Any that lend evidence to our hypotheses? There was plenty of evidence for the sun revolving around the Earth, and spontaneous generation, and the miasma theory of disease, all based on systematic observation and hypotheses, after all. Clearly, empiricism isn't enough.
In his first lecture, Ken showed this slide:
So, then Ken added the following criterion:
This is probably a better description of how scientists actually do science. And I'm writing this in Austria, so I'll mention that if you've read Austrian philosopher of science, Paul Feyerabend's "Against Method", this will sound familiar. Feyerabend believed that strict adherence to the scientific method would inhibit progress, and that a bit of anarchy is essential to good science. Further, the usual criteria, e.g. consistency and falsification, are antithetical to progress. Indeed, as a philosopher who took a hard long look at the history of scientific advances, Feyerabend concluded that the best description of good science is "anything goes," a phrase for which he is famous, and often condemned. But he didn't mean it as a principle, rather it was a description of how science is actually done. It is a social and even political process.
However, even an anarchic bent doesn't help us decide which results to believe, even if it does mean that we shouldn't consider that sticklers for method have an advantage.
How do we decide?
A few weeks ago we wrote about a paper that claimed that tick bites are causing an epidemic of red meat allergies in the US and Europe. Curious. Curious enough to lead me to read 3 or 4 papers on the subject, all of which suggested a pattern of exposure and symptoms consistent with the habitat of the tick, as well as a mechanism that explained how the tick bite could cause this often severe allergy. Seemed convincing to us.
But, someone on Twitter wasn't convinced:
Really appreciate your blogs but, as this paper outlines, one should be skeptical when it comes to ticks. http://t.co/rCmp2H6b4pThe link is to a Lancet article, but it restricts its discussion to the anti-science claims of those who believe that Lyme disease is not what 'evidence-based' medicine says it is.
Similar to other antiscience groups, these advocates have created a pseudoscientific and alternative selection of practitioners, research, and publications and have coordinated public protests, accused opponents of both corruption and conspiracy, and spurred legislative efforts to subvert evidence-based medicine and peer-reviewed science. The relations and actions of some activists, medical practitioners, and commercial bodies involved in Lyme disease advocacy pose a threat to public health.But should we skeptical about all tick-borne diseases? The CDC still lists a number of them. I don't know enough about this subject to comment further, but it's interesting indeed that antiscience claims can themselves be couched in a semblance of the scientific method. Or at least a parallel track, with its own 'experts', publications, peer reviewers, and so on. In fact this makes the question of how one decides what to believe almost mystical, or dare we say religious. Surprisingly, while it is often said that science, unlike other areas of human affairs, isn't decided by a vote, in reality group consensus about what is true is a kind of vote among competing scientists; the majority or those in most prominent positions, do tend to set established practice and criteria.
Or what about this piece, posted last week by the New York Times, on the effects of bisphenol on ovarian health? Evidence seems to be mounting, but even people in the field are cautioning that it's hard to tell cause and effect. Or, what about the causes of asthma? Environmental epidemiology has found that breast feeding is a cause, but also bottle feeding, excessive hygiene, or pollution. Same methods -- "systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses" -- contradictory results.
Or, what about climate change? How do we decide what to believe? Few of us are expert enough in meteorology, geology, or climate history to make a decision based on the data, so essentially we must decide based on whether we believe -- yes, believe -- that the science is being rigorously conducted. But, how would we know? Do we count the number of peer-reviewed papers reporting that the climate is changing? If so, that's just a belief that peer-review adds weight to findings, rather than is simply evidence of a current fad in thinking about climate, or circling of wagons, or some other sociological quirk of science. Do we count the number of papers or op/ed pieces written by US National Academy members, or Nobel prize winners? In which case, we're even further from actual scientific evidence.
We can list one criterion that, today, must be true. The results must be evolutionarily sound. Evolution is probably as close as biology comes to 'theory'; descent with modification from a common ancestor. If results don't fit within that theory, they are probably wrong. But not definitively -- we should always be testing theory.
Here's another one, that must be true when considering causation -- the cause must precede the effect. (This is one in a list of nine criteria sometimes relied upon in epidemiology, the rest of which aren't necessarily true, recognized even by Bradford Hill who devised the list.) But this isn't terribly helpful. Many things can precede an effect, not just one, and many things that precede the event are unrelated to it. Which such preceeder do we accept?
Several criteria that might help are replication and consistency, but for many reasons they can't be considered sufficient or necessary. They might confirm what we think we know -- but consistent and replicated findings of disease due to bad air prior to the germ theory of disease confirmed miasma as a cause. Life is about diversity, and that is how it evolves, so replication is not a necessary criterion for something about, say, genetic causation, to be true under some circumstances but not all.
Science is done by scientists in (and these days supported by) society. We need jobs and we try to seek truth. But one proverbial truth is that science should always be based on doubt and skepticism: rarely do we know everything perfectly. Once we stop questioning -- and the hardest person to question is oneself -- then we become dogmatists, and our science is not that different from received truth in religion.
Scientists may rarely think seriously or critically about their criteria for truth. We believe that there is truth, but it's elusive much of the time, especially in complex areas like evolutionary biology, genetics, and biomedical causation. A major frustration is that we have no formal criteria for inference that always work. Inference is a kind of collective, social decision process, based on faith, yes, faith in whatever a given scientist believes or is pressured by his/her peers to believe. The history of science shows that this year's 'facts' are next year's discards. So which study do we believe when there are important implications for that decision? If it's not true that you can "use whatever criteria you want", for various pragmatic reasons, then what is true about scientific inference in these areas of knowledge?
Tuesday, March 4, 2014
Lucretius, and stories about the Nature of Things
I never took courses in Classics, so my knowledge of the major figures and their works is only sometimes from my own informal reading, and mainly from secondary summaries or hearsay.
But, after having read of and about it for many years, I decided I should actually read Lucretius' (99-55BC) long poem entitled De Rerum Natura (On the Nature of Things). So many times, I've heard that Lucretius anticipated modern science in sometimes rather eerily prescient ways, and I felt I should see for myself. I can't read Latin, so I have read an English prose translation in English, hoping at least to see the content, even if unable to appreciate the artistry.
Lucretius was a Roman, nominally writing to persuade a friend of his, but he was basically expounding the views of the Greek Epicurean school of philosophy, led by Epicurius and Democritus and others who lived a couple of hundred years earlier. Among other things, the Epicureans argued persuasively that there were no gods and that human existence is matter-to-matter, dust-to-dust: enjoy life while you have it because that's all there is. You will decay and your material will return to the cosmos from whence it came. Nothingness cannot generate material, and material is eternal. Your mind (or spirit, or whatever term you use) is part of your body and disappears with your body, into the cosmos, to be re-used in the future, as it had come to form you when you were conceived.
There exists matter, they said, made of atoms ('a-tom', or cannot cut), which are the smallest fundamental particles, and 'void' (space). Atoms move around randomly through space and collide when they meet, at which time they can stick and form larger constructs, which today we call 'molecules'. Aggregates of these compounds make up the stuff of the world. It can separate and rejoin, but matter is neither created out of nothing nor destroyed, nor is space. To allow for our apparent free will, the Epicureans said that atoms could sometimes 'swerve' from their normally determined paths.
The atomic theory of the Epicureans sounds quite like modern molecular and atomic physics and cosmology (though it is true that modern physics does seem to allow strange things like the creation of matter and energy from cosmic vacuum, and perhaps multiple universes, and so on). Thus, the ideas Lucretius described can seem presciently scientific, two thousand years before their time. I have read such characterizations frequently. But there are some interesting points here, that have to do with how ideas are anchored in reality, and with selective reporting.
For one thing, if you read the rest of Lucretius, you'll find stories of the origins of the things in an earth-centered universe, including anthropological tales explaining the origin of humans and their cultural evolution--how we started out crude and beast-like, then discovered weapons, clothing, governments, language and song, the discovery of agriculture, domestication of animals. He also used his theory to explain the nature of lightning, earthquakes, volcanoes, weather, geology, gravity,why the Nile floods and the nature of magnetism. He explained the working of our senses like vision, touch and taste, in atomic terms--accounting, for example, for the emanations from the atoms on the surface of our bodies, that enable us to see 'ourselves' in mirrors. He raised developmental arguments to show that chimeric beasts, like Centaurs, cannot be real. He delves into racial variation and why different populations are subject to different diseases. And he goes into the clinical nature and epidemiology of plagues.
A main aim of Lucretius was to purge people of superstition. He fervently wanted to dismantle anything other than a pure materialism, even in explaining the origin of moral aspects of society. In this sense, too, he is favorably cited for his 'prescient' materialistic atomic theory of everything.
In the major sections of De Rerum, however, the apparent prescience becomes less and less, and any idea that he foreshadowed modern science dissolves. Basically, the Epicureans were applying their notion of common-sense reasoning based on very general observations. They strung out one categorical assertion after another of what 'must' be the case. In today's parlance, they were providing hand-waving 'explanations' ('accounts' would be a better term) that seemed consistent but did not require any sort of rigorous means of establishing truth.
Along comes the Enlightenment
Aristotle, Plato, and others of the Greek philosophers reinforced the idea that reasoning itself was enough to generate understanding of the world. We are basically built, they said, to see and understand truth. Such a view of knowledge lasted until about 400 years ago, the period called the Enlightenment (in Europe), the time of Francis Bacon, Descartes, Galileo, Newton, and many others. Those authors asserted that, to the contrary, to understand Nature one had to make systematic observations, and develop proper, formal, systematic reasoning to fit hypotheses to those observations, to develop general theory or laws of the world. Out of this was born the 'scientific method' and the idea that truth was to be understood by empiricism and actual testing of ideas, not just story-telling--and no mysticism.
Reading Lucretius makes one realize first, that even if a story like the Epicureans' atomic theory has aspects we'd regard today as truth, it was to them basically a sort of guessing. Secondly, just because a story is plausible does not give it a necessary connection to truth, no matter how consistent the story may seem. We now do have actual 'scientific' theories to account for--or, now, actually, explain--phenomena such as earthquakes, weather, volcanoes, the nature of metals and water, the diversity of life, a great deal of biology, and even culture history. If you think of how we know these things, even if there are major gaps in that knowledge, you can see how very powerful and correct (or at least much more accurate) a systematic approach to knowledge can be, when the subject is amenable to such study.
It is a great credit to centuries of insightful, diligent scientists, our forebears, whose legacy has brought us to this point. It is a wonderful gift from them to our own time.
Advances in technology and methods may be making some Enlightenment concepts obsolete, and we continually find new ways of knowing that go ever farther beyond our own personal biological senses. For those of us in science, reading the likes of Lucretius is an occasion to compare then and now, to see why just being intelligent and able to construct consistent explanations is not enough, and that for many areas we do now have ways to gain knowledge that has a firmer footing in reality--not just plausibility.
But....
That's all to the good, but if you do a more measured reading of Lucretius, you can see that in many ways we haven't come all that far. We do a lot of cherry-picking of things in Lucretius that sound similar to today's ideas and thus seem particularly insightful. But it is not clear that they were more than a mix of subjective insight and, mainly, good guesses--after all, there were competing theories of the nature of Nature even at the time. And other areas of Epicurean thought, well, are just not mentioned by those remarking on their apparent modernity. Selective citation gives an impression of deep insight. Most of De Rerum Natura was simply story-telling.
In many areas of science, perhaps even some aspects of fundamental physics and cosmology, but particularly in the social and even aspects of evolutionary sciences, we still make careers based on plausibility story-telling. Our use of mathematics or statistical methods--random surveys, questionnaires, arguing by analogy, and so on--and massive data collection, give the same sort of patina of professional wisdom that one can see in the rhetoric of Lucretius.
We tell our stories with confidence, assertions of what 'must' be so, or what is 'obvious'. Often, those interested in behavior and psychology are committed to purging religious mysticism by showing that behavior may seem immaterial but that this is an illusion, and purely material evolutionary and genetic explanations are offered. No 'free will'! The world is only a physical reality. The role of natural selection and competition in explaining even morality as a material phenomenon is part of this, because Darwin provided a global (may one say 'Epicurean'?) material framework for it. Evolutionary stories are routinely reported to the public in that way as well. Even if some caveats or doubts are included here and there, they are often buried by the headlines--and the same can be found in Lucretius, over two thousand years ago.
Explanations of physical and behavioral variation and its evolutionary causes, along with many 'adaptive' stories making forcefully asserted plausibility arguments about what evolved 'for' what, still abound. They are not just told on television--we can't really blame Disney and Discover for appealing to their audiences, because they are businesses; but the same stories are in the science journals and science reportage as well. We see tales every day reporting miraculous discoveries about genetic causation, for example. It is sobering to see that, in areas where we don't have a really effective methodology or theoretical basis, we are in roughly similar sandals as our ancient predecessors.
Intoxicated by the many genuinely dramatic discoveries of modern, systematic science, we do our own cherry-picking, and tend to suspend judgment where findings are less secure, dressing our explanations in sophisticated technological hand-waving.
When we don't have actual good explanations, we make up good-sounding stories, just as our forebears did, and they're often widely accepted today--just as they were then.
But, after having read of and about it for many years, I decided I should actually read Lucretius' (99-55BC) long poem entitled De Rerum Natura (On the Nature of Things). So many times, I've heard that Lucretius anticipated modern science in sometimes rather eerily prescient ways, and I felt I should see for myself. I can't read Latin, so I have read an English prose translation in English, hoping at least to see the content, even if unable to appreciate the artistry.
 |
| The deep truths? Source: crusader.bac.edu |
 |
| Lucretius |
There exists matter, they said, made of atoms ('a-tom', or cannot cut), which are the smallest fundamental particles, and 'void' (space). Atoms move around randomly through space and collide when they meet, at which time they can stick and form larger constructs, which today we call 'molecules'. Aggregates of these compounds make up the stuff of the world. It can separate and rejoin, but matter is neither created out of nothing nor destroyed, nor is space. To allow for our apparent free will, the Epicureans said that atoms could sometimes 'swerve' from their normally determined paths.
The atomic theory of the Epicureans sounds quite like modern molecular and atomic physics and cosmology (though it is true that modern physics does seem to allow strange things like the creation of matter and energy from cosmic vacuum, and perhaps multiple universes, and so on). Thus, the ideas Lucretius described can seem presciently scientific, two thousand years before their time. I have read such characterizations frequently. But there are some interesting points here, that have to do with how ideas are anchored in reality, and with selective reporting.
For one thing, if you read the rest of Lucretius, you'll find stories of the origins of the things in an earth-centered universe, including anthropological tales explaining the origin of humans and their cultural evolution--how we started out crude and beast-like, then discovered weapons, clothing, governments, language and song, the discovery of agriculture, domestication of animals. He also used his theory to explain the nature of lightning, earthquakes, volcanoes, weather, geology, gravity,why the Nile floods and the nature of magnetism. He explained the working of our senses like vision, touch and taste, in atomic terms--accounting, for example, for the emanations from the atoms on the surface of our bodies, that enable us to see 'ourselves' in mirrors. He raised developmental arguments to show that chimeric beasts, like Centaurs, cannot be real. He delves into racial variation and why different populations are subject to different diseases. And he goes into the clinical nature and epidemiology of plagues.
A main aim of Lucretius was to purge people of superstition. He fervently wanted to dismantle anything other than a pure materialism, even in explaining the origin of moral aspects of society. In this sense, too, he is favorably cited for his 'prescient' materialistic atomic theory of everything.
In the major sections of De Rerum, however, the apparent prescience becomes less and less, and any idea that he foreshadowed modern science dissolves. Basically, the Epicureans were applying their notion of common-sense reasoning based on very general observations. They strung out one categorical assertion after another of what 'must' be the case. In today's parlance, they were providing hand-waving 'explanations' ('accounts' would be a better term) that seemed consistent but did not require any sort of rigorous means of establishing truth.
Along comes the Enlightenment
Aristotle, Plato, and others of the Greek philosophers reinforced the idea that reasoning itself was enough to generate understanding of the world. We are basically built, they said, to see and understand truth. Such a view of knowledge lasted until about 400 years ago, the period called the Enlightenment (in Europe), the time of Francis Bacon, Descartes, Galileo, Newton, and many others. Those authors asserted that, to the contrary, to understand Nature one had to make systematic observations, and develop proper, formal, systematic reasoning to fit hypotheses to those observations, to develop general theory or laws of the world. Out of this was born the 'scientific method' and the idea that truth was to be understood by empiricism and actual testing of ideas, not just story-telling--and no mysticism.
Reading Lucretius makes one realize first, that even if a story like the Epicureans' atomic theory has aspects we'd regard today as truth, it was to them basically a sort of guessing. Secondly, just because a story is plausible does not give it a necessary connection to truth, no matter how consistent the story may seem. We now do have actual 'scientific' theories to account for--or, now, actually, explain--phenomena such as earthquakes, weather, volcanoes, the nature of metals and water, the diversity of life, a great deal of biology, and even culture history. If you think of how we know these things, even if there are major gaps in that knowledge, you can see how very powerful and correct (or at least much more accurate) a systematic approach to knowledge can be, when the subject is amenable to such study.
It is a great credit to centuries of insightful, diligent scientists, our forebears, whose legacy has brought us to this point. It is a wonderful gift from them to our own time.
Advances in technology and methods may be making some Enlightenment concepts obsolete, and we continually find new ways of knowing that go ever farther beyond our own personal biological senses. For those of us in science, reading the likes of Lucretius is an occasion to compare then and now, to see why just being intelligent and able to construct consistent explanations is not enough, and that for many areas we do now have ways to gain knowledge that has a firmer footing in reality--not just plausibility.
But....
That's all to the good, but if you do a more measured reading of Lucretius, you can see that in many ways we haven't come all that far. We do a lot of cherry-picking of things in Lucretius that sound similar to today's ideas and thus seem particularly insightful. But it is not clear that they were more than a mix of subjective insight and, mainly, good guesses--after all, there were competing theories of the nature of Nature even at the time. And other areas of Epicurean thought, well, are just not mentioned by those remarking on their apparent modernity. Selective citation gives an impression of deep insight. Most of De Rerum Natura was simply story-telling.
In many areas of science, perhaps even some aspects of fundamental physics and cosmology, but particularly in the social and even aspects of evolutionary sciences, we still make careers based on plausibility story-telling. Our use of mathematics or statistical methods--random surveys, questionnaires, arguing by analogy, and so on--and massive data collection, give the same sort of patina of professional wisdom that one can see in the rhetoric of Lucretius.
We tell our stories with confidence, assertions of what 'must' be so, or what is 'obvious'. Often, those interested in behavior and psychology are committed to purging religious mysticism by showing that behavior may seem immaterial but that this is an illusion, and purely material evolutionary and genetic explanations are offered. No 'free will'! The world is only a physical reality. The role of natural selection and competition in explaining even morality as a material phenomenon is part of this, because Darwin provided a global (may one say 'Epicurean'?) material framework for it. Evolutionary stories are routinely reported to the public in that way as well. Even if some caveats or doubts are included here and there, they are often buried by the headlines--and the same can be found in Lucretius, over two thousand years ago.
Explanations of physical and behavioral variation and its evolutionary causes, along with many 'adaptive' stories making forcefully asserted plausibility arguments about what evolved 'for' what, still abound. They are not just told on television--we can't really blame Disney and Discover for appealing to their audiences, because they are businesses; but the same stories are in the science journals and science reportage as well. We see tales every day reporting miraculous discoveries about genetic causation, for example. It is sobering to see that, in areas where we don't have a really effective methodology or theoretical basis, we are in roughly similar sandals as our ancient predecessors.
 |
| Cherries; Wikipedia |
When we don't have actual good explanations, we make up good-sounding stories, just as our forebears did, and they're often widely accepted today--just as they were then.
Monday, February 10, 2014
Eat, drink, and be .... confused!
Is beer good for your health or is it a slow killer? What about, say, bread, broccoli, wine, eggs, or (dare we say it?), sex?
The answers are:
1. Yes! Anything we consume will keep us alive, but that just facilitates the path to our final end. Metabolism and conjugal energy expenditure generate waste products, heat, cell division, and so on. That leads to death!
Or 2. Nobody knows! After decades of huge, expensive studies by presumably the most knowledgeable investigators doing state-of-the-art science, we know relatively little with relatively little firmness, what we eat does for or to us.
In a Times article on Sunday, nutrition journalist Gary Taubes excoriates the nutrition research mill, and we think properly so, for decades of generating ever more numerous studies on nutritional epidemiology, without garnering many firm or important conclusions.
Why can't we figure out the relationship of nutrition to disease in so many cases? Largely because we've got a reductionist science that is able to find causes of disease that have large effects -- smoking, asbestos, the cause of infectious diseases -- but lousy at explaining diseases that are due to gradual exposure to multiple interacting factors that takes place over decades. And, when some people exposed to what looks like a risk factor -- obesity, say -- develop diabetes but others don't, or when lots of sugar in the diet seems to be associated with obesity in some people but not in others, our methods really fail us. We see the exact parallel in genetics, as we've written many times. Indeed, does DDT cause Alzheimer's disease, a result we blogged about just last week?
Also on Sunday, the BBC reported that vitamin C is an effective treatment for cancer. How many times for how many decades do we still have to hear more of this-finally-is true conjecture about vitamin C? It's been going on for many decades. The current story appeared in a journal called Science Translational Medicine. The very phrase' translational medicine' reflects our rather bourgeois industrialization of the research system with its business and status basis. It is a cachet self-congratulatory catch-phrase that suggests that biomedical research in the past had no interest in preventing or combating disease. It suggests that grants were not given by NIH for such purposes (of course, NIH does fund a lot research that's irrelevant to health), which is just plain silly. Or, rather, as we often have suggested, an establishment's typical way of making itself sound salubrious to the taxpayer we milk for our careers. So why is there even a Science Translational Medicine journal? Because Nature has one? Because there might be advertising or subscription gains to be made? Because each science publisher has to keep up with the perceived proverbial Joneses? Because the bloated professoriate clamors for ever-more status-sounding places to publish their work?
What we're doing in the biomedical and health research establishment is to a great extent ever more of the same kinds of studies only bigger and with more costly and sophisticated hard and software. Anyone with a computer can get SurveyMonkey software and design a questionnaire, and if you've got a degree in public health you know how to hire a bunch of nurse-interviewers, phone-callers, data-base miners and the like, and send them out on the streets to do various sorts of random samples, test and quality-check questionnaires in a standard way, then increase samples, have lots of meetings and data-enterers, and after a few years start using push-button statistical software to pour out papers (and contact the Times and BBC 'science' journalists to trumpet your work). And, every year or so, write new grants to follow up your important 'translational' research.
Is there anything new here?
The state of play is well known, and well known at least to the thoughtful contingent of researchers in the game. Taubes, who has been guilty of simplistic advocacy as even he acknowledges by confessing his personal preference for sugar as the one-size-fits-all evil, clearly identifies the nature of the problem. He doesn't really offer a solution.
We can't either, but we do say, that what is not being done is making better use of our wet ware: our brainpower. Taubes' story was on nutrition research, and we have been harping on the same issues with respect to genomics, and have also critiqued epidemiology (including nutritional epidemiology) in past posts.
There is no magic answer for ginning up real insight and creativity. But there may be ways of numbing or mesmerizing the part of society that might produce creativity. A huge factory-like establishment of drone workers who need the factory to keep spewing out 'product' may have just that effect. Our own idea that part of what is needed is to at least make the soil -- the research environment ecology -- more likely to engender innovation. That would mean to down-size, slow down, think more and return research to being more of a profession than an industry. There are too many professors, too pressured to grind out too many papers or hustle too many grants to keep the administrators and careerists happy. Too many administrators who, dependent on the cream, need to keep the factory humming. Time to think, or re-think, or be inspired creatively by odd facts is hard to come by when the pressure is get grants or lose your job.
Better synthetic rather than narrowly technical education is needed, but we haven't been generating the kinds of people who can teach it. Instead, we have trained a body of academic professors who have been brought up and entrained in, depend on, and hence perhaps can't see or can't afford to see what is actually happening. But despite at least some people pointing the problems out, there is nothing on the horizon that seems yet able to stimulate real change.
All of this is true. It is also true that the problems are difficult, many if not most faculty have a sincere drive to help public or individual health, high technology is at least somewhat effective and more than just expensive showy toys, and administrators are needed for big, expensive systems. The methods are often canned in ready-made software, but that doesn't make them wrong, even if we clearly are thinking wrongly in some way. Until we're shown better, while we do act like fad-following sheep, we do that because that's what we, collectively, know how to do. And until our jobs and self-esteem are not constructed around the impatient, short-term factory mentality, one cannot expect us to act very differently.
We do, after all, need to eat, drink, and try to be merry, for tomorrow we die whether or not we like to accept that.
 |
| Beer; Yebisu Beer Museum, Wikipedia |
The answers are:
1. Yes! Anything we consume will keep us alive, but that just facilitates the path to our final end. Metabolism and conjugal energy expenditure generate waste products, heat, cell division, and so on. That leads to death!
Or 2. Nobody knows! After decades of huge, expensive studies by presumably the most knowledgeable investigators doing state-of-the-art science, we know relatively little with relatively little firmness, what we eat does for or to us.
In a Times article on Sunday, nutrition journalist Gary Taubes excoriates the nutrition research mill, and we think properly so, for decades of generating ever more numerous studies on nutritional epidemiology, without garnering many firm or important conclusions.
The 600,000 articles — along with several tens of thousands of diet books — are the noise generated by a dysfunctional research establishment. Because the nutrition research community has failed to establish reliable, unambiguous knowledge about the environmental triggers of obesity and diabetes, it has opened the door to a diversity of opinions on the subject, of hypotheses about cause, cure and prevention, many of which cannot be refuted by the existing evidence. Everyone has a theory. The evidence doesn’t exist to say unequivocally who’s wrong.Even the basic questions about foods and eating are not being answered with much rigor, including even the common-wisdom recommendations related to obesity, dietary abuses, and so on, are often on quite shaky ground. It is a huge research establishment that has its hands on funding agencies and is not being held accountable for delivering actual goods commensurate with the public investment.
Why can't we figure out the relationship of nutrition to disease in so many cases? Largely because we've got a reductionist science that is able to find causes of disease that have large effects -- smoking, asbestos, the cause of infectious diseases -- but lousy at explaining diseases that are due to gradual exposure to multiple interacting factors that takes place over decades. And, when some people exposed to what looks like a risk factor -- obesity, say -- develop diabetes but others don't, or when lots of sugar in the diet seems to be associated with obesity in some people but not in others, our methods really fail us. We see the exact parallel in genetics, as we've written many times. Indeed, does DDT cause Alzheimer's disease, a result we blogged about just last week?
Also on Sunday, the BBC reported that vitamin C is an effective treatment for cancer. How many times for how many decades do we still have to hear more of this-finally-is true conjecture about vitamin C? It's been going on for many decades. The current story appeared in a journal called Science Translational Medicine. The very phrase' translational medicine' reflects our rather bourgeois industrialization of the research system with its business and status basis. It is a cachet self-congratulatory catch-phrase that suggests that biomedical research in the past had no interest in preventing or combating disease. It suggests that grants were not given by NIH for such purposes (of course, NIH does fund a lot research that's irrelevant to health), which is just plain silly. Or, rather, as we often have suggested, an establishment's typical way of making itself sound salubrious to the taxpayer we milk for our careers. So why is there even a Science Translational Medicine journal? Because Nature has one? Because there might be advertising or subscription gains to be made? Because each science publisher has to keep up with the perceived proverbial Joneses? Because the bloated professoriate clamors for ever-more status-sounding places to publish their work?
What we're doing in the biomedical and health research establishment is to a great extent ever more of the same kinds of studies only bigger and with more costly and sophisticated hard and software. Anyone with a computer can get SurveyMonkey software and design a questionnaire, and if you've got a degree in public health you know how to hire a bunch of nurse-interviewers, phone-callers, data-base miners and the like, and send them out on the streets to do various sorts of random samples, test and quality-check questionnaires in a standard way, then increase samples, have lots of meetings and data-enterers, and after a few years start using push-button statistical software to pour out papers (and contact the Times and BBC 'science' journalists to trumpet your work). And, every year or so, write new grants to follow up your important 'translational' research.
Is there anything new here?
The state of play is well known, and well known at least to the thoughtful contingent of researchers in the game. Taubes, who has been guilty of simplistic advocacy as even he acknowledges by confessing his personal preference for sugar as the one-size-fits-all evil, clearly identifies the nature of the problem. He doesn't really offer a solution.
We can't either, but we do say, that what is not being done is making better use of our wet ware: our brainpower. Taubes' story was on nutrition research, and we have been harping on the same issues with respect to genomics, and have also critiqued epidemiology (including nutritional epidemiology) in past posts.
There is no magic answer for ginning up real insight and creativity. But there may be ways of numbing or mesmerizing the part of society that might produce creativity. A huge factory-like establishment of drone workers who need the factory to keep spewing out 'product' may have just that effect. Our own idea that part of what is needed is to at least make the soil -- the research environment ecology -- more likely to engender innovation. That would mean to down-size, slow down, think more and return research to being more of a profession than an industry. There are too many professors, too pressured to grind out too many papers or hustle too many grants to keep the administrators and careerists happy. Too many administrators who, dependent on the cream, need to keep the factory humming. Time to think, or re-think, or be inspired creatively by odd facts is hard to come by when the pressure is get grants or lose your job.
Better synthetic rather than narrowly technical education is needed, but we haven't been generating the kinds of people who can teach it. Instead, we have trained a body of academic professors who have been brought up and entrained in, depend on, and hence perhaps can't see or can't afford to see what is actually happening. But despite at least some people pointing the problems out, there is nothing on the horizon that seems yet able to stimulate real change.
All of this is true. It is also true that the problems are difficult, many if not most faculty have a sincere drive to help public or individual health, high technology is at least somewhat effective and more than just expensive showy toys, and administrators are needed for big, expensive systems. The methods are often canned in ready-made software, but that doesn't make them wrong, even if we clearly are thinking wrongly in some way. Until we're shown better, while we do act like fad-following sheep, we do that because that's what we, collectively, know how to do. And until our jobs and self-esteem are not constructed around the impatient, short-term factory mentality, one cannot expect us to act very differently.
We do, after all, need to eat, drink, and try to be merry, for tomorrow we die whether or not we like to accept that.
Subscribe to:
Posts (Atom)