Showing posts with label replicability. Show all posts
Showing posts with label replicability. Show all posts

Wednesday, November 28, 2018

Induction-deduction, and replicability: is there any difference?

In what sense--what scientific sense--does the future resemble the past?  Or perhaps, to what extent does it?  Can we know?  If we can't, then what credence for future prediction can we give to results of studies today, necessarily from the past experience of current samples?  Similarly, in what sense can we extrapolate findings on this sample to some other sample or population?  If these questions are not easily answerable (indeed if they are answerable at all!), then much of current, and currently very widespread and expensive science, is at best of unclear, questionable value.

We can look at these issues in terms of a couple of standard aspects of science: the relationship between induction and deduction; and the idea of replicability.  Induction and deduction basically come from the Enlightenment time in western history, when it was found in a formal sense that the world of western science--which at that time meant physical science--followed universal 'laws' of Nature.  At that time, life itself was generally excluded from this view, not least because it was believed to be the result of ad hoc creation events by God.

The induction--deduction problem
-----------------
Some terminology:  I will make an important distinction between two terms.  By induction I mean drawing a conclusion from specific observed data (e.g., estimating some presumed causal parameter's value).  Essentially, this means inferring a conclusion from the past, from events that have already occurred. But often what we want to do is to predict the future.  We do that, often implicitly, by equating observed past values as estimates of causal parameters, that apply generally and therefore to the future; I refer to that predictive process, derived from observed data, as deduction.  So, for example, if I flip a coin 10 times and get 5 Heads, I assume that this is somehow built into the very nature of coin-flipping so that the probability of Heads on any future flip is 0.5 (50%).
-----------------

If we can assume that induction implies deduction, then what we observe in our present or past observations will persist so that we can predict it in the future.  In a law-like universe, if we are sampling properly, this will occur and we generally assume this means with complete precision if we had perfect measurement (here I speculate, but I think that quantum phenomena at the appropriate scale have the same universally parametric properties).

Promises like 'precision genomic medicine', which I think amount to culpably public deceptions, effectively equate induction with deduction: we observe some genomic elements associated in some statistical way with some outcome, and assume that the same genome scores will similarly predict the future of people decades from now.  There is no serious justification for this assumption at present, nor quantification of by how much there might be errors in assuming the predictive power of past observations, in part because mutations and lifestyle clearly have major effects, but especially because these are unpredictable--even in principle.  Indeed, there is another, much deeper problem of a similar kind, that has gotten recent--but to me often quite naive attention: replicability.

The replicability problem
Studies, perhaps especially in social and behavioral fields, report findings that others cannot replicate.  This is being interpreted as suggesting that (ignoring the rare outright fraud), there is some problem with our decision-making criteria, other forms of bias, or poor study designs.  Otherwise, shouldn't studies of the same question agree?  There has been a call for the investigators involved to improve their statistical analysis (i.e., keep buying the same software!! but use it better), report negative results, and so on.

But this is potentially, and I think fundamentally, naive.  It assumes that such study results should be replicable.  It assumes, as I would put it, that at the level of interest, life = physics.  This is, I believe not just wrong but fundamentally so.

The assumption of replicability is not really different from equating induction to deduction, except in some subtle way applied to a more diverse set of conditions.  Induction of genomic-based disease risk is done on a population like, say, case-control samples, and then applied to the same population in terms of its current members' future disease risks.  But we know very well that different genotypes are found in different populations, so it is not clear what degree of predictability we should, or can, assume.

Replicability is similar except that in general a result is assumed to apply across populations or samples, not just to the same sample's future.  That is, I think, an even broader assumption than the genomics-precision promise that does, at least nominally, now recognize population differences.

The real, the deeper problem is that we have absolutely no reason to expect any particular degree of replicability between samples for these kinds of things.  Evolution is about variation, locally responsive and temporary, and that applies to social behavior as well.  We know that 'distance' or difference accumulates (generally) gradually over time and separation as a property of cultural as well as biological evolution.  The same obviously applies even more to psychological and sociological samples and inferences from them.

Not only is it silly to think that samples of, say, this year's college seniors at X University will respond to questionnaires in the same way as samples of some other class or university or beyond.  Of course, college students come cheap to researchers, and they're convenient.  But they are not 'representative' in the replicability sense except by some sort of rather profound assumption.  This is obvious, yet it is a tacit concept of very much research (biological, psychological, and sociological).

Even social scientists acknowledge the local and temporary nature of many of the things they investigate, because the latter are affected by cultural and historical patterns, fads, fashions, and so much more.  Indeed, the idea of replicability is to me curious to begin with.  Thus, a study that fails to replicate some other study may not reflect failings in either, and the idea that we should replicate in this kind of way is a carryover of physics envy.  Perhaps in many situations, a replication result is what should be examined most closely! The social and even biological realms are simply not as 'Newtonian', or law-like, as is the real physical realm in which our notions of science--especially the very idea of a law-like replicability, arose. Not only is failure to replicate not necessarily suspect at all, but replicability should not generally be assumed.  Or, put an other way, a claim that replicability is to be expected is a strong claim about Nature that requires very strong evidence!

This raises the very deep problem that in the absence of replicability assumptions, we don't know what to expect of the next study, after we've done the first.....or is this a justification for just keeping the same studies going (and funded) indefinitely?  That's of course the very rewarding game being played in genomics.

Monday, August 14, 2017

The state of play in science

I've just read a new book that MT readers would benefit from reading as well.  It's Rigor Mortis, by Richard Harris (2017: Basic Books).  His subtitle is How sloppy science creates worthless cures, crushes hope, and wastes billions.  One might suspect that this title is stridently overstated, but while it is quite forthright--and its argument well-supported--I think the case is actually understated, for reasons I'll explain below.

Harris, science reporter for National Public Radio, goes over many different problems that plague biomedical research. At the core is the reproducibility problem, that is, the numbers of claims by research papers that are not reproducible by subsequent studies.  This particular problem made the news within the last couple of years in regard to using statistical criteria like p-values (significance cutoffs), and because of the major effort in psychology to replicate published studies, with a lot of failure to do so.  But there are other issues.

The typical scientific method assumes that there is a truth out there, and a good study should detect its features.  But if it's a truth, then some other study should get similar results.  But many many times in biomedical research, despite huge media ballyhoo with cheerleading by the investigators as well as the media, studies' breakthrough!! findings can't be supported by further examination.

As Harris extensively documents, this phenomenon is seen in claims of treatments or cures, or use of animal models (e.g., lab mice), or antibodies, or cell lines, or statistical 'significance' values.  It isn't a long book, so you can quickly see the examples for yourself.  Harris also accounts for the problems, quite properly I think, by documenting sloppy science but also the careerist pressures on investigators to find things they can publish in 'major' journals, so they can get jobs, promotions, high 'impact factor' pubs, and grants. In our obviously over-crowded market, it can be no surprise to anyone that there is shading of the truth, a tad of downright dishonesty, conveniently imprecise work, and so on.

Since scientists feed at the public trough (or depend on profits and sales for biomedical products to grant-funded investigators), they naturally have to compete and don't want to be shown up, and they have to work fast to keep the funds flowing in.  Rigor Mortis properly homes in on an important fact, that if our jobs depend on 'productivity' and bringing in grants, we will do what it takes, shading the truth or whatever else (even the occasional outright cheating) to stay in the game.

Why share data with your potential competitors who might, after all, find fault with your work or use it to get the jump on you for the next stage?  For that matter, why describe what you did in enough actual detail that someone (a rival or enemy!) might attempt to replicate your work.....or fail to do so? Why wait to publish until you've got a really adequate explanation of what you suggest is going on, with all the i's dotted and t's crossed?  Haste makes credit!  Harris very clearly shows these issues in the all-too human arena of our science research establishment today.  He calls what we have now, appropriately enough, a "broken culture" of science.

Part of that I think is a 'Malthusian' problem.  We are credited, in score-counting ways, by chairs and deans, for how many graduate students we turn (or churn) out.  Is our lab 'productive' in that way?  Of course, we need that army of what often are treated as drones because real faculty members are too busy writing grants or traveling to present their (students') latest research to waste--er, spend--much time in their labs themselves.  The result is the cruel excess of PhDs who can't find good jobs, wandering from post-doc to post-doc (another form of labor pool), or to instructorships rather than tenure-track jobs, or who simply drop out of the system after their PhD and post-docs.  We know of many who are in that boat; don't you?  A recent report showed that the mean age of first grant from NIH was about 45: enough said.

A reproducibility mirage
If there were one central technical problem that Harris stresses, it is the number of results that fail to be reproducible in other studies.  Irreproducible results leave us in limbo-land: how are we to interpret them?   What are we supposed to believe?  Which study--if any of them--is correct?  Why are so many studies proudly claiming dramatic findings that can't be reproduced, and/or why are the news media and university PR offices so loudly proclaiming these reported results?  What's wrong with our practices and standards?

Rigor Mortis goes through many of these issues, forthrightly and convincingly--showing that there is a problem.  But a solution is not so easy to come by, because it would require major shifting of and reform in research funding.  Naturally, that would be greatly resisted by hungry universities and those who they employ to set up a shopping-mall on their campus (i.e., faculty).

One purpose of this post is to draw attention to the wealth of reasons Harris presents for why we should be concerned about the state of play in biomedical research (and, indeed, in science more generally).  I do have some caveats, that I'll discuss below, but that is in no way intended to diminish the points Harris makes in his book.  What I want to add is a reason why I think that, if anything, Harris' presentation, strong and clear as it is, understates the problem.  I say this because to me, there is a deeper issue, beyond the many Harris enumerates: a deeper scientific problem.

Reproducibility is only the tip of the iceberg!
Harris stresses or even focuses on the problem of irreproducible results.  He suggests that if we were to hold far higher evidentiary standards, our work would be reproducible, and the next study down the line wouldn't routinely disagree with its predecessors.  From the point of view of careful science and proper inferential methods and the like, this is clearly true.  Many kinds of studies in biomedical and psychological sciences should have a standard of reporting that leads to at least some level of reproducibility.

However, I think that the situation is far more problematic than sloppy and hasty standards, or questionable statistics, even if they are clearly a prominent ones.  My view is that no matter how high our methodological standards are, the expectation of reproducibility flies in the face of what we know about life.  That is because life is not a reproducible phenomenon in the way physics and chemistry are!

Life is the product of evolution.  Nobody with open eyes can fail to understand that, and this applies to biological, biomedical, psychological and social scientists.  Evolution is at its very core a phenomenon that rests essentially on variation--on not being reproducible.  Each organism, indeed each cell, is different. Not even 'identical' twins are identical.

One reason for this is that genetic mutations are always occurring, even among the cells within our bodies. Another reason is that no two organisms are experiencing the same environment, and environmental factors affect and interact with the genomes of each individual organism of any species.  Organisms affect their environments in turn. These are dynamic phenomena and are not replicable!

This means that, in general, we should not be expecting reproducibility of results.  But one shouldn't overstate this because while obviously the fact that two humans are different doesn't mean they are entirely different.  Similarity is correlated with kinship, from first-degree relatives to members of populations, species, and different species.  The problem is not that there is similarity, it is that we have no formal theory about how much similarity.  We know two samples of people will differ both among those in each sample and between samples.  And, even the same people sampled at separate times will be different, due to aging, exposure to different environments and so on. Proper statistical criteria and so on can answer questions about whether differences seem only due to sampling from variation or from causal differences.  But that is a traditional assumption from the origin of statistics and probability, and isn't entirely apt for biology: since we cannot assume identity of individuals, much less of samples or populations (or species, as in using mouse models for human disease), our work requires some understanding of how much difference, or what sort of difference, we should expect--and build into our models and tests etc.

Evolution is by its very nature an ad hoc phenomenon in both time and place, meaning that there are no fixed rules about this, as there are laws of gravity or of chemical reactions. That means that reproducibility is not, in itself, even a valid criterion for judging scientific results.  Some reproducibility should be expected, but we have no rule for how much and, indeed, evolution tells us that there is no real rule for that.

One obvious and not speculative exemplar of the problem is the redundancy in our systems. Genomewide mapping has documented this exquisitely well: if variation at tens, hundreds, or sometimes even thousands of genome sites' affects a trait, like blood pressure, stature, or 'intelligence' and no two people have the same genotype, then no two people, even with the same trait measure have that measure for the same reason.  And as is very well known, mapping only accounts for a fraction of the estimated heritability of the studied traits, meaning that much or usually most of the contributing genetic variation is unidentified.  And then there's the environment. . . . .

It's a major problem. It's an inconvenient truth.  The sausage-grinder system of science 'productivity' cannot deal with it.  We need reform.  Where can that come from?

Monday, August 31, 2015

It's not just about psych studies; it's about core aspects of inference.

A paper just published in Science by the "Open Science Collaboration" reports the results of a multi-year multi-institution effort to replicate 100 psychology studies published in three top psychology journals in 2008.  This effort has often been discussed since it began in 2011, in large part because the importance of replicability in confirming scientific results is integral to the 'scientific method,' but replicability studies aren't a terribly creative use of a researcher's time, and they're difficult to publish so they aren't often on researchers' To-Do lists.  So, this was unusual.

Les Twins; Wikipedia
There are many reasons a study can't be replicated.  Sometimes the study was poorly conceived or carried out (assumptions and biases not taken into account), sometimes the results pertain only to the particular sample reported (a single family or population), sometimes the methods in an original study aren't described well enough to be replicated, sometimes random or even systematic error (instrument behaving badly) skews the results.

Because there's no such thing as a perfect study, replication studies can be victims of any of the same issues, so interpreting lack of replication isn't necessarily straightforward, and certainly doesn't always mean that the original study was flawed.

The Open Science Collaboration was scrupulous in its efforts to replicate original studies as carefully and faithfully as possible.  Still, the results weren't pretty.  The authors write:
Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects.
Interestingly enough, the authors aren't quite sure what any of this means.  First, as they point out, direct replication doesn't verify the theoretical interpretation of the result, which could have been flawed originally, and remain flawed.  And it's impossible to know when a study is not replicated why that is, whether the original was flawed or the replication effort was flawed, or even both were flawed.

This effort has been the subject of much discussion, naturally enough.  In a piece published last week in The Atlantic, Ed Yong quotes several psychologists, including the project's lead author, saying that this project has been a welcome learning experience for the field.  There are plans afoot to change how things are done, including pre-registration of hypotheses so that the reported results can't be cherry-picked, or increasing the size of studies to increase their power, as has been done in the field of genetics.

We'll see whether this is just a predictable wagon-circling welcome, or really means something.  One has every reason to be skeptical, and wonder if these fields really are sciences in the proper sense of the term.  Indeed, it's quite interesting to see genetics held up as an exemplar of good and reliable study design.  After billions of dollars being spent on studies large and small of the genetics of asthma, heart disease, type 2 diabetes, obesity, hypertension, stroke, and so on, we've got not only a lot of contradictory findings, but most of what has been found are genes with small effects.  And epidemiology, many of the 'omics fields, evolutionary biology, and others haven't done any better.

Why?  The vagueness of the social and behavioral sciences is only part of the problem (unlike, say, force, outcome variables such as stress, aggression, crime, or intelligence are hard to consistently define, and can vary according to the instrument with which they are measured).  Biomedical outcomes can be vague and hard to define as well (autism, schizophrenia, high blood pressure).  We don't understand enough about how genes interact with each other or with the environment to understand complex causality.

Statistics and science
The problem may be much deeper than any of this discussion of non-replicable results suggests.  First, from an evolutionary point of view, we expect organisms to be different, not replicates.  This is because mutational changes (and recombination) are always making each individual organism's genotype unique and, second, the need to adapt--Darwin's central claim or observation--means that organisms have to be different so that their 'struggle for life' can occur.

We have only a general theory for this, since life is an ad hoc adaptive/evolutionary phenomenon. Far more broadly than just the behavioral or social sciences, our investigative methods are based on 'internal' comparisons (e.g., cases vs controls, various levels of blood pressure and stroke, fitness relative to different trait values) to evaluate samples against each other, rather than as representations of an externally derived, a priori theory.  When we rely on statistics and p-value significance tests and probabilities and so on, we are implicitly confessing that we don't in fact really know what's going on, and all we can get are a kind of shadow of the underlying process that is cast by the differences we detect, and we detect them with generic (not to mention subjective) rather than specific criteria. We've written about these things several times in the past here.

The issue is not just weakly defined terms and study designs. As Freeman Dyson (in "A meeting with Enrico Fermi") wrote in 2004:
In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, "How many arbitrary parameters did you use for your calculations?" I thought for a moment about our cut-off procedures and said, "Four." He said, "I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk." With that, the conversation was over. . . .
Here, parameters refers to what are more properly called 'free parameters', that is, ones not fixed in advance, but that are estimated from data. By contrast, for example, in physics the speed of light and gravitational constant are known, fixed values a priori, not estimated from data (though data were used to establish those values). We just lack such understanding in many areas of science, not just behavioral sciences.

In a sense we are using Ptolemaic tinkering to fit a theory that doesn't really fit, in the absence of a better (e.g., Copernican or Newtonian) theoretical understanding.  Social and behavioral sciences are far behind the at least considerably more rigorous genetic and evolutionary sciences when the latter are done at their best (which isn't always).   Like the shadows of reality seen in Plato's cave, statistical inference reflects the shadows of the reality we want to understand, but for many cultural and practical reasons we don't recognize that, or don't want to acknowledge it.  The weaknesses and frangibility of our predictive 'powers' could, if properly understood by the general public, be a threat to our business and our culture doesn't reward candor when it comes to that business.  The pressures (including from the pubic media, with their own agenda and interests) necessarily lead to reducing complexity to simpler models and claims far beyond what has legitimately been understood.

The problem is not just with weakly measured variables or poorly defined terms of, for example, outcomes.  Nor is the problem if, when, or that people use methods wrongly.  The problem is that statistical inference is based on a sample and is often retrospective, or mainly empirical and based on only rather generic theory.  No matter how well chosen and rigorously defined, in these various areas (unlike much of physics and chemistry) the estimates of parameters and the like fitted to data that is necessarily about the subjects past, such as their culture or upbringing or lifestyles, but in the absence of adequate formal theory, these findings cannot be used to predict the future with knowable accuracy.  That is because the same conditions can't be repeated, say, decades from now, and we don't know what future conditions will be, and so on.

Rather alarmingly, we were recently discussing this with a colleague who works in very physics- and chemistry-rigorous material science.  She immediately told us that they, too, face problems in data evaluation with the number of variables they have to deal with, even under what the rest of us would enviously say were very well-controlled conditions where the assumptions of statistics--basically amounting to replicability of some underlying mathematical process--should really apply well.

So the social and related sciences may be far weaker than other fields, and should acknowledge that. But the rest of us, in various purportedly 'harder' biological, biomedical, and epidemiological sciences, are often not so much better off. Statistical methods and theory work wonderfully well when their assumptions are closely met.  But there is too much out-of-the-box analytic toolware, that lures us into thinking that quick and definitive answers are possible.  Those methods never promise that because what statistics does is account for repeated phenomena following the same rules, and the rule of many sciences is that, in their essence they are not following such rules.

But the lure of easy-answer statistics, and the understandable lack of deeply better ideas, perpetuates the expensive and misleading games that we are playing in many areas of science.

Thursday, May 28, 2015

Discovering the "laws of life": where are we?

For several years, I've been in an off-and-on discussion about topics relevant to MT readers, with a thoughtful undergraduate student, named Arjun Plakett.  He has now graduated, but we're still in at least occasional contact.  He is a chemical engineering graduate, but is interested in the wider issues of science.

I mention this because Arjun recently brought to my attention an old, but thought-provoking and, I think, relevant book.  It is by the late Milton Rothman, and called Discovering the Natural Laws (1972, reprinted by Dover, 1989).  This book is a very interesting one, and I wish I had known of it long ago. It concerns the history by which some of the basic 'laws' of physics were shown to be just that: laws. And, while its specific topic is physics, it is very relevant to what is going on in genetics at present.

Source: my copy's cover

Rothman deals with many presumed laws of Nature, including Newton's laws of motion, conservation, relativity, and electromagnetisim.  For example, we all know that Newton's universal law of gravitation is that the gravitational attraction between two objects, of mass M and N, a distance r apart, is

F = G MN/r^2

(the denominator is r-squared).  G is the universal gravitational constant.  But how do we know this?  And how do we know it is a 'universal law'?  For example, as Rothman discusses, why do we say the effect is due to the exact square (power 2) of the distance between the objects?  Why not, for example, the 2.00005 power?  And what makes G a, much less the, constant? Why constant? Why does the same relationship apply regardless of the masses of the objects, or what they're made of? And what makes it universal--how could we possibly test that?

The fact that these laws are laws, as Rothman details for this and other laws of physics, was not easy to prove.  For example, what two objects would be used to show the gravitational law, where measurement errors and so on would not confound the result?  The Earth and Moon won't do, despite Newton's use of them, because they are affected by the pull of the Sun and other planets etc.  Those effects may be relatively small, but they do confound attempts to prove that the law is in fact a true, exact law.

A key to this demonstration of truth is that as various factors are accounted for, and measurement becomes more accurate, and different approaches triangulate, the law data become asymptotically close to the predictions of the theory: predictions based on theory get ever closer to what is observed.

In fact, and very satisfyingly, to a great approximation these various principles do appear to be universal (in our universe, at least) and without exception.  Only when we get to the level of resolution of individual fundamental particles, and quantum effects, do these principles break down or seem to vary.  But even then the data approach a specifiable kind of result, and the belief is that this is a problem of our incomplete understanding, not a sign of the fickleness of Nature!

Actually if a value, like G or the power 2 were not universal but instead were context-specific, but replicable in any given context, we could probably show that, and characterize those situations in which a given value of the parameter held, or we could define with increasing accuracy some functional relationship between the parameter's value and its circumstances.

But Mermaid's Tale is about evolution and genetics, so what is the relevance of these issues?  Of course, we're made of molecules and have to follow the principles--yes, the laws--of physics.  But at the scale of genes or organisms or evolution or disease prediction, what is its relevance?  The answer is epistemological, that is, about the nature of knowledge.

Theory and inference genetics and evolution: where are the 'laws'?  Are there laws?
When you have a formal, analytic or even statistical theory, you start with that, a priori one might say, and apply it to data either to test the theory itself, or to estimate some parameter (such as F in some setting such as a planet's orbital motion).  As I think it can be put, you use or test an externally derived theory.

Rothman quotes Karl Popper's view that "observation is always observation in the light of theories" which we test by experiments in what Rothman calls a 'guess and test' method, otherwise known as 'the scientific method'.  This is a hypothesis-driven science world-view.  It's what we were all taught in school.  Both the method and its inferential application depend on the assumption of law-like replicability.

Popper is most known for his idea that replication of observations never prove an hypothesis because the next observation might falsify it, but it only takes one negative observation to do that.  There may be some unusual circumstance that is valid but you hadn't yet encountered.  In general, but perhaps particularly in biomedical sciences, it is faddish and often self-serving to cite 'falsifiability' as our noble criterion, as if one has a deep knowledge of epistemology.  Rothman, like almost everyone, quotes Popper favorably.  But in fact, his idea doesn't work.  Not even in physics!  Why?

Any hypothesis can be 'falsified' if the experiment encounters design or measurement error or bad luck (in the case of statistical decision-making and sampling).  You don't necessarily know that's what happened, only that you didn't get your expected answer.  But what about falsifiability in biology and genetics?  We'll see below that even by our own best theory it's a very poor criterion in ways fundamentally beyond issues of lab mistakes or bad sampling luck.

We have only the most general theories of the physics-like sort for phenomena in biology.  Natural selection and genetic drift, genes' protein-coding mechanisms, and so on are in a sense easy and straightforward examples. But our over-arching theory, that of evolution, says that situations are always different.  Evolution is about descent with modification, to use Darwin's phrase, and this generates individuals as non-replicates of each other.  If you, very properly, include chance in the form of genetic drift, we don't really have a formal theory to test against data.  That's because chance involves sampling, measurement, and the very processes themselves.  Instead, in the absence of a precisely predictive formal theory, we use internal comparisons--cases vs controls, traits in individuals with, and without, a given genotype, and so on.  What falsifiability criterion should you apply and, more importantly, what justifies your claim of that criterion?

Statistical comparison may be an understandable way to do business under current circumstances that do have not provided a precise enough theory, but it means that our judgments are highly dependent on subjective inferential criteria like p-value significance cutoffs (we've blogged about this in the past).  They may suggest that 'something is going on' rather than 'nothing is going on', but even then only by the chosen cutoff standard.  Unlike the basic laws of physics, they do not generate asymptotically closer and closer fits to formally constructed expectations.

What is causal 'risk'?
We may be interested in estimating 'the' effect of a particular genetic variant on some trait.  We collect samples of carriers of one or two copies of the variant, and determine the risk in them, perhaps comparing this to the risk in samples of individuals.  But this is not 'the' risk of the variant!  It is, at best, an estimate of the fraction of outcomes in our particular data.  By contrast, a probability in the prediction sense is the result of a causal process with specific properties, not an empirical observation.  We usually don't have the evidence to assert that our observed fraction is an estimate of that underlying causal process.

Collecting larger samples will not in the appropriate sense lead asymptotically to a specific risk value.  That's because we know enough about genes to know that an effect is dependent on the individual's genomic background and life-experience.  Each individual has his/her own, unique 'risk'.  In fact, the term 'risk' itself, applied to an individual, is rather meaningless because it is simply inappropriate to use such individual concepts in this context, when the value is based on a group sample.  Or, put another way, the ideas about variance in estimates and so on just don't apply, because individuals are not replicable observations.  The variance of group estimates are different from what we might want to know about individuals.

These are the realities of the data, not a fault with the investigators.  If anything, it is a great triumph that we owe to Darwin whatever his overstatement of selection as a 'law' of nature, that we know these realities!  But it means, if we pay attention to what a century of fine life science has clearly shown, that fundamental context-dependence deprives us of the sort of replicability found in the physical sciences.  That undermines many of the unstated assumptions in our statistical methods (the existence of actual probability values, proper distributions from which we're sampling, replicability, etc.).

This provides a generic explanation for why we have the kinds of results we have, and why the methods being used have the problems they have.  In a sense, our methods, including statistical inference, are borrowed from the physical sciences, and may be inappropriate for many of the questions we're asking in evolution and contemporary genetics, because our science has shown that life doesn't obey the same 'laws' at the levels of our observation.

Darwin and Mendel had Newton envy, perhaps, but gave us our own field--if we pay attention
Rothman's book provides, to me, a clear way to see these issues. It may give us physics envy, a common and widespread phenomenon in biology.  I think Darwin and Mendel were both influenced indirectly by Newton and the ideas of rigorous 'laws' of nature.  But they didn't heed what Newton himself said in his Principia in 1687: "Those qualities of bodies that . . . belong to all bodies on which experiments can be made should be taken as qualities of all bodies universally." (Book 3, Rule 3, from the Cohen and Whitman translation, 1999)  That is, for laws of nature, if you understand behavior of what you can study, those same laws must apply universally, even to things you cannot or have not yet studied.

This is a cogent statement of the concept of laws, which for Newton was "the foundation of all natural philosophy."  It is what we mean by 'laws' in this context, and why better designed studies asymptotically approach the theoretical predictions.  But this principle is one that biomedical and evolutionary scientists and their students should have tattooed on so they don't forget.  The reason is that this principle of physics is not what we find in biology, and we should be forced to keep in mind and take seriously the differences in the sciences, and the problems we face in biology.  We do not have the same sort of replicability or universality in life unless non-replicability is our universal law.

If we cannot use this knowledge to ferret out comparable laws of biology, then what was the purpose of 150 years of post-Darwinian science?  It was to gain understanding, not to increase denial.  At least we should recognize the situation we face, which includes stop making misleading or false promises about our predictive, retrodictive, or curative powers, that essentially assume as laws what our own work has clearly proven are not laws.

Thursday, May 24, 2012

Magical science: now you see it, now you don't. Part I: What we mean by 'risk'

The  life, social, and evolutionary sciences have a problem.  We posted about the issue of their non-replicability last Friday but that is only part of the problem.  They also have non-predictability (see a recent Times commentary), but both replicability and predictability are key elements of science as we know it. 

It is difficult to make rigorous assertions that have the kind of predictive power we have come (rightly or wrongly) to expect of science, on the typical if often unstated assumption that our world is law-like, the way it seems that the physical and chemical universe are.

A clear manifestation of the problem is the way that findings in epidemiology and genetic risk say yes factor X is risky, then a few years later no, then yes again, and so on.   Can't we ever know if factor X is a cause of some outcome Y?  In particular, is X a risk factor?  X could be a reason for natural selection, a component of some disease, and so on.

Often, we think of this as a probability.  As we've posted before, that means that exposure to that factor yields some probability that the outcome will be observed.  If you have two copies of a gene, there is usually a 50% risk or probability that a given one of your children (or parents, or sibs) will also have it.

So, in a strange turn of phrase, coffee or a given HDL cholesterol level is said to be a risk factor for heart disease, or a given genetic variant is a risk factor for cancer.  And we try to estimate the level of that risk--the probability that if that factor is present, you will manifest the outcome.  Among various possible risk factors, the modern concept of science has it that you are at some net or overall risk of the outcome, like having a heart attack, depending on your exposure to those risk factors.

In evolutionary terms, having a particular genetic variant can have some probability  (or some similar measure) of reproducing, or surviving to a particular age.  Among various possible genetic risk factors, what you have puts you at some net risk of such outcome, which is your evolutionary fitness in the face of natural selection, for example. 

If we assume that enumerated causes of this sort, and that they really are causes, are responsible for a trait then your exposure level can be specified.  The causes might truly be deterministic, in the way gravity determines the rate an object will fall--here or anywhere in the universe--but that our incomplete level of knowledge is such that we can only express its effect in terms of probability.

Still, we assume that probabilistic causation is real.  When things are the result of probabilities, we can know the causes but can't predict the specific outcome of any given instance.  This is the sense in which we know a fair coin will come up Heads 50% of the time, but can't predict the result of a given flip.  Actually, and we've posted about this before but the issue of probability is so central to much of science that we keep repeating it, the coin may be perfectly deterministic but we just don't know enough, so that for all practical purposes the result is probabilistic.

In such cases, which are clearly at the foundation of evolutionary inference and of genetic and other biomedical problems, we must estimate the risk associated with a given cause by choosing a sample from all those at risk, and seeing what happened to them.  Then, we assume we know the causal structure and can then do what we must be able to do, if this is actual science: predict the outcome.  This must be so if the world is causal, even if our predictions are expressed in terms of probabilities: given your genotype you have xx probability of getting yy disease.

So, with our huge and munificently funded science establishment, why is it that day after day the media tout the latest Dramatic Finding....that is just as noisily touted the next day when the previous assertion is overturned?

Why is it that we don't know if coffee is a risk factor for disease?  Or isn't?  Or isn't for the moment until some new study comes along?  Or maybe until some environmental factor changes, like the type of filter paper McDonald's uses in its coffee maker, say--but how would we ever know whether that explains the flip-flopping findings?

Why indeed do we have to continue doing studies of the same purported risk factor to see if they are really, truly risks?  These are fundamental questions not about the individual studies, but about the current practice of science itself.

If we look at the reasons, which is tomorrow's post, we'll see how shaky our knowledge really is in these areas, and we can ask whether it is even 'science'.

Friday, May 18, 2012

Non-replicability in science: your antelope for the day

A piece in the May 17 Nature supports one of Ken's favorite observations, something he says while wearing his Anthropologist's hat -- "Journal articles are just an academic's antelope for the day."  We're still just hunter/gatherers -- our published papers are, more often than not, nothing more than the way we feed ourselves.  Our basket of berries -- eaten today, droppings tomorrow.

Blackbuck male, females; Photo from Wikimedia, Mr Raja Purohi
Ed Yong, in "Replication studies: Bad copy," reports that most published studies can't be replicated.  This is something we often talk about with respect to genetic studies, and there are many reasons for this that are specific to genetic data, but apparently it's even more rampant in psychology, for reasons also specific to the field.  

And there is the notorious problem that 'negative' results are not published very often.  They're not glamorous and won't get you tenure--even if some of the most important findings in science are 'negative' if they steer work towards valid rather than dreamt-of theory or hypothesis.  Clinical trials are a major example, but less noticed are ephemeral natural selection stories about evolution.

A paper published last year claiming support for extrasensory perception, or psi, for example, produced a major kerfuffle (we blogged about it at the time).  The aftermath has been no less interesting, and informative about the world of publishing, as researchers who tried to replicate the findings but failed also failed to find publishers for their results.  This lead to a lot of discussion about the implications of negative results not being published, a discussion that has flared up frequently in academia, as well it should, although we're no closer to resolving it than ever.
There are some experiments that everyone knows don't replicate, but this knowledge doesn't get into the literature,” says [Eric-Jan] Wagenmakers [mathematical psychologist at the University of Amsterdam]. The publication barrier can be chilling, he adds. “I've seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.”
But we'll leave that issue for another time.

The question of why studies so often aren't replicable is a different, if related one.  And one that The Reproducibility Project, a large scale collection of scientists from around the world, is addressing head on, as they attempt to replicate every study published in three major psychology journals in 2008, as described last month in the Chronicle of Higher Education.  
For decades, literally, there has been talk about whether what makes it into the pages of psychology journals—or the journals of other disciplines, for that matter—is actually, you know, true. Researchers anxious for novel, significant, career-making findings have an incentive to publish their successes while neglecting to mention their failures. It’s what the psychologist Robert Rosenthal named “the file drawer effect.” So if an experiment is run ten times but pans out only once you trumpet the exception rather than the rule. Or perhaps a researcher is unconsciously biasing a study somehow. Or maybe he or she is flat-out faking results, which is not unheard of. 
According to Yong, the culture in psychology is such that experimental designs that "practically guarantee positive results" are perfectly acceptable.  This is one of the downsides of peer review -- when all your peers are doing it, good scientific practice or not, you can get away with it, too.
And once positive results are published, few researchers replicate the experiment exactly, instead carrying out 'conceptual replications' that test similar hypotheses using different methods. This practice, say critics, builds a house of cards on potentially shaky foundations.
So, if a study isn't replicated exactly (or however exactly it can be), it's possibly because the methods were not described in enough detail for the study to be replicated.  Or, and this is a problem certainly not confined to psychology, the effect was small and significant by chance, as epidemiologist John Ionnides suggested in a paper published in 2005 that garnered a lot of attention for saying most Big-Splash studies are false.  He explained this in statistical terms, having to do with bias in significance levels of studies of new hypotheses and similar issues.

As the Chronicle story says about non-replicability:
The researchers point out, fairly, that it’s not just social psychology that has to deal with this issue. Recently, a scientist named C. Glenn Begley attempted to replicate 53 cancer studies he deemed landmark publications. He could only replicate six. Six! Last December I interviewed Christopher Chabris about his paper titled “Most Reported Genetic Associations with General Intelligence Are Probably False Positives.” Most!
So, psychology is under attack.  We blogged not long ago about an op/ed piece in the New York Times by two social scientists calling for an end to the insistence that the social sciences follow any scientific method.  Enough with the physics envy, they said, we don't do physics.  Thinking deeply is the answer.  But, would giving these guys free rein to completely make stuff up really be the solution?  Well, it might just be, if their peers agree. But, let's not just pick on psychology.  The problem is rampant throughout the sciences. 

Meanwhile, the motto seems to be:  Haste makes....nutrition for scientists!