Even more on posing the well-posed....

Here's another installment on the idea of well-posed questions in evolutionary and biomedical genetics, and related areas of epidemiology.  We wrote about why the kinds of questions that these fields have been able to ask about causation has changed, and why well-posed questions are less and less askable now.  We've tried to deconstruct the questions that are being asked currently and explore why they aren't well-posed and why they can't be.  Today, we think further about what it means to pose answerable questions about causality.

The causal assumption
A well-posed question can be about the fact of a numerical association, say between some specified variable and some outcome.  Or, it can be about the reason for that association.  Do we live in a causal universe?  This may seem like a very stupid question, but it is actually quite profound. If we live in a causal universe, then everything we see must have a cause: there can be no effect, no observation, without an antecedent cause--a situation that existed before our observation and that is responsible for it.

Actually, some physicists argue from what is known of quantum mechanics (physical processes on the atomic or smaller scale) that there are, indeed, effects that do not have causes.  One way to think of this, without being quite as strange relative to common sense as much of quantum mechanics is, is this:  the world must be ultimately deterministic.  Some set of conditions must make our outcome inevitable!  If you knew those conditions, the probability of the effect would be 1 (100%).

In that sense, the word 'probability' has no real meaning.  But if to the contrary, outcomes were not inevitable, because things were truly probabilistic, then our attempts to understand nature would or should be different in some way.  The usual idea of 'laws' of Nature is that the cosmos is deterministic, even if many things like sampling and measurement yield errors relative to our ability to infer the whole, exact truth.  And 'chaos' theory says that it may be that even the tiniest errors, even in a truly deterministic system, can lead to uncontrollably or unknowably large errors.

These ideas are fundamental when we think about our topics of interest in genetics, public health, and evolutionary biology.

The statistical gestalt
A widespread approach in these areas is to see how much juice we can squeeze out of a rock with ever-fancier statistical methods (this is helped by the convenient software we have to play with).  Whether the variables are genomic or environmental or both, we'll slice and dice this way and that, and find the fine-points of 'risk' (that is, retrospectively ascertained already achieved outcomes).

Now, one may think that there really is some 'juice' in the rock:  that the world is truly deterministic, and that the juice we see is the causal truth masked by all the mismeasured and unmeasured variables, incomplete samples, and the other errors we fallible beings inevitably make.   Our statistical results may not be good for prediction, or may be good but to an unknown extent, but if we think of the world as causal, there is for each instance of some trait, the wholly deterministic conditions that made it as inevitable as sunrise.  In defense of our hard-to-interpret methods and results, we'll say "it's the best we can do, and it will work at least for some sets of factors." And so let's keep on using our same statistical methods to squeeze the rock.

One may explain this view by saying that life is complex, and statistical sampling and inferential methods are how to address it.   This could be, and certainly in part seems to be, because many issues of importance involve so many factors, interacting in so many different ways that each individual is unique (even if causation is 100% deterministic).  This raises fundamental issues as to how we could ever deliver the promised successes (people will still get sick, despite what Francis Collins and so many others implicitly promise).  Perhaps what we're doing is not just the best we can do now, it's the best that can be done, because it's just how Nature is!

Rococo mirror and stuccowork, Schloss Ludwigsburg, Stuttgart

But we should keep in mind one aspect of the usual sorts of statistical methods, no matter how many rococo filigrees they may be gussied up with.  These methods find associations (correlated presence) among variables, and if we can specify which occurred first, then we can interpret those as causes and the latter as effects.  We can estimate the strength or 'significance' of the observed association.  But we should confess openly that these are only what was captured in some particular sample.  Even if they truly do reflect underlying causation, this can be so indirect as to be largely impenetrable (unless the cause is so strong that we can design experiments to demonstrate it, or find it with samples so small and cheap that our Chair will penalize us for not proposing a bigger, more expensive study).

Now associations of this sort can be very indirect relative to the actual underlying mechanism.  Why?  The mix of factors--their relative proportion or frequency--in our sample may affect the strength of association.  An associated,  putatively causal factor may be correlated with some truly causal, but unmeasured, factors.  The complexity of contexts may mean that causation only occurs in the presence of other factors (context-dependency).

If we do not have a theory, or mechanism, of causation, we are largely doomed to floundering in the sea of complexity, as we clearly are experiencing in many relevant areas today.  More data won't help if it generates more heterogeneity rather than enough replications of similar causal situations.  More data will feed SPSS or Plink, very fine analysis software, but not necessarily real knowledge.

This might be true, but to me it's just stalling to justify business more or less as usual.  We have had a century of ever increasing data scale and numerical statistical tricks and we are where we are.  As we and others note, there have been successes.  But most of them are because of strong, replicable effects (that were or could have been found with cheaper methods and smaller, more focused studies).   We're not that much better off for many of the more common problems.

Often it is argued that we can use a simplified 'model' of causation, ignoring minor effects and looking for just the most important ones.  This can of course be a good strategy, and some times a true attempt is made to connect relevant variables with at least half-way relevant interactions (linear, multiplicative, etc.); but more often what is modeled is a set of basic regression or correlation associations between some sets of variables (such as a 'network'), but without a serious attempt at a mechanism that explains the quantitative causal relationships and in a sense still relies on very indirect associations (usually we just don't have sufficient knowledge for being more specific). We've posted before (in a series that starts here, e.g.) on some of the 'strange' phenomena we know apply to genetics that mean that local sites along the genome can have their effects only because of complex, very non-local aspects of the genome.  Significant associations only sometimes lead to understanding of why.

But if we have a good causal model, both the statistical associations, and nature of causation become open to understanding--and to prediction.  With a theory, or mechanism, we can sample more efficiently, control for and understand measurement and other errors more effectively, select variables more relevantly, separate out human fallibility variation (the truly statistical aspects of studies) from causation, interpret the results more specifically, and predict--an ultimate goal--with more confidence and precision.  A good causal mechanism provides these and many other huge advantages over generic association testing, which is mainly what is going on today.

Of course, it's hellishly difficult even in controlled areas like chemistry, to develop good mechanistic understanding of life.  That is to me no excuse for the kind of many-layered removal between what we estimate from data now, and what we should be trying to understand.  Even if it's harder to get grants if we actually try to do things right.

There are some seemingly insuperable issues, and we've written about them before.  One is that we fit data to restrospective data, about events that have already happened, yet we often speak of that as if it is retrodiction as if that means it has predictive value.  But we know that many risk factors, even genetic ones, depend on future events unpredictable in principle, so we can't know enough about what we're modeling.  And second, if we're modeling one outcome--diabetes, say--then the risks of other 'competing' outcomes, say cancer rates, clearly affect our risks for the target outcome, and in subtle ways.  If cancer were eliminated, that alone would raise the risk of diabetes because people would live to have longer exposures to diabetes risk factors.  Since we don't know about such changes in other competing factors that may occur, this is not really a problem that we can build into causal or mechanistic models of a given trait--a real dilemma and, probably, a rationale for doing the best statistical data-fitting that we can.  But I think we too routinely we use such things as excuses for not thinking more deeply

Of course, if the causation in our cosmos is truly probabilistic, and effects just don't have causes, or we don't know the probability values involved or how such values are 'determined' (or if they are), then we may really be in the soup when it comes to predicting things we want to predict that have many complex contributing factors.

Forming well posed questions, not easy but ....
To me personally, it's time to stop squeezing the statistical rock.  Whether or not the world is ultimately deterministic or not, it is time to ask:  What if our current concepts about how the world works are somehow wrong, or more particularly if the concepts underlying statistical approaches are not of the right kind?  Or if we rely on vanilla methods that are chosen because they allow us to go collect data without actually understanding anything (this is, after all, rather what genomics' boasts of 'hypothesis free' methods confesses).

Even if we have no good alternatives (yet), one can ask how might we look for something different--not while we continue to do the same sorts of statistical sampling approaches, but instead of continuing to do that?  Theory can have a bad name, if it just means woolly speculation or great abstraction irrelevant to the real world.  But much of standard statistical analysis is based on probability theory and degrees of indirection and 'models' so generic that they are just as woolly, even if they generate answers, because the 'answers' aren't really answers, and won't  really be answers, until we are somehow forced to start asking well-posed questions.

Some day, some lucky person, will hit upon a truly new idea....

More on well-posed questions and how to ask them

Yesterday we discussed what are called 'well-posed" questions.  Science is the enterprise by which we try to answer questions we have about Nature and the world.  Science is, if anything, a kind of method of investigation, a philosophy of what counts as knowledge.  But it's not trivial that you can't answer a question until you've asked it, and if you haven't done that properly, well....

The history of western approaches to asking and answering questions about Nature is long and varied.  In the earliest western writings, the 'classics' such as Ptolemy, Aristotle, the Pythagoreans, Hippocrates, Euclideans, and the Epicurians, the idea was that one could observe the world and then think-out theories of how the world is.  There were experimental works and formal theories, like geometry, but much was done without formal or systematic ways of codifying knowledge. Science was, to a great extent, opinion.  There were schools of thought populated by groups with different opinions, but little way to resolve them.  Some became dominant (such as the four humors and medicine), but their epistemology--their knowledge basis--was informal.

French salon in the age of Enlightenment; Wikipedia

In the so-called 'Enlightenment' period in European history (to our chagrin, we know very little about the history of science and philosophy elsewhere in the world), the idea took hold that only by careful, systematic observation of specific, constrained instances of Nature could answers to general questions be developed.  Induction based on repeated observation, reduction to basic causal elements, and formal types of reasoning (especially, mathematics) were approaches to the world.  Underlying this (as in the classical times before, to an extent), was the idea that Nature was a material phenomenon (perhaps started and occasionally interrupted by God, depending on one's religious views), that followed consistent processes, or 'laws'.  If Nature were not, then it would be un-knowable because what we thought we knew could always be excepted here, there, or anywhere.

Along with laws came notions that we now know as well-posed questions.  As we said yesterday, a well-posed question is one that is available for study by proper science.  Whether the Mona Lisa or pepperoni pizza are good are not well-posed questions.  Among many reasons, the answer differs for any person, and without any seriously known predictability, and they are personal judgments that, unlike laws of Nature, can change willy-nilly.

The fallibility of well-posed questions
But, as we noted yesterday, a well-posed question should have these sorts of properties:

1. It is structured so that a solution exists, at least in principle 
2. Such a solution should be unique 
3. The solution's behavior changes systematically with the initial conditions 
4. An appropriately sufficient set of variables should be adequately measured.

But there are ways perhaps to clarify some of the problems that we didn't mention.  For example, most statements in science rest upon definitions.  Definitions of terms for things in the world can be arbitrary--we choose them as humans, assuming they apply to reality.  But they need not be as applicable as they sound.  So a scientific question can be clear-sounding but in fact not well-posed:

Q1:  "What is the cause of type 2 diabetes?"

Sounds fine, and it is a simple sentence, but does it meet the criteria of a well-posed problem?  And is it answerable?

For starters, one must define what is meant by 'type 2 diabetes'. The implication is that it is a state of being (assuming we're discussing humans, or at least mammals).  But what 'state'?  One must specify more, or as the humanities-jargon would have it, 'unpack' the phrase.  Some measurement is implied, but what?  Or is it some outcome?  If we define 't2d' to be some level of blood glucose, measured in some specific way, then we might argue that Q1 is well-posed.  Indeed, we can answer it with a discussion of the biomechanism that causes high blood glucose.  But that's not helpful if what we really want to know is why that mechanism has gone off in some people.

And, often what happens is that the definition changes--diagnostic glucose cutoffs are changed, or the definition changes from glucose levels to some other trait, such as blood pressure, neuropathy, vision problems or some set of these purported glucose-dependent outcomes.  That is we can define the trait in terms of a risk state (glucose levels) or an outcome state (kidney failure).

Blood glucose meter; Wikipedia

Now the question, no matter how simple in structure, is not so clearly well-posed.  Even with a constant definition, the measure can change.  We measure not blood glucose but glycosylated hemoglobin, or we use a different measuring machine.  Then we face various data consistency problems, and certainly whether past data can be used any longer.

Or, we could say that 'diabetes' is the state when you have a particular genotype.  That may sound strange, but in a law-like universe isn't the genotype just a form of trait, and other things like glucose level or kidney failure just secondary, indirect measures of the  'true' trait?

Q2:  "Do factors f1, f2, f3, and f4 cause diabetes?"

This version sounds better because it's more specific.  Here, let's now forget the problems with defining diabetes, taking that for granted.  We just move the same issues towards f1--f4: defining them and how they are measured.  Now we have to consider whether the question means all these factors, or just 'some combination' of them.  If the latter, then the question is not well-posed until it is re-phrased to be clear about what 'combination' means.  Presence or absence?  Some quantitative measure on the factors?  If f1 alone never leads to diabetes, is it a 'cause'?

If we define terms clearly then one might assume that 'a' solution to the question does exist, at least in principle, at least for some specific data.  But if definitions can change then the solutions will change, having no bearing on Nature but only on human culture--the way we choose to make our definitions and measurements.  One thing that is clear is that most of the time questions like Q2 are not clearly posed.

What about uniqueness?  If the answer is not unique, then the law-like assumptions about Nature are somewhat strange.  Causes should have outcomes and if you measure carefully, shouldn't you get just one outcome?  Or one cause per outcome?  But we know that's not true (in the broadest sense, there are many ways to die, but there are also many paths to heart disease, or asthma and allergic response, and so forth.)  If many different causes have purely law-like properties, then Q1 is actually not a well-posed question, no matter how simple and clear it sounds.  Q2 might be better, for example, if each possible combination of the f's has a unique outcome; but if there is a factor f5 that is not being measured, the question will not have a unique answer.

For this reason, the escape from the idea of uniqueness is to say that many different causal situations each uniquely cause diabetes is correct in a sense, but then even Q2 is poorly posed.

Q3:  "Do factors f1, f2, f3, and f4 cause diabetes by altering its probability of occurring?"

This is a common escape-valve way to pose the question.  It's an escape valve because it introduces probabilities, which has to do with repetition of identical conditions, which never happens. But it allows almost anyone to collect almost any sort of data and 'answer' the question in a technical sense of getting an answer.  Probability is an ethereal concept but one that inevitably involves fractions of outcomes of a given type out of multiple outcomes, real or imagined.  Or it may mean something about the fundamental way that the factors act or interact.  One attractive attribute is that it can mean anything and you don't have to specify in clear terms.  Statistical analysis yields a test, but since the outcome may or may not occur, Q3 can always have an 'answer'.  If some statistical significance level is used, the answer is 'yes' or 'no', but such levels are subjective not objective.  And even if the true (but unknown) answer is 'no', we know from sampling theory that finding exactly zero effect is unlikely even if there is no actual causation involved.

So, criterion 3 for well-posed questions:  Does 'diabetes' change in some orderly way with changes in the test variables?  Often we can't really tell or we use some assumed patterning test to make statistical judgments (e,g., regression coefficients).  Again this requires many aspects of definitions and assumptions about the nature of replicability.  In particular, given what we know about genes and environmental factors, the mix of factors, the sampled contexts, is unique in each study or sample.  This means that we cannot actually replicate findings, and replication is one of the classical ways to check our answers to well-posed questions (or even to ill-posed questions).

Next?

In a law-like universe, we should be able to replicate exactly, at least in principle.  In the living world, we can't.  In the living world, replication is not part of evolution's deal--variation is.  In that sense, even non-replication doesn't invalidate a finding from a previous study.  So we're stuck to a great extent with approximate statistical answers to questions as we happen to pose them today, that we want to extend to the future but know that this depends on changes in the factors which we cannot in principle predict.

Statistical sampling methods and probabilistic thinking are the ways we address the ill-posed nature of many of our evolutionary and biomedical questions. Sometimes this works well enough to answer poorly-posed questions satisfactorily enough for our particular purposes.  With strong causal factors that are prevalent in the population of inference, things work and Enlightenment scientific criteria take us where we want to go.  In that case, we don't even need well-posed questions, or we don't care how they fail to measure up, because we get what we want.

But what we are facing along a broad front of evolutionary and biomedical (and agricultural) interests are poorly posed questions that are not giving us what we want.  What we tried to suggest yesterday, and we've discussed in many posts, is that what is needed are well-posed questions for which our methods and concepts of causality are apt, or questions that force us to develop appropriate methods and concepts of causality.

It's time for someone to fill in the blank:

Q4: