What's the point? Causation strong, weak, and undetectable

We often write and, yes, complain about the kinds of sample-based survey statistical studies of health risk, and this week's news about the cause of obesity is another good reason to revisit the subject.  (In case you missed it, we've got two options this week, it's either sugar or antibiotics.)  Whether the risk factor is genetic or environmental, the fields use similar conceptual approaches.  Essentially, they are 19th century concepts largely originating in physics but applied to society at the time, and now dressed in 20th century computational and ready-made statistical methods.

To drink, or not to drink. Is that a question?

Here's a wonderful send up of the problem on the BBC Radio 4 program, The News Quiz.  (Start at 13:05.)  "Remember the Mediterranean Diet?  The last time I was in the Mediterranean, we had kebabs and San Miguel!"  "One week we are told to drink red wine, and the next week, white."  And so on. When the inability of science to determine risk factors becomes the butt of jokes, it's no laughing matter. (H/T Tom Farsides.)

Or is this all white to drink?

When there is what is referred to as point causation, a single major factor with very high predictive power, the approach works well.  Somewhat different methods are and have been used for genetic causation (Mendelian inheritance methods) or infectious or some environmental disease causes, when a single exposure or an exposure to a single 'point' cause is responsible.   If the cause has strong effects, even probabilistic aspects of its action or appearance are well-analyzed by fundamental statistical ideas and methods. These depend on replication (basically, that each observation is a repeat of the others) and a high-level of what essentially amounts to causal determinism by the risk factor, even if there are probabilistic aspects involved (such as the chance of being exposed to a pathogen, or of inheriting a Mendelian disease variant).

A good and interesting survey of the way that probabilistic and statistical thinking entered western science is given by Ian Hacking's book The Taming of Chance (Cambridge University Press, 1990). Hacking retraces debates over whether causation was just apparently probabilistic or was fundamentally so, and how 19th century's versions of 'big data', in the form of national statistical survey data used for things like insurance rate determination, legitimized what was essentially the borrowing of methods from physics to apply them to society.

Because of the success of epidemiology during the infectious disease (pre-antibiotic) era, and in genetics the Mendelian segregation analysis era, the idea grew that we could, with sufficient samples, apply them essentially to all forms of causation.  That has led to today's obsession with Big Data and elaborate (often off-the-shelf) statistical methods, basically the same cookie-cutter approach to any question.

But there is a problem with that.  It's what we often write about here, because we think it is a serious one.  We think that for reasons of cultural and scientific inertia and lack of sufficiently imaginative training, we cling to these methods even though we have good theoretical as well as experiential reasons for knowing they are inappropriate--they basically don't work, but we do them anyway.

When there are many different factors at play, varying among samples or populations, and most of the factors are either of low prevalence (rare in the population) or weak effect, the gussied-up 19th century mode of thinking is just not very effective at analysis or prediction.  That is why, no matter how one may defend the continuation of big-investment, big-data, long-term, high-throughput investment, we are daily seeing the uncertain, changeable, non-definitive results reported in the news and in the journals themselves.  Questions like the risks associated per gram of sugar, or of our dietary intake of antibiotics in food, or of PSA testing or regular mammography, or GWAS and other claims of genes 'for' this or that trait, are changing daily with each study being reported by the authors and the media as if, somehow, it is now, finally, definitive.

To increase sample size and get results, without having to actually think about the problem or frame well-posed questions, researchers propose exhaustive national or international studies, high-throughput data analysis, meta-analysis and the like.  Of course, if you look for associations and use statistical tests of different kinds, you will always get a 'result'.  But among other things, the approaches basically assume a well-behaved signal-to-noise ratio, that is, that the bigger the sample the closer the estimates of risk will be to the (assumed) underlying truth with less 'noise' due to measurement and sampling errors. That assumes that as we add new samples, their heterogeneity will be less than their collective refinement of the risk estimates--less error in those estimates--assuming they have true values. We know very well and from very extensive experience that there are reasons why this assumption is unlikely to be true, that are borne out in the data, but the expediency of proposing just to collect more, more, more seems too convenient to abandon.

Likewise, the idea that risk factors and their relative frequencies will be the same in the future as in our samples from the past.  But we know that that is bunkum even from a genetic point of view (variants' frequencies depend heavily on samples chosen, new variants are introduced and others lost), as well as environmental (lifestyles change in ways that are inherently unpredictable, and predicting environmental effects on gene expression, perhaps of genes never before found to be involved in disease, is also equally impossible).

In the face of daily now you see it, now you don't stories, always hyped up by the professors, the funders, and the media, one might expect that the science would stop and take stock, realizing that we are driving a model-T on a modern highway and need to do differently.  But that seems just too difficult.  Like turning a tanker around in the ocean, it's a slow, cumbersome process.  And, to a great extent, we have not yet collectively agreed that we even have a problem (even though we know very well that we do).

Of course, it's at least possible that what we see is what there is: that there simply isn't a better way besides fickle statistical association analysis to understand the risks we want to understand.  At least, then, we should show that that is the case with some definitive evidence demonstrating that it's impossible to do better.  But even that would require more careful thought than is being given to these problems.

In truth can anyone not realize that a significant factor in the current way of doing business is that investigators know that fickle results are good business--that one never need say a study is over or a question answered, and can keep asking for more data, more computers, longer, larger studies?

Strong point causation is like a drug.  Its initial pleasures addict us to hope for the high of simple answers for every question.  But the truth, weak causation, has lured us into addiction.