Hot under the (epidemiological) collar

Blogs like this are venues for expressing views on the current scene, in our case, related to genetics, evolution and a few other things we throw in.  If you express a view, unless it's just plain vanilla, you will irritate some readers.  In a sense, if you don't then there's no point in writing the blogpost.  In this case, we heavily criticized the recent NYTimes article reporting that the government has now backed off its claim that dietary cholesterol is a heart disease risk factor.  We try to be responsible, but that doesn't mean we have to expect agreement or to mince words!

We argued that the kind of herky-jerky yes/no results from huge long-term megastudies are so common that this shows the studies are rather useless and we think should be phased out, the results to date archived for anyone who wants to mine them, and the funds put to something that actually generates more trustworthy and stable results (if risks are stable enough to be estimated in these ways).

Well, this generated a very heated message from an old friend, a prominent genetic epidemiologist, who said that if we were listened to, it would lead to throwing the baby out with the bathwater.  He was upset because he said it was not the data but the analysis of these big epidemiological (environmental or genetic) studies that was at fault.  The studies are based essentially on correlation or regression models that assume everyone starts out as an equal blank slate, and whose individual risk is the basically additive total of the various risk factor exposures.  Your sex gives you a 'dose' of risk, which age adds to, then smoking history, diet, and so on.  Once all your risk-factor measures are toted up, your net risk can be estimated.  It is this general approach that regardless of sophisticated details in the statistical method, is not good at finding what we really should be looking for.

Our friend's idea is that, for example, dietary cholesterol may on average not be harmful, but there are likely some subsets of the population for which it is a risk.  Standard models may be convenient to apply, and everyone knows how to do that....but they miss the boat.  The key problem is basically that risk factors interact and searching for complex interaction is rarely done because it is very demanding in terms of sample size, sample structure, and analytic tools.  But there is no reason, for example, to think that males and females respond to a given risk factor equally per exposure dose.  So interactions ('epistasis' in relation to genome elements)  are given very light treatment and basically wished away.

My irate friend basically argued that what is needed is not an end to the data but to the methods. 
One recent paper I was referred to applies application of a method for finding high-risk subsets, and has references to earlier descriptions of the methodology is this: Int J Epidemiol. ("Subgroups at high risk for ischaemic heart disease:identification and validation in 67 000 individuals from the general population", Frikke-Schmidt R et al."), 2015 Feb;44(1):117-28 (but unfortunately it is not freely available).

How effectively this will find really different subgroups is open.  We know, for example, that males and females are not at equal risk and respond differently to other factors, as mentioned above.  We know genomic components interact.  We know that as you get older you get closer to various risks, such as heart disease or cancer, and that the same exposure has different impacts with age, and so on.

The idea both in public health and in medicine (and in evolutionary inference) of identifying causation as effectively as possible, and that includes identifying high risk individuals as early as possible, is of course absolutely the right thing.  There are many instances of genetic risk factors like some variants in the gene responsible for cystic fibrosis, or in the BRCA1 gene related to breast cancer, where the Who Cares? principle applies: the single factor's effect is so predictably strong that one intervenes regardless of the details of how much risk is associated or what other factors might slightly modify the outcome.  We know that the nominal risk factor (e.g., a mutation) doesn't always lead to the same degree of severity, but the variation isn't enough to cause doubt: Who Cares about the details?

But whether in general this sort of method of searching for statistical associations can identify risk earlier enough, where preventive measures might be more helpful, is unclear.  We know about age and sex and smoking and so on, and maybe we don't really gain much from adjusting the exact values.  Or maybe we would.  Likewise for the complex interactions among hundreds of contributing genomic factors.  But there the number of factors and the assumption of independence and so on need to be recognized as being as daunting as they are, relative to statistical risk analysis.

In my view, this is still walking in dreamland.  The major factors will be identified, perhaps with more precision, but we face a huge, open-ended kind of 'multibody' problem.  It's just not possible to analyze all the possible combinations of factors and their interactions to get combination-specific risk estimates.  First, risks are contingent, one factor's effect depending on what else is present, as discussed above.  Second, not all combinations will show up in the data, even in huge samples, so estimating risks if interactions must be accounted for will simply come up short, or perhaps better-put, with unknown or even unknowable precision.

Third, we know very well even from just the recent few decades, that incidence of outcomes changes hugely, yet the genomes basically don't, and while we may estimate the effects of the particular combinations of environments our sampled individuals were exposed to, we simply cannot, even in principle, know what environmental factors current individuals for whom we are being promised precise predictions, will be exposed to.  Yet heritabilities, with all their problems, clearly show that genomes contribute typically far less than half of all risk.  Environmentally and genomically, specific factors or variants come and go, and no two people are identical or even close to it.

A major issue is not just that there is no way, that is no way to know what risks are associated with most risk factors, much less interactions among them, even in principle, but we have no way of knowing the degree of precision of predictions.   This is why, among other things, even if increasing our understanding is a very noble pursuit, promising 'precision' in prediction based on genomes or, really, almost any other risk factors, is irresponsible.

What's the point? Causation strong, weak, and undetectable

We often write and, yes, complain about the kinds of sample-based survey statistical studies of health risk, and this week's news about the cause of obesity is another good reason to revisit the subject.  (In case you missed it, we've got two options this week, it's either sugar or antibiotics.)  Whether the risk factor is genetic or environmental, the fields use similar conceptual approaches.  Essentially, they are 19th century concepts largely originating in physics but applied to society at the time, and now dressed in 20th century computational and ready-made statistical methods.

To drink, or not to drink. Is that a question?

Here's a wonderful send up of the problem on the BBC Radio 4 program, The News Quiz.  (Start at 13:05.)  "Remember the Mediterranean Diet?  The last time I was in the Mediterranean, we had kebabs and San Miguel!"  "One week we are told to drink red wine, and the next week, white."  And so on. When the inability of science to determine risk factors becomes the butt of jokes, it's no laughing matter. (H/T Tom Farsides.)

Or is this all white to drink?

When there is what is referred to as point causation, a single major factor with very high predictive power, the approach works well.  Somewhat different methods are and have been used for genetic causation (Mendelian inheritance methods) or infectious or some environmental disease causes, when a single exposure or an exposure to a single 'point' cause is responsible.   If the cause has strong effects, even probabilistic aspects of its action or appearance are well-analyzed by fundamental statistical ideas and methods. These depend on replication (basically, that each observation is a repeat of the others) and a high-level of what essentially amounts to causal determinism by the risk factor, even if there are probabilistic aspects involved (such as the chance of being exposed to a pathogen, or of inheriting a Mendelian disease variant).

A good and interesting survey of the way that probabilistic and statistical thinking entered western science is given by Ian Hacking's book The Taming of Chance (Cambridge University Press, 1990). Hacking retraces debates over whether causation was just apparently probabilistic or was fundamentally so, and how 19th century's versions of 'big data', in the form of national statistical survey data used for things like insurance rate determination, legitimized what was essentially the borrowing of methods from physics to apply them to society.

Because of the success of epidemiology during the infectious disease (pre-antibiotic) era, and in genetics the Mendelian segregation analysis era, the idea grew that we could, with sufficient samples, apply them essentially to all forms of causation.  That has led to today's obsession with Big Data and elaborate (often off-the-shelf) statistical methods, basically the same cookie-cutter approach to any question.

But there is a problem with that.  It's what we often write about here, because we think it is a serious one.  We think that for reasons of cultural and scientific inertia and lack of sufficiently imaginative training, we cling to these methods even though we have good theoretical as well as experiential reasons for knowing they are inappropriate--they basically don't work, but we do them anyway.

When there are many different factors at play, varying among samples or populations, and most of the factors are either of low prevalence (rare in the population) or weak effect, the gussied-up 19th century mode of thinking is just not very effective at analysis or prediction.  That is why, no matter how one may defend the continuation of big-investment, big-data, long-term, high-throughput investment, we are daily seeing the uncertain, changeable, non-definitive results reported in the news and in the journals themselves.  Questions like the risks associated per gram of sugar, or of our dietary intake of antibiotics in food, or of PSA testing or regular mammography, or GWAS and other claims of genes 'for' this or that trait, are changing daily with each study being reported by the authors and the media as if, somehow, it is now, finally, definitive.

To increase sample size and get results, without having to actually think about the problem or frame well-posed questions, researchers propose exhaustive national or international studies, high-throughput data analysis, meta-analysis and the like.  Of course, if you look for associations and use statistical tests of different kinds, you will always get a 'result'.  But among other things, the approaches basically assume a well-behaved signal-to-noise ratio, that is, that the bigger the sample the closer the estimates of risk will be to the (assumed) underlying truth with less 'noise' due to measurement and sampling errors. That assumes that as we add new samples, their heterogeneity will be less than their collective refinement of the risk estimates--less error in those estimates--assuming they have true values. We know very well and from very extensive experience that there are reasons why this assumption is unlikely to be true, that are borne out in the data, but the expediency of proposing just to collect more, more, more seems too convenient to abandon.

Likewise, the idea that risk factors and their relative frequencies will be the same in the future as in our samples from the past.  But we know that that is bunkum even from a genetic point of view (variants' frequencies depend heavily on samples chosen, new variants are introduced and others lost), as well as environmental (lifestyles change in ways that are inherently unpredictable, and predicting environmental effects on gene expression, perhaps of genes never before found to be involved in disease, is also equally impossible).

In the face of daily now you see it, now you don't stories, always hyped up by the professors, the funders, and the media, one might expect that the science would stop and take stock, realizing that we are driving a model-T on a modern highway and need to do differently.  But that seems just too difficult.  Like turning a tanker around in the ocean, it's a slow, cumbersome process.  And, to a great extent, we have not yet collectively agreed that we even have a problem (even though we know very well that we do).

Of course, it's at least possible that what we see is what there is: that there simply isn't a better way besides fickle statistical association analysis to understand the risks we want to understand.  At least, then, we should show that that is the case with some definitive evidence demonstrating that it's impossible to do better.  But even that would require more careful thought than is being given to these problems.

In truth can anyone not realize that a significant factor in the current way of doing business is that investigators know that fickle results are good business--that one never need say a study is over or a question answered, and can keep asking for more data, more computers, longer, larger studies?

Strong point causation is like a drug.  Its initial pleasures addict us to hope for the high of simple answers for every question.  But the truth, weak causation, has lured us into addiction.