Showing posts with label probability. Show all posts
Showing posts with label probability. Show all posts

Wednesday, August 15, 2018

On the 'probability' of rain (or disease): does it make sense?

We typically bandy the word probability around, as if we actually understand it. The term, or a variant of it like probably, can be used in all sorts of contexts that, on the surface seem quite obvious and related to some sense of uncertainty; e.g., "That's probably true," or "Probably not."  But is it so obvious?  Are the concepts clear at all?  When are they, actually, more than just informally and subjectively, meaningful?

Will it rain today?  Might it?  What is the chance of rain?
One of the typical uses of probabilistic terms in daily life has to do with weather predictions.  As a former meteorologist myself, I find this a cogent context in which to muse about these terms, but with extensions that have much deeper relevance.

Here is an episode of a generally very fine BBC Radio 4 program called More or Less, whose mission is to educate listeners on the proper use and understanding of numbers, statistics, probabilities and the like.  This episode deals, somewhat unclearly and to me quite vaguely, unsatisfactorily, and even somewhat defensively, about the use and interpretation of weather forecasts.

So what does a forecast calling for an x% chance of rain mean?  Let's think of an imaginary chessboard laid over a particular location.  It is raining under the black, but not under the white squares.  There is nothing probabilistic about this.  50% of people in the area will experience rain.  If I don't know where you live, exactly, I'd have to say that you have a 50% chance of rain, but that has nothing to do with the weather itself but rather with my uncertainty of where you live.  Even then it's misleadingly vague since people don't live randomly across a region (they are, for example, usually clustered in some sub-regions).

Another interpretation is that I don't know where the black and white squares will be exactly, at any given time, but my weather models predict that in about half of the region, rain will fall.  This could be because my computer models, necessarily based on imperfect measurement and imperfect theory, are therefore imperfect--but I run them many times, making small random changes in various values to account for that imperfection, and I find that among these model runs, 50% of the time at any given spot, or 50% of the entire area under consideration, experiences rain.

Or, is it that there is an imaginary chessboard moving overhead and so the 50% of the land will be under the black and hence getting rain at any given time, and thus that any given area will only get it 50% of the time, but every area will certainly get rain at some time during the forecast period, indeed every area will be getting rain half of the period?  Then the best forecast is that you will get wet if you stay outside all day, but if you only run out to get the mail you might not?  Might??

Or is it that my models are imperfect but theory or experience tell me that there is a 50% chance of any rain in the area--that is, my knowledge can tell me no more than that.  In that case, any given place will have this guesstimated chance of rain.  But does that mean at any given time during the forecast period, or at every time during it?  Or is it that my knowledge is very good, but the meteorological factors--the nature of atmospheric motion and so on--only probabilistically form droplets that are large enough not just to be clouds but to fall to earth?  That is, is it the atmospheric process itself that is probabilistic--at least based on the theory, since I can't observe every droplet.

If a rain-generating front is passing through the area, it could rain everywhere along the front, but only until the front has moved past the area.  Thus, it may rain with 100% certainty, but only 50% of the specified time, if the front takes that amount of time to pass through.

I've undoubtedly only mentioned some of the many ways that weather forecast probabilities can be intended or interpreted as meaning.  It is not clear--and the BBC program shows this--that everyone or perhaps even anyone making them actually understands, or is thinking clearly about, what these probability forecasts mean.  Even meteorologists themselves, especially when dumbing down for the average Joe who only wants to know if he should carry his brolly with him, are likely ('probably'?!) unclear about these values.  Probably they mean a bit of this and a bit of that.  I wonder if anyone can know which of the meanings are being used in any given forecast.

Well, fine, everyone knows that nobody really knows everything about the weather.  Anyway, it's not that big of a deal if you get an unexpected drenching now and then, or more often haul your raincoat to work but never need it.

But what about things that really matter, like your future health?  My doc takes my blood pressure and looks at my weight, and may warn me that I am at 'risk' of a heart attack or stroke--that without taking some preventive measures I may (or probably will) have such a fate.  That's a lot more important than a soaked shirt.  But what does it mean?  Isn't everybody at some risk of these diseases?Does my doc actually know?  Does anybody?  Who is thinking clearly about these kinds of risk pronouncements?

OK, caveats, caveats: but will I get diabetes?
In genomics 'precision' genomic medicine is one of the genomics marketing slogans of the day, the very vague (I would say culpably false) promise that from your genotype we can predict your future--that's what 'precision' implies.  The same applies even if weaseling now would include environmental factors as well as genomic ones.  And the idea implies knowledge not just of some vague probability, but by implication it means perfection--prediction with certainty.  But to what extent--if any at all--is the promise, or can the promise be true?  What would it mean to be 'true'?  After all, anyone might get, say type 2 diabetes, mightn't they?  Or, more specifically, what does such a sentence itself even mean, if anything?

We know that, today at least, some people get diabetes sometime in their lives, and even if we don't know why or which ones, that seems like a safe assertion.  But to say that any person, not specifically identified, might become diabetic is rather useless.  We want a reason--a cause--and if we have that we assume it will enable us to identify specifically vulnerable individuals.  Even then, however, we don't know more than to say, in some sense that we may not even understand as well as we think we do, that not all the vulnerable will get the disease: but we seem to think that they share some probability of getting it.  But what does that mean, and how do we get such figures?

Does it mean that among all those with a given GWAS! genotype, (1) a fraction f will get diabetes?(2) a fraction f will get diabetes if they live beyond some specified age? (3) a fraction f will get diabetes before they die if they live the same lifestyle diet as those from whom the risk was estimated? (4) a net fraction f will get diabetes, pro-rated year by year as they age; (5) a net fraction related to f will get diabetes, but that is adjusted for current age, sex, race, etc.?

What about each individual consulting their Big Data genomic counselor?  Are these fractions f related to each individual as a probability p=f that s/he will get diabetes (conditional on things like items 1-5 above)?  That is, is every person at the same risk?

Only if we can equate our past sample, from which we estimated f by induction to the probability p used by deduction to assert for each new individual might this, even in principle, lead to 'precision genomic medicine'.  It is prediction, not just description that we are being promised.  Even if we were thinking in public health terms, this is essentially the same, because it would relate to the fraction of individuals who will be affected in the future, because each person is exposed to the same probability.

Of course, we might believe that each person has some unique probability of getting diabetes (related, again, to the above items), and that f reflects the mix (e.g., average) of these probabilities.  But then, we have to assume that all the genotypes and lifestyles and so on in the current group whose future we're offering 'precision' predictions is exactly like the sample from which the predictions were derived, that this mix of risks is, somehow, conserved.  How can such an assumption ever be justified?

Of course, we know very well that no current sample whose future we want to be precise about will be exactly the same as the past sample from which the probabilities (or fractions) were derived.  Obviously, much will differ, but we also know that we simply have no way to assess by how much it will differ.  For example, future diets, sociopolitical, and other factors that affect risk will not be the same as those in the past, and are inherently unpredictable.  So, on what meaningful basis can 'precision' prediction be promised?

Just for fun, let's take the promise of precision genomic medicine at its face value.  I go to the doc, who tells me
"Based on your genome sequence, I must advise you of your fate in regard to diabetes."
"Thanks, doc.  Fire away!"
"You have a 23.5% chance of getting the disease."
"Wow!  That sounds high!  That means I have a 23.5% chance that I won't die in a car or plane crash, right?  That's very comforting.  And if about 10% of people get cancer, then of my 76.5% chance of not getting diabetes, it means only a 7.65% chance of cancer!  Again, wow!"
"But wait, Doc!  Hold on a minute.  I might get diabetes and cancer, right?  About a 7.65% percent chance of that, right?"
"Um, well, um, it doesn't work quite that way [to himself, sotto voce: "at least I think so..."].....that's because you might die of diabetes, so you wouldn't get cancer.  Of course, the cancer could come first, but it would linger, because you have to live long enough to experience your 23.5% risk of diabetes.  That would not be good news.  And, of course, you could get diabetes and then get in a crash.  I said get diabetes, not die of it, after all!"
I gather you, too, can imagine how to construct many different sorts of fantasy conversations like this, even rashly assuming that your doctor understood probability, had read his New England Journal regularly when not too sleepy after a day's work at the clinic--and that the article in the NEJM was actually accurate.  And that NIH knew in sincerity what they were promising in the way of genomic predictability promises.  But wait!  The medical journals, and even the online genotyping scam companies--you can probably name one or two of them--change your estimated risks from time to time as new 'data' come in.  So when can I assume case-closed and I (well, the Doc) really knows the true probabilities?

I mean, what if there are no such true probabilities, because even if there were, not just knowledge, but also circumstances (cultural, not to mention mutations) continually change, and what if we have no way whatever to know how they're gonna change?  Then what is the use of these 'precision' predictions?  They, at best, only apply to a single, current instance.  So what (if anything at all) does 'precision' mean?

It only takes a tad of thinking to see how precisely imprecise these promises all are--must be, except very short-term extrapolations of what past data showed, and extrapolations of unknown (and unknowable) 'precision'.  Except, of course, the very precise truth that you, as a taxpayer, are going to foot the bill for a whole lot more of this sort of promises.

Unlike the weather, we don't have anything close to as rigorous an understanding of human biology and cultures as we do of the behavior of gases and fluids (the atmosphere).  We might want to say, self-protectingly and more honestly modest, that our use of 'probability' is very subjective and really just means an extrapolated rough average of some unspecifiable sort.  But then that doesn't sound like the glowing promise of 'precision', does it?  One has to wonder what sort of advice would make scientifically proper, and honorable, use of the kind of probabilistic, vague, ephemeral evidence we have when we rely on 'omics approaches, or even when it's the best we can do at present.

In meteorology, it used to be (when I was playing that game) that we'd joke "persistence is the best forecast".  This was, of course, for short range, but short range was all we could do with any sort of 'precision'.  We are pretty much in that situation now, in regard to genomics and health.

The difference is, weather forecasters are honest, and admit what they don't know.

Thursday, July 23, 2015

Heart disease - the 7.5% solution?

Statins are in the news again, and not just because of the new PCSK9-based drugs, at least one of which is likely to be approved by the FDA this week, probably for a small class of at-risk patients.  These drugs will drive LDL cholesterol levels through the floor, while generating an estimated 17.8 billion for pharma by the year 2023 (and that's before we even know whether they will reduce risk of heart attack and stroke).

No, this is about your run-of-the-mill class of LDL-lowering statins.  In late 2013, the American Heart Association and the American College of Cardiology recommended new guidelines for determining who should be on statins.
  • anyone who has cardiovascular disease, including angina (chest pain with exercise or stress), a previous heart attack or stroke, or other related conditions
  • anyone with a very high level of harmful LDL cholesterol (generally an LDL above greater than 190 milligrams per deciliter of blood [mg/dL])
  • anyone with diabetes between the ages of 40 and 75 years
  • anyone with a greater than 7.5% chance of having a heart attack or stroke or developing other form of cardiovascular disease in the next 10 years.
Risk score is based on the ASCVD calculator, which uses basic data (age, sex, total cholesterol, HDL cholesterol, systolic blood pressure, and smoking and diabetes status) to calculate risk.  Unlike the previous Adult Treatment Panel III (ATP III) guidelines which were based on a target LDL level and risk factors determined by the long-term Framingham heart disease study (using the Framingham Risk Calculator), these new guidelines were based on a risk profile.  With these new guidelines, it was thought that about 13 million additional Americans would benefit from statins, for a total of a third of all Americans.

A study published in the Journal of the American Medical Association last week asks whether these guidelines were better at identifying at-risk individuals than the old ATP III guidelines.  The prospective study followed up 2435 people from the Framingham study who had never taken statins.  Based on the ATP III guidelines, 14% would have been 'eligible' compared with 39%, based on the 2013 guidelines.
The median follow-up was 9.4 (interquartile range, 8.1-10.1) years. There were a total of 74 (3.0%) incident CVD events (40 nonfatal myocardial infarctions, 31 nonfatal strokes, and 3 with fatal CHD) and 43 (1.8%) incident CHD events (40 nonfatal myocardial infarctions and 3 with fatal CHD).
Among those eligible for statin treatment by the ATP III guidelines, 6.9% (24/348) developed incident CVD compared with 2.4% (50/2087) among noneligible participants (HR, 3.1; 95% CI, 1.9-5.0; P less than .001). Applying the ACC/AHA guidelines, among those eligible for statin treatment, 6.3% (59/941) developed incident CVD compared with only 1.0% (15/1494) among those not eligible (HR, 6.8; 95% CI, 3.8-11.9; P less than .001). Therefore, the HR of having incident CVD among statin-eligible vs noneligible participants was significantly higher when applying the ACC/AHA guidelines’ statin eligibility criteria compared with the ATP III guidelines (P less than .001).
That is, according to this study, the 2013 ACC/AHA guidelines identified more people at risk of heart disease than the ATP III guidelines.  That's presumably progress in understanding heart disease risk, and so a good thing. (Does anyone else find the use of the word 'eligible' odd, though?  Like statins are a reward for passing the risk threshold?)

But why don't they ask about family history?  That is one of the most useful bits of data a physician can have about a patient's risk of heart disease (and other things).  Is it too cynical to suggest that acknowledging its usefulness might diminish the importance of what has been learned from the Framingham study?

Less cynically, one reason, though we don't know if the various investigators considered it in this way, is that family history integrates all factors, including those that are being specifically measured (like blood pressure, LDL levels, and so on). Whether they are genetic or environmental, they went into determining whether the relative had heart disease.  So counting family history and LDL, or for that matter, weight and BMI, also not included, may be redundant to an unknown extent.  For risk factors, this would perhaps inflate the apparent risk, but for protective factors the opposite.  But family history is debatably the best single factor, perhaps as important as all the test-battery factors.  At least, it's important to consider why that alone, or that somehow corrected for redundancy, should be a part of all of this.

So, apparently we don't know more about the causes of ASCVD now than we did before 2013, we're just evaluating what we know differently.  So, assuming that statins really do reduce risk of ASCVD, that more people are 'eligible' is thought to be a good thing.  Though, as the JAMA commentary on this article notes in urging increased treatment with statins, "Although a 10-year ASCVD risk threshold of 7.5% or higher might initially seem to be a low threshold, many, indeed most, CVD events occur among the low-risk members of the population."

Wait!  "Low-risk" is defined by us, based on what we know about heart disease!  Our understanding is clearly wrong if all these 'low-risk' people are really high-risk!  Not to mention that there's clearly a huge false-positive pool if a risk estimate of 7.5 out of 100 makes a person eligible for statins!  That means that 92.5 of those 100 people are taking statins even though they weren't going to have a stroke or heart attack.  And, all this means, at least to me, that we really don't understand what causes heart attacks or stroke. The Framingham study identified cholesterol, particularly LDL, as a risk factor, but we're not really sure why, and we don't know what levels are in fact most risky, and people with low LDL can have heart attacks, too.  Statins may or may not work by reducing LDL cholesterol, and lower LDL cholesterol may or may not reduce risk.

And, statins can have serious side effects -- physical as well as the cost burden.  So, if of 100 people taking statins a large majority weren't going to have heart attacks anyway, statins are causing a lot of unnecessary side effects without preventing disease.  Though, to be fair, physicians can't predict the future, and must do their best with the information they have.  They don't know who will or won't have a heart attack, because epidemiology hasn't given them enough information.  They've got to treat people with 7.5% risk as if they are at 100% risk of disease.

So, it's not physicians who are failing here, it's epidemiologists.  But I'll even be fair to epidemiologists -- it's the methods, based on population data and probability (which may not even exist; see our series of posts on this starting here), that are failing.  Epidemiologists are doing their best with what they've got.  We don't know precisely what causes heart attacks, but to prevent them, we've got to treat people with low risk as though they are at high risk, and that's because some people at low risk really are at high risk.

No one has 7.5% of a heart attack.  They have 0% or 100% of a heart attack. Figuring out who is in which group is currently impossible.  What we do know for certain is that putting everyone on statins, as though they have 100% risk is very good for the pharmaceutical companies that make them, and good for people whose heart attack or stroke was prevented, even if we will never know which people these were, and unnecessary and even harmful for everyone else.

This is a lousy way to do medicine.  But it's currently the only way we've got.

Monday, February 16, 2015

Occasionality (vs Probability)?

In genetics and epidemiology, we want to explain the causes of diseases, and we want to be able to predict who will get them. We generally do this statistically, by working out the probability that we are right in our guesses about alleles or environmental causal factors.  But this is problematic for a number of reasons.

The concept of probability and the term itself have always been elusive or variable in meaning.  But they refer to collections of objects and the relative proportions of some specified component, or alternatively to the distribution of outcomes of repeated identical trials.  They work great for truly repeatable phenomena such as flipping coins or rolling dice or sampling colored balls from the proverbial urn (actually, there are problems even here, of both practical and theoretical sorts, but they are minor relative to the point we wish to make).

Dice; Wikipedia


In discussions of the phenomena pertaining to evolutionary genetics, 'probability' arises all the time.  We have genetic drift--the 'probability' of change in frequency of a given allele in a population of specified size; natural selection and evolutionary fitness--the relative probability of the bearer of a given genotype to reproduce relative to the probability of alternative genotypes (adaptive fitness); the probability of migration and hence gene flow, and much more.

The same sorts of things arise in biomedical genetics: Mendelian inheritance--the probability of a given allele being transmitted to a given offspring; the relative probability of a disease arising in a person of a given genotype; the probability of lung cancer in a smoker (per cigarette consumed, etc.); the probability of having a given stature or blood pressure in a person of a given genotype, and much more.

Such things are as routinely discussed as the daily news, or what some guest said on The Daily Show. But there is a problem.  The meaning of the term is unclear, often to the user as well as hearer. The term is deeply built into many areas of evolutionary and causative genetics, our means of drawing conclusions from both observational and experimental data.  But upon closer scrutiny, the term is  being used based on highly questionable or hopeful assumptions that at best only apply to a subset of situations.

Replicability and probability: p-eeing into the wind
The standard way to represent a probability is by a symbol, often the letter p.  Once you have identified this value, it is easy to write it into various equations, like, for example, the probability of observing 3 Heads in a row is p^3 (p cubed) where p is the probability of Heads on any given flip. Or, if you assume that your data involve sampling from the same urn filled with certain proportion of red and blue balls (or, say, of diabetics or tall people in a sample from a population of people with a given genotype) and the probability of getting your result just by chance (that is, no genetic causation involved),  p is less than 0.05, then you declare to the world that something unusual is going on--that is, your research has found something (such as that the genotype is a causal factor for diabetes or tall stature).  But if your actual set-up is not something simple like balls in an urn, or dice—that sort of thing, then what is it that is really happening?  How reliable or interpretable are these probabilities?

The assumed sort of regular repeatable condition, in the real evolutionary and biomedical world where we live, is usually a fiction!  That doesn't mean there is no truth involved.  It means that we think there is some truth in how we've organized our description of what's going on, and in a somewhat circular way we're using our data to check that.  Unfortunately, we usually don't know even what we think is going on in a rigorous enough way for the p's we employ to be accurate, or to put it more fairly, they are inaccurate to an unknown extent.  Is that extent even knowable in principle?

In the areas of biology and health discussed here, we are usually far from having such knowledge. We use 'p' values or their statistical equivalents all the time, both for the processes (e.g., risk of some disease or outcome) or unusualness in support of our causal assertion (e.g., statistical significance).

In most or perhaps every case, we are making strong underlying assumptions about the real world that is our objective to understand.  That assumption is one of replicability--that is, that we are dealing with repeatable events, often that there is some ‘random’ aspect of causation (itself usually a vague assumption not seriously explained).  That in turn implies that there truly exists some p-values or probability distributions underlying what we are studying.  Repeatability is a basic assumption, what one might even call a tenet about the nature of the organization of the world we are studying.  

In what sense do such distributions or probabilities actually exist?  Is the world of our interest organized, symmetric, and repeatable enough for the models being tested to be realistic? If the cosmos were purely deterministic (e.g., no truly probabilistic quantum phenomena), then we would always be sampling from worlds of proportions and our probability values would reflect repeated sampling events ('random' assuming we really knew what that meant in practice).

But suppose to the contrary that to an important extent each event is unique, even if perhaps wholly predictable if had we enough information, and that it is literally impossible for us to be both in the world and have enough information about the world in order to make such predictions.  Perhaps either truly, or pragmatically, the 'probabilities' on which so much of what we are about in modern genetics, and, of course, the promises being made, simply don't exist.  To the extent that's true, the edifice of probabilistic science and statistical inference are mistaken or misleading, to an unknown and probably unknowable extent.

Occasionality: a more appropriate alternative concept--where there's no oh!
When many factors contribute to some measured event, and these are either not all known or measured, or in non-repeatable combinations, or not all always present, so that each instance of the event is due to unique context-dependent combination, we can say that it is an ‘occasional’ result.  In the usual parlance, the event occasionally happens and each time the conditions may or may not be similar.  This is occasionality rather than probability, and there may not be any 'o-value' that we can assign to the event.

This is, in fact, what we see.  Of course, regular processes occur all around us, and our event will involve some regular processes, just not in a way to which probability values can be assigned.  That is, the occasionality of an event is not an invocation of mystic or immaterial causation.  The word merely indicates that instances of the event are individually unique to an extent not appropriately measured, or not measured with knowable accuracy or approximation, by probabilistic statistical (or tractable deterministic) approaches.  The assumption that the observations reflect an underlying repeated or repeatable process is inaccurate to an extent as to undermine the idea of estimation and prediction upon which statistical and probabilistic concepts are based.  The extent of that inaccuracy is itself unknown or even unknowable.

There are clearly genetic causal events that are identifiable and, while imperfect because of measurement noise and other unmeasured factors, sufficiently repeatable for practical understanding in the usual way and even treated with standard probability concepts.  Some variants in the CFTR gene and cystic fibrosis fall into that category.  Enough is known of the function of the gene and hence of the causal mechanism of the known allele that screening or interventions need not take into account other contextual factors that may contribute to pathogenesis but in dismissible minor ways. But this seems to be the exception rather than the rule.  Based on present knowledge, I would suggest that that rule is occasionality.

One problem clearly is definitional.  Where causation has traditionally been assigned to some specific outcome, we now realize that outcomes, like causes, constitute a spectrum or syndrome of effects, so that we have as much variation in 'the' outcome as in the input variables.  Clearly things are not working!

Occasionality can be vaguely viewed as something like a rubbery lattice of non-rigid interacting factors, with some connections and factors (nodes) missing or added from time to time without known (or knowable?) causal processes (e.g., what new mutations might lead to part of the genome to contribute to some spectrum of phenotype outcomes?), connecting rungs being broken or new ones added as the lattice jostles down a roiling stream in and around obstacles.  Particular vortices, or outcomes, a set of vertices (traits) that may or may not bob, occasionally, up to the surface individually or in different combinations to be measured as 'outcomes'.  This is at least an attempt at a heuristic way to envision the elusive fish the biological community are trying to land.

Metaphysics and physics envy
It has often been observed that biology's love affair with the kinds of mathematical/statistical approaches being taken is a form of physics envy.  It's an aspect of the belief system that pervades both evolutionary and genetic thinking.  It avoids the somewhat metaphysical reality, that we know life involves emergent properties that are not enumerable in ways ordinary physical and chemical systems seem (at least to outsiders like us, and at the macro-scale) to involve.  If what is wrong or missing (if anything) were clear, these areas of the life sciences would be doing things differently.

This may sound far too vague and airy, but the evidence is all around us.  Last week, as we wrote about here, even 40 or more years of nutritional studies, very sophisticated and extensive, are reported to have been 'wrong' in regard to the risk of dietary cholesterol, and salt consumption.  'Wrong' means not supported by recent research.  The nature of even of blood-levels of many factors like these also seems ephemeral, that is, findings  of one study are contradicted by another study.  Which (if any) are to be believed?  Why do we think our new studies or methods obsolesce the older ones, when in essence there aren't very substantial differences?  Why haven't we got solid, replicable, clear-cut causal understanding?

The difference between occasionality and probability is somewhat like the difference between continuous and discontinuous (quantitative vs qualitative) variation.  The challenge is to understand and make inferences about the latter when we don’t have underlying analytic (mathematical function based) theory to apply.  We don’t want to say an effect has no cause (unless we are involved in quantum mechanics, and even that's debated!), but we do need to understand unique combinatorial causation in ways we currently don’t.  This will have to be mixed with the statistical (probabilistic) aspects we can't avoid, such as sampling and measurement effects.

There are many reasons to dismiss such a term or concept as occasionality.  First, because it is not operationalized: we don’t really know what to do about it.  Second, there is a tendency to reject some new term that is invoked in a critical vein.  Thirdly, the practical reason is that even if the idea is correct, if it were acknowledged too fully it would undermine the business that most in this area of science are about, and certainly would be resisted.  But if we were to force ourselves to ask what we would do if we knew the phenomena we want to understand were manifestations of occasionality rather than probability, it might force us to think and act differently.

Tuesday, November 25, 2014

Let's Abandon Significance Tests

We thought we'd re-run the first blog post Jim Wood wrote (or read), from May 2013, on significance testing.  It was an excellent post the first time, and it's an excellent post again, with a message that doesn't get old.


By Jim Wood

It’s time we killed off NHST.


Ronald Aylmer Fisher
   (1890-1962)



 

NHST (also derisively called “the intro to stats method”) stands for Null Hypothesis Significance Testing, sometimes known as the Neyman-Pearson (N-P) approach after its inventors, Jerzy Neyman and Egon Pearson (son of the famous Karl). There is also an earlier, looser, slightly less vexing version called the Fisherian approach (after the even more famous R. A. Fisher), but most researchers seem to adopt the N-P form of NHST, at least implicitly – or rather some strange and logically incoherent hybrid of the two approaches. Whichever you prefer, they both have very weak philosophical credentials, and a growing number of statisticians, as well as working scientists who care about epistemology, are calling – loudly and frequently – for their abandonment. Nonetheless, whenever my colleagues and I submit a manuscript or grant proposal that says we’re not going to do significance tests – and for the following principled reasons ­­– we always get at least one reviewer or editor telling us that we’re not doing real science. The demand by scientific journals for “significant” results has led over time to a substantial publication bias in favor of Type I errors, resulting in a literature that one statistician has called a “junkyard” of unwarranted conclusions (Longford, 2005).


Jerzy Neyman
(1894-1981)
Let me start this critique by taking the N-P framework on faith. We want to test some theoretical model. To do so, we need to translate it into a statistical hypothesis, even if the model doesn’t really lend itself to hypothesis formulation (as, I would argue, is often the case in population biology, including population genetics and demography). Typically, the hypothesis says that some predictor variable of theoretical interest (the so-called “independent” variable) has an effect on some outcome variable (the “dependent” variable) of equal interest. To test this proposition we posit a null hypothesis of no effect, to which our preferred hypothesis is an alternative – sometimes the alternative, but not necessarily. We want to test the null hypothesis against some data; more precisely, we want to compute the probability that the data (or data even less consistent with the null) could have been observed in a random sample of a given size if the null hypothesis were true. (Never mind whether anyone in his or her right mind would believe in the null hypothesis in the first place or, if pressed on the matter, would argue that it was worth testing on its own merits.)   

Egon Pearson
(1895-1980)
Now we presumably have in hand a batch of data from a simple random sample drawn from a comprehensive sample frame – i.e. from a completely-known and well-characterized population (these latter stipulations are important and I return to them below). Before we do the test, we need to make two decisions that are absolutely essential to the N-P approach. First, we need to preset a so-called Î± value for the largest probability of making a Type I error (rejecting the null when it’s true) that we’re willing to consider consistent with a rejection of the null. Although we can set Î± at any value we please, we almost inevitably choose 0.05 or 0.01 or (if we’re really cautious) 0.001 or (if we’re happy-go-lucky) maybe 0.10. Why one of these values? Because we have five fingers – or ten if we count both hands. It really doesn’t go any deeper than that. Let’s face it, we choose one of these values because other people would look at us funny if we didn’t. If we choose Î± = 0.10, a lot of them will look at us funny anyway.

Suppose, then, we set Î± = 0.05, the usual crowd-pleaser. The next decision we have to make is to set a Î² value for the largest probability of committing a Type II error (accepting the null when it’s not true) that we can tolerate. The quantity (1 – Î²) is known as the power of the test, conventionally interpreted as the likelihood of rejecting a false null given the size of our sample and our preselected value of Î±. (By the way, don’t worry if you neglect to preset Î² because, heck, almost no one else bothers to – so it must not matter, right?) Now make some assumptions about how the variables are distributed in the population, e.g. that they’re normal random variates, and you’re ready to go.

So we do our test and we get p = 0.06 for the predictor variable we’re interested in. Damn. According to the iron law of Î± = 0.05 as laid down by Neyman and Pearson, we must accept the null hypothesis and reject any alternative, including our beloved one – which basically means that this paper is not going to get published. Or suppose we happen to get p = 0.04. Ha! We beat a, we get to reject the null, and that allows us to claim that the data support the alternative, i.e. the hypothesis we liked in the first place. We have achieved statistical significance! Why? Because God loves 0.04 and hates 0.06, two numbers that might otherwise seem to be very nearly indistinguishable from each other. So let’s go ahead and write up a triumphant manuscript for publication. 
Significance is a useful means toward personal ends in the advance of science – status and widely distributed publications, a big laboratory, a staff of research assistants, a reduction in teaching load, a better salary, the finer wines of Bordeaux…. [S]tatistical significance is the way to achieve these. Design experiment. Then calculate statistical significance. Publish articles showing “significant” results. Enjoy promotion. (Ziliak and McCloskey, 2008: 32)
This account may sound like a crude and cynical caricature, but it’s not – it’s the Neyman-Pearson version of NHST. The Fisherian version gives you a bit more leeway in how to interpret p, but it was Fisher who first suggested p less than or equal to  0.05 as a universal standard of truth. Either way, it is the established and largely unquestioned basis for assessing hypothesized effects. A p of 0.05 (or whatever value of a you prefer) divides the universe into truth or falsehood. It is the main – usually the sole – criterion for evaluating scientific hypotheses throughout most of the biological and behavioral sciences. 

What are we to make of this logic? First and most obviously, there is the strange practice of using a fixed, inflexible, and totally arbitrary a value such as 0.05 to answer any kind of interesting scientific question. To my mind, 0.051 and 0.049 (for example) are pretty much identical – at least I have no idea how to make sense of such a tiny difference in probabilities. And yet one value leads us to accept one version of reality and the other an entirely different one.

To quote Kempthorne (1971: 490):
To turn to the case of using accept-reject rules for the evaluation of data, … it seems clear that it is not possible to choose an a beforehand. To do so, for example, at 0.05 leads to all the doubts that most scientists feel. One is led to the untenable position that one’s conclusion is of one nature if a statistic t, say, is 2.30 and one of a radically different nature if t equals 2.31. No scientist will buy this unless he has been brainwashed and it is unfortunate that one has to accept as fact that many scientists have been brainwashed.

Think Kempthorne’s being hyperbolic in that last sentence? Nelson et al. (1986) did a survey of active researchers in psychology to ascertain their confidence in non-null hypotheses based on reported p values and discovered a sharp cliff effect (an abrupt change in confidence) at p = 0.05, despite the fact that p values change continuously across their whole range (a smaller cliff was found at p = 0.10). In response, Gigerenzer (2004: 590) lamented, “If psychologists are so smart, why are they so confused? Why is statistics carried out like compulsive hand washing?”

But now suppose we’ve learned our lesson: and so, chastened, we abandon our arbitrary threshold a value and look instead at the exact p value associated with our predictor variable, as many writers have advocated. And let’s supposed that it is impressively low, say p = 0.00073. We conclude, correctly, that if the null hypothesis were true (which we never really believed in the first place) then the data we actually obtained in our sample would have been pretty unlikely. So, following standard practice, we conclude that the probability that the null hypothesis is true is only 0.00073. Right? Wrong. We have confused the probability of the data if you are given the hypothesis,  P(Data|H0), which is p, with its inverse probability P(H0|Data), the probability of the hypothesis if you are given the data, which is something else entirely. Ironically, we can compute the inverse probability from the original probability – but only if we adopt a Bayesian approach that allows for “subjective” probabilities. That approach says that you begin the study of some prior belief (expressed as a probability) in a given hypothesis, and adjust that in light of your new data.

Alas, the whole NHST framework is by definition frequentist (that means it interprets your results as if you could do the same study countless times and your data are but one such realization)  and does not permit the inversion of probabilities, which can only be done by invoking that pesky Bayes’s theorem that drives frequentists nuts. In the frequentist worldview, the null hypothesis is either true or false, period; it cannot have an intermediate probability assigned to it. Which, of course, means that 1 – P(H0|Data), the probability that the alternative hypothesis is correct, is also undefined. In other words, if we do NHST, we have no warrant to conclude that either the null or the alternative hypothesis is true or false, or even likely or unlikely for that matter. To quote Jacob Cohen (1994), “The earth is round (p < 0.05).” Think about it.

(And what if our preferred alternative hypothesis is not the one and only possible alternative hypothesis? Then even if we could disprove the null, it would tell us nothing about the support provided by the data for our particular pet hypothesis. It would only show that some alternative is the correct one.)

But all this is moot. The calculation of p values assumes that we have drawn a simple random sample (SRS) from a population whose members are known with exactitude (i.e. from a comprehensive and non-redundant sample frame). There are corrections for certain kinds of deviations from SRS such as stratified sampling and cluster sampling, but these still assume an equal-probability random sampling method. This condition is almost never met in real-world research, including, God knows, my own. It’s not even met in experimental research – especially experiments on humans, which by moral necessity involve self-selection. In addition, the conventional interpretation of p values assumes that random sampling error associated with a finite sample size is the only source of error in our analysis, thus ignoring measurement error, various kinds of selection bias, model-specification error, etc., which together may greatly outweigh pure sampling error.

And don’t even get me started on the multiple-test problem, which can lead to completely erroneous estimates of the attained “significance” level of the test we finally decide to go with. This problem can get completely out of hand if any amount of exploratory analysis has been done. (Does anyone keep careful track of the number of preliminary analyses that are run in the course of, say, model development? I don’t.) As a result, the p values dutifully cranked out by statistical software packages are, to put it bluntly, wrong.

One final technical point: I mentioned above that almost no one sets a Î² value for their analysis, despite the fact that Î² determines how large a sample you’re going to need to meet your goal of rejecting the null hypothesis before you even go out and collect your data. Does it make any difference? Well, one survey calculated that the median (unreported) power of a large number of nonexperimental studies was about 0.48 (Maxwell, 2004). In other words, when it comes to accepting or rejecting the null hypothesis you might as well flip a coin.

And one final philosophical point: what do we really mean when we say that a finding is “statistically significant”? We mean an effect appears, according to some arbitrary standard such as p < 0.05, to exist. It does not tell us how big or how important the effect is. Statistical significance most emphatically is not the same as scientific or clinical significance. So why call it “significance” at all? With due deference to R. A. Fisher, who first came up with this odd and profoundly misleading bit of jargon, I suggest that the term “statistical significance” has been so corrupted by bad usage that it ought to be banished from the scientific literature.

In fact, I believe that NHST as a whole should be banished. At best, I regard an exact p value as providing nothing more than a loose indication of the uncertainty associated with a finite sample and a finite sample alone; it does not reveal God’s truth or substitute for thinking on the researcher’s part. I’m by no means alone (or original) in coming to this conclusion. More and more professional statisticians, as well as researchers in fields as diverse as demography, epidemiology, ecology, psychology, sociology, and so forth, are now making the same argument – and have been for some years (for just a few examples, see Oakes 1986; Rothman 1998; Hoem 2008; Cumming 2012; Fidler 2013; Kline 2013).

But if we abandon NHST, do we then have to stop doing anything other than purely descriptive statistics? After all, we were taught in intro stats that NHST is a virtual synonym for inferential statistics. But it’s not. This is not the place to discuss the alternatives to NHST in any detail (see the references provided a few sentences back), but it seems to me that instead of making a categorical yes/no decision about the existence of an effect (a rather metaphysical proposition), we should be more interested in estimating effect sizes and gauging their uncertainty through some form of interval estimation. We should also be fitting theoretically-interesting models and estimating their parameters, from which effect sizes can often be computed. And I have to admit, despite having been a diehard frequentist for the last several decades, I’m increasingly drawn to Bayesian analysis (for a crystal-clear introduction to which, see Kruschke, 2011). Thinking in terms of the posterior distribution, of support for a model provided by previous research as modified by new data seems a quite natural and intuitive way to capture how scientific knowledge actually accumulates. Anyway, the current literature is full of alternatives to NHST, and we should be exploring them.

By the way, the whole anti-NHST movement is relevant to the “Mermaid’s Tale” because most published biomedical and epidemiological “discoveries” (including what’s published in press releases) amount to nothing more than the blind acceptance of p values less than 0.05. I point to Anne Buchanan’s recent critical posting here about studies supposedly showing that sunlight significantly reduces blood pressure. At the p < 0.05 level, no doubt.



REFERENCES

Cohen, J. (1994) The earth is round (p < 0.05). American Psychologist 49: 997-1003.

Cumming, G. (2012) Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York: Routledge.

Fidler, F. (2013) From Statistical Significance to Effect Estimation: Statistical Reform in Psychology, Medicine and Ecology. Routledge, New York.

Gigerenzer, G. (2004) Mindless statistics. Journal of Socio-Economics 33: 587-606.

Hoem, J. M. (2008) The reporting of statistical significance in scientific journals: A reflexion. Demographic Research 18: 437-42.

Kempthorne, O. (1971) Discussion comment in Godambe, V. P., and Sprott, D. A. (eds.), Foundations of Statistical Inference. Toronto: Holt, Rinehart, and Winston.

Kline, R. B. (2013) Beyond Significance Testing: Statistics Reform in the Behavioral Sciences. Washington: American Psychological Association.

Kruschke, J. K. (2011) Doing Bayesian Analysis. Amsterdam: Elsevier.

Longford, N. T. (2005) Model selection and efficiency: Is “which model…?” the right question? Journal of the Royal Statistical Society (Series A) 168: 469-72.

Maxwell, S. E. (2004) The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 9: 147-63.

Nelson, N., Rosenthal, R., and Rosnow, R. L. (1986) Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist 41: 1299-1301.

Oakes, M. (1986) Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York: John Wiley and Sons.

Rothman, K. J. (1998) Writing for Epidemiology. Epidemiology 9: 333-37.

Ziliak, S., and McCloskey, D. N. (2008) The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press.

Monday, November 10, 2014

Dragonflies and innate understanding of physics

"The human mind possesses a basic probabilistic knowledge."  So say Fontanari et al. in a newly published paper in PNAS ("Probabilistic cognition in two indigenous Mayan groups").  They asked whether formal schooling was a necessary foundation for a sense of chance by comparing two unschooled Mayan groups with Mayan schoolchildren and a control, and determined that no formal education is required for making "correct probabilistic evaluations."

This paper hit the popular news media.  "We are all natural bookmakers," said New Scientist.  And the senior author, Vittorio Girotto, said,
"We wanted to show that this sense of chance exists, that it is universal, and that you do not need to be trained to evaluate uncertainty," says Girotto. "We have good evidence now that the human mind does possess this ability."
Researchers have also reported that infants have a sense of "intuitive physics," seemingly being born with the ability to understand gravity (that is, by 2 months of age, they expect an object to fall -- really, who understands gravity?), and to expect that an object doesn't cease to exist when hidden from view.

And, studies (e.g., here and here) suggest that by 5 or 6 months, infants have a sense of numbers.  But then, so, apparently, do non-human primates, such as tamarins.  When two objects were hidden behind a screen, tamarins expected to see two objects when the screen was lifted; when there were three, the animals looked at the objects longer than when they were presented with the expected number, suggesting surprise or confusion.  But then, dogs are good at playing Frisbee because they understand physics, too -- what does up there, comes down here.  

And, even crows understand water displacement, knowing that if they raise the water level in a small beaker, they'll be able to pluck out a piece of floating food.



And look at how bats, and even dragonflies track their in-flight prey.



There seem to be several things going on when things like the probability study in unschooled Mayans make the news.  To those of us who are schooled, probability, mathematics, physics -- or even grammar -- can seem like rather esoteric subjects that take years of training to master, or to even vaguely understand (though really, who understands probability?).  Traditional schooling has divided the world we know into disciplines that have names and bodies of knowledge that must be mastered.

But, in large part, formal education is giving names to things we already knew.  We have already internalized grammar as infants, we have a grasp of essential physical or mathematical principles, and it seems some basic understanding of chance as well. Essentially we're formalizing our description of the world we know from experience, but clearly we -- and dogs and tamarins, and crows and dragonflies and many other animals -- know that world before we know words or equations or models or principles that describe it.  And indeed, most animals never get to that stage.  I think we all can do this not because we have an innate sense of physics, or grammar, but because our brains have evolved to be able to recognize some kind of order, and to make generalizations from what we experience.  It's apparently important to survival, because so many organisms have evolved the same ability.

In this context, we should keep in mind that mathematics is just an elegant way of describing relationships and really exists only because over the millennia humans did in fact realize that relationships had regularity.  The long-known fact to western science, for example, that the Mayans had very sophisticated calendars shows that the recent news story is no surprise at all -- indeed, it would be very surprising were it not so.   How things work in the brain is, however, a different order of question.

Holly elegantly suggests it simply comes down to pattern recognition.  Frisbees follow predictable arcs, objects don't disappear inexplicably, if there are 5 yellow tokens and only 1 red, the chance of choosing a yellow one is higher than the chance of choosing a red one.  I'm happy with that.

Monday, November 3, 2014

TMI? Personalized probability

I went to my dentist on Friday expecting to go home with a permanent crown on one of my molars but I went home with an appointment with an endodontist for evaluation for a root canal instead.  My dentist, who I think is a very good one, said that 6% of people getting crowns go on to need a root canal, and, based on my symptoms, he was 90% sure I'd be one of those 6%.  Lovely.

But what does that 6% even mean?  Just off the top of my head, it could reflect the fallibility of indications for the need for a crown, it could mean that the need for a crown and a root canal are unrelated events that just happened to occur at the same time, it could mean that some dentists are better diagnosticians than others.  And of course, it's based on the assumption that all crowns are necessary, and are fixing something real, because if not, people getting a crown who didn't need them aren't actually at risk of a root canal.  Maybe all of these.  Someone might know, but not I.  That 90% was, I assume, mostly professional courtesy, allowing for the possibility that the endodontist may not agree--that is, it was a subjective but not really a probability statement and how he arrived at it is anyone's guess (maybe even his own!).

Root canal; Wikipedia


My dentist also told me that there is probably a subgroup of crown patients that is more likely to need root canals than others.  If I remember correctly what he said -- keep in mind that I had just been told I'd need one of the most dreaded dental procedures there is, so my memory could be faulty here -- he said that the subgroup consists of people who experienced more pain before the crown procedure than the other 94%.  Which I translated to mean that these could be people with problems that their dentists misdiagnosed; maybe they needed a root canal from the beginning, but I could be totally wrong about that because I really know next to nothing about how dentists decide when a tooth needs a crown or a root canal.  In any case, it did make me think about the whole idea of probability and prediction.

First, I wasn't at all surprised that I landed in that 6% group.  My family has experienced what we think is an unlikely number of rare events, so of course I would be in that 6%.  Right?  Well, no.  Each event has been seemingly independent, with separate causes, so what has happened before has no bearing on whether or not I'm going to have a root canal.  One person being hit by lightning (say) doesn't make it more likely that another will be hit by a bus.  Unless, you know, bad luck is genetic.

But let's say these things aren't genetic.  Let's say they are random events and everyone at risk is equally at risk.  It's easy to calculate the chances of something like winning the lottery because the denominator, the number of people at risk, is readily available -- it's the number of people who buy a lottery ticket.  It's a lot harder to calculate the chances of being hit by lightning or by a bus because we can't know how many people are at risk.  We'd have to know how many people are outside during a storm, or crossing a street when a bus comes along, and that's impossible unless we're asking about your street corner or my backyard.

In that case, we've made it easy to calculate the denominator, but the likelihood of the event happening exactly there has now become astronomically low, no matter how precisely we know how many are at risk.  And if we expand the geographic area again to a city or a state so that the event has a reasonable probability of happening there, we might know how many people are struck by lightning or a bus (though, that's a hard one too because surely people don't always report being hit by a bus) but again, we've lost sight of the denominator so it's impossible to calculate the probabilities.  Unless, like the National Weather Service, we assume the whole population of the US is at risk every year, in which case, according to their calculations, each of us has a probability of being hit by lightning in a given year ranging from 1/1,900,000 to 1/960,000 depending on whether we're using actual reported lightning strikes or estimated strikes.

What are the odds of being hit by a bus in a given year?  This is a hard statistic to find, and it's necessarily just an estimate not only because such accidents aren't always reported, and we haven't specified whether we mean fatal or not, but also because sometimes bus accidents involve people inside the bus.  Given all that, the odds of being hit by a bus per year in the UK have been estimated to be 1/500,000.  That would mean something like that there were x many reported bus accidents and so many people in the UK in that year (but are they counting tourists and undocumented workers?).  But that's also a bit suspect, because some people live and walk around in cities, and others in the country where there are no buses.  Or what if they drive?  Does being hit by a bus while in one's car count in the reported statistic?  

Let's say someone in your family has been hit by a bus, and another family member has been hit by lightning. If the two are independent -- and how would we know that? -- then the odds would be the product of the two figures given above, roughly ten to the minus 11th power, or 11 zeroes after the decimal point.  Since there are only about 10 to the 9th power humans on earth, your nuclear family is probably the only family on Earth in which these two events have occurred together (actually, this is impossible to estimate for a given family if we don't specify family size and so on, but the point is that there can be events so rare that the fact of both of them happening in one family is a one-time occurrence.)  If we excuse indigenous people living where there are no buses (there is likely to be lightning), even just these two events make you super-unique. And it's such a rare circumstance that it's hard to know what it has to teach us about probabilities of such things happening.  Most of the useful stuff we learn from science is based on reproducible events -- because if not, the odd thing that happened ends up on the News of the Weird page in your local paper.  But even things like lightning strikes are not truly reproducible.

We can enumerate all the steps that led to my Uncle Ed being hit by a bus, but they won't necessarily apply to your Uncle Ed's experience with a bus.  The outcome, that they both ended up in front of a bus at the wrong time, is the only generalizable and useful fact here.  That my Uncle Ed stopped to buy a newspaper to read this week's News of the Weird, and was so absorbed in the story about the elephant that he stepped off the curb to cross the street just when the bus came around the corner is irrelevant to the facts of your Uncle Ed's ill-fated meeting with his bus. Your Uncle Ed, as it happened, was in a hurry and, yes, texting, and he stepped off the curb to get around the throng of tourists gawking at the Empire State Building just as a bus came along.

New York City bus; Wikipedia

The facts in each case, interesting and important as they are to each Uncle Ed and his family, can't be used to predict the steps by which someone else's Uncle Ed will end up being hit by a bus. We might conclude that reading the News of the Weird is dangerous, but that seems illogical.  We might, but probably shouldn't conclude that tourists are dangerous.  But if we begin to collect enough stories about people being hit by a bus while texting, we can start to suspect that texting might be dangerous. But by looking at causation in aggregate, we're likely to pay attention to common causes and ignore the rare ones.  Indeed, my dentist didn't feel he needed to tell me that my tooth that seemed to need a simple crown might end up also needing a root canal.

But, in every case, we've got a lot of information that doesn't apply to everyone else, or even anyone else, even if it applies in the individual case.  TMI, except that if the event is rare -- and particularly if it's unique -- we can't always determine which information is relevant to its cause, especially if there are multiple pathways to the outcome.  It's only when we've got a lot of replications of the same event that we can begin to sort that out, but even then we usually can't begin to predict the event.  Not everyone texting gets hit by a bus, just as not everyone who smokes gets cancer, and not everyone with a BRCA risk allele gets breast cancer.  Public health policy warns against the effects of smoking anyway, even when only 10% of smokers get cancer.  We're not good at predicting who will be among that 10%, so we warn everyone.

The danger of numerology
We often read about coincidences like the Uncle Eds'.  Probably, you can find the story in the News of the Weird -- two Uncle Eds hit by a bus.  And if it was the same bus, all the better.  But there is a very common danger in that.  If you sort through enough possible events, what in itself seems like an incredible coincidence becomes essentially expectable.  There are enough rare events in this world that the chance two Eds share at least some such event is not so small.  The chances that my family has experienced the particular rare events that we've experienced is probably so small that we're the only family ever to have experienced these specific events in the history of families on Earth, but the number of families that are unique in the same way is in fact probably quite common.  And depending on how we classify events, every family is unique.  (A good discussion of this is in a recent podcast of a BBC Radio4 program More or Less, that tries to debunk misuses of statistics.)  We see the problem of assessing the uniqueness of events all the time in twin studies, too.  If my dog and your dog have the same name it's just a curious coincidence, but if my dog and my twin's dog have the same name, Nature will publish that as a genetic finding (thank you, Holly).  So we have to be careful of that sort of numerological game.

The nature of prediction, done properly
The concept of 'probability' is a very elusive one, and really it has no single definition.  But most uses refer to repeatable events, as mentioned above.  That is, the probability of an outcome is the fraction of times that outcome is observed among repeated experiences.  Likewise, prediction is generally based on aggregate data. That's what epidemiology does; collect a pile of data in a population and figure out which variables, or exposures, are most likely associated with an outcome or disease. In genetic epidemiology, if enough people with a disease share a genetic variant, that variant starts to look causal.  In the easiest cases, which are usually rare diseases, the variant can then often, but certainly not always, be considered predictive, or at least diagnostic.

As we've said many times before, epidemiology and many other areas of observational science do best when causation is simple.  Simple point, or single-factor causes, make the notion of replication more sensible and more reliably observable.  But this is far less clear when causation is complex, involving multiple simultaneous contributing factors, no one combination of which is necessary.  

Too often, even if hundreds of millions of dollars have been spent on trying to understand the cause of a disease, prediction is hard.  In a very nice post on the blog 'chronic health, Luzt Kraushaar shows that this is true for heart disease.  He writes:
If weather forecasts were as reliable as cardiovascular risk prediction tools, meteorologists would miss two thirds of all hurricanes, expect rain for 8 out of 10 sunny days, and fail to see the parallels to fortune telling.  
Epidemiologists have tried for decades to separate the wheat from the chaff with heart disease, to determine which risk factors are predictive; which are necessary and sufficient.  In Uncle Ed terms, this means they've tried to determine whether reading News of the Weird is causal or whether the cause is actually texting.  Or maybe both are causal. And even though heart disease is a major cause of death in rich countries, we still don't know very well how to predict who'll be affected.  In a way, that's because every heart attack is a rare, even unique event, and aggregating data may discard the important variables about the cause of my heart attack.  For my Uncle Ed, the News of the Weird was an important cause of his getting hit by the bus, but not predictive for anyone else, or at least not reliably predictive.  

But we're all rare events.  Every genome that has ever existed has been unique.  We are each now a 1 in 7.5 billion event (and that's without counting our also entirely unique ancestors).  And this means that every event for everyone of us follows its own path.  Even the phenotype of single gene diseases can differ between individuals because while they may share a causal variant it's in a unique genomic context, and the modifying effects of that context are unique.  Further, when a disease involves environmental exposures, everyone's will be unique, and this makes retrodicting cause (which is what epidemiology does) difficult, but predicting it is even more difficult because we can't predict future environments, and we can't know which particular genomic contexts, each one of them unique, are significant risk factors or what kinds of modifiers they'll be.

So, while we're being promised miracles by believers in personalized medicine based on our DNA, this in some ways will be too much information that we don't know what to make of.  Predicting complex diseases, those due to environmental exposures interacting with many genes, is going to be impossible, in principle, much of the time.  Who knew News of the Weird was going to cause Uncle Ed to walk in front of a bus?

Wednesday, March 12, 2014

What's the point? Causation strong, weak, and undetectable

We often write and, yes, complain about the kinds of sample-based survey statistical studies of health risk, and this week's news about the cause of obesity is another good reason to revisit the subject.  (In case you missed it, we've got two options this week, it's either sugar or antibiotics.)  Whether the risk factor is genetic or environmental, the fields use similar conceptual approaches.  Essentially, they are 19th century concepts largely originating in physics but applied to society at the time, and now dressed in 20th century computational and ready-made statistical methods.

To drink, or not to drink. Is that a question?

Here's a wonderful send up of the problem on the BBC Radio 4 program, The News Quiz.  (Start at 13:05.)  "Remember the Mediterranean Diet?  The last time I was in the Mediterranean, we had kebabs and San Miguel!"  "One week we are told to drink red wine, and the next week, white."  And so on. When the inability of science to determine risk factors becomes the butt of jokes, it's no laughing matter. (H/T Tom Farsides.)

Or is this all white to drink?
When there is what is referred to as point causation, a single major factor with very high predictive power, the approach works well.  Somewhat different methods are and have been used for genetic causation (Mendelian inheritance methods) or infectious or some environmental disease causes, when a single exposure or an exposure to a single 'point' cause is responsible.   If the cause has strong effects, even probabilistic aspects of its action or appearance are well-analyzed by fundamental statistical ideas and methods. These depend on replication (basically, that each observation is a repeat of the others) and a high-level of what essentially amounts to causal determinism by the risk factor, even if there are probabilistic aspects involved (such as the chance of being exposed to a pathogen, or of inheriting a Mendelian disease variant).

A good and interesting survey of the way that probabilistic and statistical thinking entered western science is given by Ian Hacking's book The Taming of Chance (Cambridge University Press, 1990). Hacking retraces debates over whether causation was just apparently probabilistic or was fundamentally so, and how 19th century's versions of 'big data', in the form of national statistical survey data used for things like insurance rate determination, legitimized what was essentially the borrowing of methods from physics to apply them to society.

Because of the success of epidemiology during the infectious disease (pre-antibiotic) era, and in genetics the Mendelian segregation analysis era, the idea grew that we could, with sufficient samples, apply them essentially to all forms of causation.  That has led to today's obsession with Big Data and elaborate (often off-the-shelf) statistical methods, basically the same cookie-cutter approach to any question.

But there is a problem with that.  It's what we often write about here, because we think it is a serious one.  We think that for reasons of cultural and scientific inertia and lack of sufficiently imaginative training, we cling to these methods even though we have good theoretical as well as experiential reasons for knowing they are inappropriate--they basically don't work, but we do them anyway.

When there are many different factors at play, varying among samples or populations, and most of the factors are either of low prevalence (rare in the population) or weak effect, the gussied-up 19th century mode of thinking is just not very effective at analysis or prediction.  That is why, no matter how one may defend the continuation of big-investment, big-data, long-term, high-throughput investment, we are daily seeing the uncertain, changeable, non-definitive results reported in the news and in the journals themselves.  Questions like the risks associated per gram of sugar, or of our dietary intake of antibiotics in food, or of PSA testing or regular mammography, or GWAS and other claims of genes 'for' this or that trait, are changing daily with each study being reported by the authors and the media as if, somehow, it is now, finally, definitive.

To increase sample size and get results, without having to actually think about the problem or frame well-posed questions, researchers propose exhaustive national or international studies, high-throughput data analysis, meta-analysis and the like.  Of course, if you look for associations and use statistical tests of different kinds, you will always get a 'result'.  But among other things, the approaches basically assume a well-behaved signal-to-noise ratio, that is, that the bigger the sample the closer the estimates of risk will be to the (assumed) underlying truth with less 'noise' due to measurement and sampling errors. That assumes that as we add new samples, their heterogeneity will be less than their collective refinement of the risk estimates--less error in those estimates--assuming they have true values. We know very well and from very extensive experience that there are reasons why this assumption is unlikely to be true, that are borne out in the data, but the expediency of proposing just to collect more, more, more seems too convenient to abandon.

Likewise, the idea that risk factors and their relative frequencies will be the same in the future as in our samples from the past.  But we know that that is bunkum even from a genetic point of view (variants' frequencies depend heavily on samples chosen, new variants are introduced and others lost), as well as environmental (lifestyles change in ways that are inherently unpredictable, and predicting environmental effects on gene expression, perhaps of genes never before found to be involved in disease, is also equally impossible).

In the face of daily now you see it, now you don't stories, always hyped up by the professors, the funders, and the media, one might expect that the science would stop and take stock, realizing that we are driving a model-T on a modern highway and need to do differently.  But that seems just too difficult.  Like turning a tanker around in the ocean, it's a slow, cumbersome process.  And, to a great extent, we have not yet collectively agreed that we even have a problem (even though we know very well that we do).

Of course, it's at least possible that what we see is what there is: that there simply isn't a better way besides fickle statistical association analysis to understand the risks we want to understand.  At least, then, we should show that that is the case with some definitive evidence demonstrating that it's impossible to do better.  But even that would require more careful thought than is being given to these problems.

In truth can anyone not realize that a significant factor in the current way of doing business is that investigators know that fickle results are good business--that one never need say a study is over or a question answered, and can keep asking for more data, more computers, longer, larger studies?

Strong point causation is like a drug.  Its initial pleasures addict us to hope for the high of simple answers for every question.  But the truth, weak causation, has lured us into addiction.

Wednesday, February 26, 2014

Godel's principle and respect for failure

In 1931 Kurt Godel shocked the mathematical world.  Math is the ultimate sanctuary for those who believe that some ultimately Platonic sense there is absolute, universal, unexceptionable--and understandable truth. The facts of geometry and mathematics are cosmically true (the Pythagorean theorem, the sum of angles =180 degrees, etc., 2+2=4, and the fact that the derivative of x-squared is 2x, etc). The way we do math may be a human or cultural convention, but the facts are not.  Our cultural convention of how we do it, or whatever it is, may for cultural or historical reasons simply overlook many equivalent truths, but it is working with at least some set of ultimate truths.  But how is it that Pythagorean theorem is true about right triangles, but there are no actual, perfect right triangles in the world??

2+2=4

At least, at the turn of the 20th century, for those studying such ultimate truths, it was widely thought that the principles of logic and logical reasoning were essentially the same as the principles of mathematics.  Not only were both equally true but logical reasoning could be expressed in the same kinds of terms as those of mathematics.  This would make the world a certain place, in a sense.  Truth is truth. Truth is internally consistent.  And truth is discoverable!

Infinity symbol in various typefaces; Wikipedia


There were some problems.  For example, what do we do with or about the notion of 'infinity'?  Nineteenth century mathematicians, notably Cantor, showed that there are even different levels of infinity.  The whole numbers are one level.  You can match even numbers up one for one with odd numbers.  But you can't match either up like that for the numbers between just 0 and 1.  That's because there are far more of the latter:  if we match 0, 1, and 2 with 0, 0.1, and 0.2 it might seem fine, but then how do we match up 0.001, 0.002, 0.00001, and so on?

The constant Ï€ is represented in thismosaic outside the mathematics building at the Technische Universität Berlin. Wikipedia

And then there is the little problem of what 'randomness' means.  For example, I recently learned that the digits in the value of  pi (relating radius to circumference in a circle) are randomly distributed.  Take any sequence, like 7623116, and you'll find it, on average every 10 million 7-digit sequencs in pi.  Or look at Stephan Wolfram's 'cellular automata'; these are simple rules for transitions in a string of (say) black or white boxes if each box produces a 'descendant' box and the rule says what color it is based on the array of colors in the current generation.  The resulting black/white pattern, determined by a simple rule, is, Wolfram says, indistinguishable from random: no pattern can be found along the string at any given time.  Yet 'random' seems such an obvious concept...until you think about them too much, and then they become quite disturbing.

And I have not mentioned the very problematic notion of probability.  We know how a cause can lead to an effect--well, we think we know that--but it is totally unclear how a cause can only lead to the future probabilistically.   Usually, the probabilistic nature of such results is attributed to measurement error, poor theoretical understanding, and so on, in a cosmos that, were we to know everything, would be purely and rigidly law-like.  But how could something cause something else 'with 27% probability'?  If you think about it, it is not at all clear what that means, in terms of actual causation, beyond errors and sampling effects.

And then there is 'chaos' theory.  Even in a purely deterministic, rule-bound process of cause and effect, where there is no uncertainty or probability involved, unless you have 100% measurement accuracy of things at some given time, you cannot predict with any accuracy what things will be like over the future.  Your predictive power, even with perfectly true theory, is zilch.  But how can you know what the underlying reason is?

Such things fly in the face of the views of the cosmos as a law-like place where certainty rules.  We want a knowable universe.  We spend our puny lives as scientists trying to understand it, and assuming it at least exists, even if it's hard to understand!

What we have learned
To general chagrin, what mathematician Kurt Godel showed halfway through the last century, was that even if it were true that the mathematical realm was all-of-perfection, an unknown fraction of it was unknowable.  That is, things that are true can't be proved and, worse, you could never know whether something you thought were true and were trying to prove it, was in reality untrue or just unprovably true.

Given all of this, it is surprising that in so many cases what we have learned is that the universe may not be law-like in the way we'd thought, true probability may or may not exist, not all things can be shown to be true even if they are true, and (in quantum mechanics and relativitiy and gravity at least) there are phenomena that seem truly to be unlike any of the above concepts or, as physicists often say, just are not consistent with 'common sense'....even if they're true.

Even physics and chemistry, not to mention biology and psychology or economics, are often swimming in uncertainty and claims of knowledge that, no matter how confidently asserted, simply don't hold up.

These various incarnations of indeterminacy, like probability, can shake our faith in the idea of a knowable universe whose causal nature we can pin down tight.  In ordinary sciences, and sometimes even in our daily lives, we don't know exactly how we should be viewing, much less approaching, causation.  What we end up doing is designing studies or experiments that we know how to design, using methods we know how to use, and crossing our fingers.  Whether we're being ostriches to our peril, or whether it doesn't matter and we should just carry on regardless, is unclear.

To the young and thoughtful, these provide things to think about, both in the practical sense of actually moving the science forward more than a millimeter at a time.....and in terms of our ultimate hope to understand life as it really is, not just in a statistical analysis.