Monday, August 23, 2010

The inconstancy of life...or is it just of our findings? Irreproducible results

We are awash in inconsistent findings from the kinds of research that is often done in epidemiology, genetics, and even evolutionary biology.  One study leads to a given assertion--some risk factor or selective agent causes some outcome--but the excited follow-up study fails to confirm it.  Today Twinkies are good for you, tomorrow they're nearly lethal!

We post about this regularly, and it can be found almost every day in the popular science news and even in the scientific literature itself.  GWAS are of course a very good example that we mention frequently.

These results are basically statistical ones resulting from various kinds of sampling.  If they are consistent in anything, it's their inconsistency.  And therein lies a serious challenge to observational science (including evolutionary biology).

An important part of the problem is that when effects are small (assuming they're real), there's a substantial probability that the excess risk they're responsible for won't be detected in a study sample.  Study samples match cases and controls, or some similar kind of comparison, and a minor cause will be found about as often in both groups, and that means by chance may be found either more often in the controls or not sufficiently more often in cases to pass a test of statistical significance.

A second problem is complexity and confounding.  When many variables are responsible for a given outcome, the controls and cases may differ in ways not actually measured, so that the net effect of the risk factor under test may be swamped by these other factors.

Finally, the putative risk factor may have appeared on someone's radar by chance or by fluke or by a hyperactive imagination, a prejudicial bias, or a Freudian nightmare.  We tend to see bad things all around us, and since we don't want anything bad at all of any kind ever, and we have a huge and hungry epidemiology industry, we're bound to test countless things.  Puritanism may lead us inadvertently to assume that if it's fun it must be bad for you.   Yet, negative findings aren't reported as often as positive ones, and that leads to biased reporting: the flukes that turn out positive get the ink.

We published a paper a while ago in which we listed a number of inconsistent findings.  Ken has been told that the existence of this list has made its way into a book, and consequently he's gotten requests for the list.  So, we thought we'd post it here.  It's out of date now, and we could update it with a lot more examples, but we're sure you can think of plenty on your own. 

Wishful thinking and legitimate hopes for knowledge lead us to tend to believe things that are far more tentative than they may appear on the surface.  It's only natural--but it's not good science.  It's a major problem that we face in relating science to society today.

Table of irreproducible results?

Hormone replacement therapy and heart disease
Hormone replacement therapy and cancer
Stress and stomach ulcers
Annual physical checkups and disease prevention
Behavioural disorders and their cause
Diagnostic mammography and cancer prevention
Breast self-exam and cancer prevention
Echinacea and colds
Vitamin C and colds
Baby aspirin and heart disease prevention
Dietary salt and hypertension
Dietary fat and heart disease
Dietary calcium and bone strength
Obesity and disease
Dietary fibre and colon cancer
The food pyramid and nutrient RDAs
Cholesterol and heart disease
Homocysteine and heart disease
Inflammation and heart disease
Olive oil and breast cancer
Fidgeting and obesity
Sun and cancer
Mercury and autism
Obstetric practice and schizophrenia
Mothering patterns and schizophrenia
Anything else and schizophrenia
Red wine (but not white, and not grape juice) and heart disease
Syphilis and genes
Mothering patterns and autism
Breast feeding and asthma
Bottle feeding and asthma
Anything and asthma
Power transformers and leukaemia
Nuclear power plants and leukaemia
Cell phones and brain tumours
Vitamin antioxidants and cancer, aging
HMOs and reduced health care cost
HMOs and healthier Americans
Genes and you name it!

No comments: