What is scientific knowledge and how hard is it to come by?

Two stories we're looking at reflect issues that we care about, and that we think everyone who thinks about how we know what we know should care about.  They have to do with the fragility of knowledge in at least some areas of science, including biomedical and genetic knowledge.  There are some lessons here, and they are perhaps surprisingly indirect.

What's in a name?  A rose by any other name (except Rosenkrantz) would be as sweet
The tweetosphere is resounding with comments on the study that looked at whether hurricane names affect the damage they do.  This is not some group of mystics studying Karma, but real scientists (perhaps with nothing else to do, since the subject would seem rather remote from what people on a payroll ought to be spending their time at).  

Hurricane Katrina; Wikimedia

As reported in this PNAS paper, ("Female hurricanes are deadlier than male hurricanes," Jung et al.),  more people die in hurricanes with women's names than hurricanes with male names.  This sounds wholly silly since the names are chosen before any hurricanes arise, and, anyway, were selected in different ways over the years.  But according to this paper, hurricanes with female names give people the impression that they are kinder and gentler than those with male names; the Rose that smells sweet.  At least, the authors' interpretation of their result is that when a female-named hurricane is barreling toward you, you may be tempted to stay home rather than evacuate, given that the storm is just a girl.  Or, if you expect that male-named hurricanes are more deadly, the treacherous Rosenkrantz to be suspected and feared, you either evacuate or you stay home, in the basement, with a helmet on, under a cover so you can pretend the monster isn't at your very door trying to huff and puff and blow your house down.

This paper is not being treated with much respect, and at least some critiques of the findings are over at Ed Yong's post about this story, and there are important issues about science communication and the respect that prominent scientific journals deserve, or not.  And, methodological issues as well.  These raise serious doubts about the validity, or at least the interpretability of the study, and about the journal's acceptance criteria (if the study's methods or conclusions were only marginally solid, it perhaps belonged in a supposedly lesser journal).  

However, our point here is different, and has to do with the many subtle ways in which exposures to all sorts of things epidemiologists or geneticists might never think of, much less be able to measure, could have major effects on our behavioral and physical traits.

What if this study is not poorly done and its reports have identified a real effect, and at least a handful of University of Illinois undergrads associate gender with relative danger--or power, or aggressivity, or some other trait that they associate with a hurricane's force?  Fear-by-name would be a cultural trait, and it might be very different in other age groups, or states, or other cultures--in matriarchal cultures, for example, the effect might be reversed.  This would be an instance of what is an undeniably real problem with many epidemiological studies--the problem of confounders, unmeasured variables that influence the outcome but whose effect is unseen because it's unconsidered, but that are actually causal. The identified cause could be so indirect in a multiplicity of ways, that it may be 'true' but basically uninterpretable in terms of what one might hope to find as the actual primary causes, or to determine whether there was or was not a causal link between the correlated risk factors.

Whether one accepts or questions the validity of this particular correlation study, how many other such kinds of subtle causation actually do affect our lives?  How many studies' interpretations such problems they undermine?  How can the outcomes be predicted in a sea of causes?

What's up, Doc?
The second story, old but good, is a commentary by a physician who's been around the block a few times, observing that older physicians are more likely to admit that neither they, nor anyone, understands biomedical causation as well as they like to claim.

Even as a med student, I was struck by the discrepancy between how much the junior doctors (particularly the interns and second-year residents) seemed to know, and how much the more experienced doctors knew: with few exceptions, the junior doctors seemed to know a lot more.

Students and young physicians have explanations for everything, and seem very confident about the extent and precision of knowledge about cause and treatment and so on.

In contrast, the expert physicians – the doctors who had spent decades of their lives treating particular types of patients, and studying a specific disease – tended to be far less definitive, and much more likely to say, “to tell you the truth, we really don’t know.”

One would expect, if science and experience in seeing purportedly causal symptoms and the effect of treatment are so fundamental, that senior physicians would be spot-on much more of the time than their greenhorn juniors.  They have had the experience by which to put two and two together, after all. Instead, at least the thoughtful and candid seniors have far less confidence than their younger upstarts. Why is this?

Holly Dunsworth reminded us the other day that there's a name for this phenomenon.  It's the Dunning-Kruger effect and Wikipedia describes it this way: "Unskilled individuals suffer from illusory superiority, mistakenly rating their ability much higher than is accurate. This bias is attributed to a metacognitive inability of the unskilled to recognize their ineptitude," citing a NYTimes piece about a bank robber who painted his face with lemon juice, under the misapprehension that it would make him invisible to surveillance cameras.  As David Dunning (he of the effect) read about this bank robber, who had been immediately caught from video tape pictures, he thought, if the guy "was too stupid to be a bank robber, perhaps he was also too stupid to know that he was too stupid to be a bank robber — that is, his stupidity protected him from an awareness of his own stupidity."

So, medical students don't know yet how much they don't know.  Further, young people are have just been taught courses by professors who don't get paid to teach what isn't known, but instead (especially, perhaps, in medical school) teach the truth--the symptoms and their causal results and the miracles of physicians' interventions.  They have a vade mecum drummed into their heads: a code book of diagnosis and treatment.  Indeed, the growth of HMO and computing power has to a great extent led to a hegemony that leads them to be trained not to use their individual judgment.

The young are also starting their careers, and they have the excited belief that they'll be implementing this crisp knowledge they've been taught.  Even if they go into academic research, careers are made by declaring what we know and don't know in confidently specific terms.  Give me a big grant and I'll answer this big question.  They are committed to the idea that questions, as they're often posed, actually have answers.  Doubt doesn't sell so there's no incentive to doubt a simple line.  Indeed, even some skilled individuals are trained to have the belief, if not illusion, of their superiority as experts, with degrees and salaries to match, and journalists hanging on their every word.

Both phenomena--the indirect effects of hurricane names and the over-confidence of those who don't take (or care not to take) what science tells us seriously enough, lead to expensive over-confidence in science and scientists, and not nearly enough humility in the face of the complexity that Nature throws at us.