Ignoring our ignorance

Seeing more
Two pieces interesting in their own right herein collide.  Here's a video from the NYT about an enhanced technique that allows the viewer to detect motion that is invisible to the naked eye -- or camera.



This new technology potentially has clinical usefulness, according to the developers, and if so, that's a good thing.  But it's also interesting as a stark reminder of how much we can't and don't see around us.  Granted, this technology primarily makes visible things we can detect with more familiar (and cheaper) techniques (like stethoscopes), but still, new ways of seeing -- your heartbeat right there on your face -- are not only eye-openers, they're also brain-openers.

Seeing less
Here's another reminder that we don't always see what we are looking at.  A paper in the February issue of BioEssays, "When peers are not peers and don't know it: The Dunning-Kruger effect and self-fulfilling prophecy in peer-review," Sui Huang, is a discussion of the problem of peer reviewing by reviewers who don't recognize that they don't understand what they are being asked to evaluate.

The Dunning-Kruger effect is when people don't recognize that how much they don't know about a subject, and rate their own abilities or knowledge higher than merited.  This has implications in academia, as this paper points out; when peers aren't in fact intellectual peers, they can determine the fate of a paper, or grant application, without even recognizing that they've missed the crucial points.

The Dunning-Kruger effect is happening in science more and more, according to Huang, for a number of reasons.  One, perhaps a reviewer doesn't recognize the specialized use of a word, and assumes its colloquial or older meaning instead -- chaos, or epigenetics are two examples Huang cites.

Failure to consider the “other” meanings of a term prevents the recognition of one's own ignorance of concepts used in other fields. Second, because of the parceling of science into small kingdoms, authors often are the sole authority in their province with no equal. Finally, the increasingly interdisciplinary nature of research creates an asymmetry of knowledge: the reviewer as a single person faces the daunting combined knowledge of an entire team of coauthors. Thus, statistically, we can safely accept our first claim and assume that on average, reviewers nowadays are with high probability less knowledgeable about the subject matter of a manuscript than its authors.

Of course, there are many other possible reasons that papers or grants don't get adequately reviewed.  Editors don't know everything either, and don't know everything that potential reviewers don't know, so there's potentially a lot of ignorance determining the shape of other people's careers.  And they are overloaded trying to find knowledgeable and willing reviewers, and are in no position to really judge the reviewer's formal qualifications, patience, meticulousness in reviewing, or, of course, understanding of a particular paper.  Online publishing (or inclusion of essentially limitless Supplemental information) overwhelms reviewers.  Grant reviewing takes another toll.  And paper authors aren't always clear about what they did or how they did it--and there's a lot of intentional obfuscation, too.  But this is really a topic for another time.

Huang speculates as to why reviewers might not recognize the limits of their knowledge.  Pride (though this would perhaps indicate the limits recognized but not acknowledged to others), self-deception, simple ignorance -- if you don't know that 'chaos' has a specialized meaning, there's nothing to recognize.

The Dunning-Kruger effect is always true and always has been
But of course, ignorance of their own ignorance isn't particular to reviewers.  Aren't we all subject to the Dunning-Kruger effect all the time, bumping up against the limits of our knowledge but drawing conclusions anyway?  The Earth was once flat, the sun once revolved around the Earth, the continents were stationary, evolution wasn't true.  No one ever worries that they don't know as much as people will in 50, 200, 2000 years, and therefore decides she has no business making observations and drawing conclusions.

It's humbling, and sobering.  Or, it would be, if we weren't all busy ignoring our ignorance.