A lesson on lessening, from economics

By now we've all heard that an Excel spreadsheet error nearly brought down the world economy.  It has been reported by New York Magazine, for example, and the BBC, The Economist, among many other places.  It's a sobering story, a cautionary tale not only about economics but also about science and belief. It's an unwelcome caution, but one we should heed.

Confronting trying economic times, the question became whether governments should spend their way out of the crisis or cut spending to manage the crisis -- should they go the way of Hayek and embrace austerity or Keynes, and increase their spending.  Austerity was the way many countries chose to go -- too many, according to Keynesians, of course -- buoyed by a study done by Harvard economics professors, Carmen Reinhart and  Ken Rogoff, former chief economist of the International Monetary Fund, who delivered their results in a talk called 'Growth in a Time of Debt' at an economics meeting in 2010 (subsequently published here).  They reported that economic growth slows dramatically when a country's debt is more than 90 percent of it's gross domestic product.  Indeed, that there is a "non-linear response" to debt. 

That seemed clear enough, and strong justification for austerity.  In fact, there was a widespread near-panic about the catastrophe that would ensue if budgets were not cut drastically, and quickly, and of course the debate is ongoing as Greece, Spain, and other European countries struggle to right their economies.

But a graduate student at UMASS/Amherst, Thomas Herndon, tried to replicate the Reinhart/Rogoff study, and could not.  He repeatedly wrote to Reinhart and Rogoff to ask for their data, and to his surprise eventually they sent him the original spreadsheet in which they'd made their calculations.  That's when Herndon found that they'd mistakenly neglected to include five major nations in their figures and had selectively included data sets in their calculations in a way that seemed, to Herndon, to yield results that were, at best, disputable, and called their conclusions into question.  Herndon and colleagues wrote up their findings in a paper published on April 15, that has gotten huge play.

Here's the abstract from that paper:

Herndon, Ash and Pollin replicate Reinhart and Rogoff [RR] and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogo ff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower.

The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff 's claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.

They conclude the paper saying, "Specfically, RR's findings have served as an intellectual bulwark in support of austerity politics. The fact that RR's findings are wrong should therefore lead us to reassess the austerity agenda itself in both Europe and the United States."

From Herndon et al.

Reinhart and Rogoff have responded (e.g., the Monday 22 episode of BBC Radio 4's More or Less and at length here) thanking Herndon et al. for finding the error, but reiterating their conclusion that high debt and slowed growth go hand-in-hand.  "We do not...believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."  Though we must say that, to our eye, the figure above from the Herndon et al. paper suggests the correlation is weak, at best.

The Economist has published a comparison of the original and the revised figures.  You decide whether you think the differences are significant or not.  Note that for the 'Above 90' value, which is the issue at hand, the RR report showed a negative mean value, whereas the corrected value is strongly positive.  So the average growth rate for countries with debt above 90% should have been 2.2 rather than -0.1.  Whatever your interpretation of the critique and what it means about austerity measures around the globe, we'd bet that it has a lot to do with how you felt about austerity measures before the Herndon paper came out.  And correlates strongly with how you voted.
 

Published April 17, The Economist

This is clearly another nail in the coffin of the self-flattering myth that science is about objectively doing one's best to falsify his/her hypotheses. No one's going to change their mind.  Not politicians.  Not even scientists.

An analogy: The 9/12 Syndrome
On 9/11, the US was infamously attacked in what is still often described as the worst way in our history (well, that's if you don't count the Revolutionary War, the Civil War, or the genocidal wars we waged on the Native Americans).  We were attacked by rabid fundamentalists who somehow thought that killing people flying on business or vacation was a way to correct some political wrongs they imagined that we were doing.

Americans shared their shock and horror at these attacks.  But how was this tragedy explained?  On 9/12, the day after the attacks, the punditry crept out of the woodwork and.....not a single person's ideas were changed!  If you were a gun-toting right-winger, you said "See, I told you we needed to be getting tough with the rest of the world!"  But if you a dove-releasing left-winger, you said "See, I told you we should not have been being the world's bully!"

The events were used, after the fact, to reinforce polar opposite opinions by those who had been waging political battles to advance their views. The reason is that people simply have a hard time seeing somebody's view other than their own and that of their friends.  We in science are human (despite our occasional claims to superiority) and are vulnerable to exactly the same kind of complacency.  Bragging, not apoligizing, is rather too much our way of life.

In the case of recent economics, hugely negative effects have resulted, because politicians bought into convenient ideas, in part citing this influential 'research' in their support.  The word's in quotes because it's treated by the public, politicians, and scientists as if it were the same as 'gospel.  But who knows how many thousands--or millions--of people lost homes or jobs, were driven into crime, disease, divorce or dispair and the like, or even died because of lost access to affordable medical care, because government policy did not come to their rescue--because of a polarized commitment to some preconception?

The lesson is to lessen our claims, not just to adopt things uncritically if they fit our preconceptions.  It's the hardest kind of lesson to learn, in a society that does not reward modesty.  Still, we should do it.

Fasification of the CFS/ME retrovirus link

The cause of chronic fatigue syndrome, or myalgic encephalomyelitis (CFS/ME) has not yet been identified, but a non-cause has been.  A 2009 paper in Science suggested that the disorder might be caused by an infectious retrovirus.  This made sense clinically because many patients report having a viral illness before becoming ill with CFS/ME.  From the 2009 paper:

Chronic fatigue syndrome (CFS) is a debilitating disease of unknown etiology that is estimated to affect 17 million people worldwide. Studying peripheral blood mononuclear cells (PBMCs) from CFS patients, we identified DNA from a human gammaretrovirus, xenotropic murine leukemia virus–related virus (XMRV), in 68 of 101 patients (67%) as compared to 8 of 218 (3.7%) healthy controls. Cell culture experiments revealed that patient-derived XMRV is infectious and that both cell-associated and cell-free transmission of the virus are possible. Secondary viral infections were established in uninfected primary lymphocytes and indicator cell lines after their exposure to activated PBMCs, B cells, T cells, or plasma derived from CFS patients. These findings raise the possibility that XMRV may be a contributing factor in the pathogenesis of CFS.

The investiagors also found evidence of pMLV (polytropic murine leukemia virus) in some patients. This work gave many people suffering from this disease hope that now the cause was known, and so a cure might be just on the horizon.  Many began to take anti-retrovirals in the belief that, as for HIV, this could stave off the illness.  Clinics were established to treat CFS/ME with anti-retrovirals, and public health concerns were raised over the safety of the blood supply with respect to XMRV and pMLV. 

But many scientists weren't convinced that XMRV was the explanation, and a number of subsequent studies failed to confirm the finding.  And to make things even more confusing, some labs could readily detect the virus and others couldn't find it at all.  We (and many others) blogged about the confusion at the time, here and here. Indeed, analysis of the complete XMRV genomic sequence suggested that the virus was an artifact of a laboratory recombination event that took place around 1993-1996, and these investigators concluded "that association of XMRV with human disease is due to contamination of human samples with virus originating from this recombination event."

Well, now a new multi-institutional study now proposes a definitive answer.  The investigators who originally found the retroviral association "report that a blinded analysis of peripheral blood from a rigorously characterized, geographically diverse population of 147 patients with CFS/ME and 146 healthy subjects" finds no evidence of XMRV or pMLV. 

The paper concludes that the original finding was an artifact of the use of sensitive molecular methods, PCR, that can essentially make mountains out of molehills.  Small amounts of contaminant can appear to be causative with these techniques and pose expensive difficulties for public health as laboratories go through the "pathogen dediscovery process."

We have often said here on MT that although one of the cornerstones of the scientific method is falsiability (yes, this can be and has been disputed, but it is certainly still considered by many to be so).  Even so, many scientists cling to their original hypotheses even in light of very well-demonstrated refuting evidence.  Thus, it's good to see this statement now, from the author of the 2009 Science paper that suggested the link between CFS/ME and XMRV, as reported in ScienceDaily. 

 "I greatly appreciated the opportunity to fully participate in this unprecedented study. Unprecedented because of the level of collaboration, the integrity of the investigators, and the commitment of the NIH to provide its considerable resources to the CFS community for this important study. Although I am disappointed that we found no association of XMRV/pMLV to CFS, the silver lining is that our 2009 Science report resulted in global awareness of this crippling disease and has sparked new interest in CFS research. I am dedicated to continuing to work with leaders in the field of pathogen discovery in theeffort to determine the etiologic agent for CFS."

The problem with falsifiability
Falsifiability was advocated, largely by the philosopher Karl Popper, on the grounds that induction--repeatedly observing the same thing--was not a reliable criterion for inferring something about the world.  The reason is that the next observation could be different and ruin the generalization that had seemed true.

The idea is that one can never show something to be true--indeed, tomorrow the same data could be shown to fit some different idea from the one you had.  But, at least, you could show your idea to be false.  That is, you set up a test and expect the result your generalization would predict....but you get some other result.  That leads you to develop a different idea (hypothesis) about what is going on.  
If one idea after another is falsified in this way, then this is a systematic way to approach knowledge, because whatever explanation remains is more likely to be the true one (this is essentially what Sherlock Holmes said, long before Popper ever came up with the idea!). 

But experiments can fail, studies can be poorly designed, and so on.  As we noted, scientists tend to use those explanations when the result isn't what they want: we need bigger studies!  This negative study was not properly designed!  The methods were not up to date!   And so on...  This makes science a social and political, rather than strictly objective and empirical, form of human endeavor.

That's why falsification is just not a definitive criterion. In this case, the author has given up on the idea  and is just grateful that people are thinking about the problem.  Previous studies that did not confirm the hypothesis, however, were not considered robust enough to actually falsify it. 

The deeper problem is that we have no definitive criteria for making conclusions in science.  We do it by some informal, usually temporary, consensus.  Maybe there will be found some truly definitive way to make inferences.  Or, maybe science as a social and political activity is either the right way, or even the best way we can get.

Taking science on faith

There's a news story this week in the journal, Nature, entitled "Classical behavioural studies flawed".

"One of the most famous experiments in biology isn't the solid piece of work it's usually portrayed as, say Dutch researchers who have replicated the study. Instead, it's more like an anecdote that became slightly more legendary each time its author retold the story.

The work in question was done in 1947 by the Dutch researcher Niko Tinbergen on the begging behaviour of herring-gull chicks. At the time, the dominant idea in animal behaviour was that learning was all-important. Tinbergen argued that animals co into the world with instincts already adapted to their environments." (The photo is taken from the Wikipedia article on herring gulls.)

Tinbergen's idea became a classical textbook study that guided much of the conceptual thinking in the field of 'ethology' as it was then called, and led to much Darwinian and hence genetic determinism and what has become evolutionary psychology. The finding was that chicks hatch with the instinct to peck at red, the color of a spot on the adult herring gull's bill, to coax their parents into feeding them. His experiments, however, showed that chicks were more likely to peck at any color other than red. Over the course of writing up the results for 10 different publications, he slightly altered the way he described his results so that he explained away those findings -- he said, for example, that he had showed them red more often than other colors and he presumed that this caused them to become acclimated to it -- but he always concluded that they prefer red. He wasn't making up data -- he was explaining away results he didn't like. The Nature authors in fact conclude that he was essentially correct (arguments about such studies will be left for another day), but of course he didn't know that at the time.

The thrust of the Nature story is that we shouldn't call this sloppy science because science was less rigorous then. Even Darwin's work would be pummeled if it were to be peer reviewed today. While that's probably true, it's only partly true. Modern criteria for inference and study design were not invented last year, but have been around for some time. Darwin did not have many of the tools, such as statistical methods, that we use but (as Mendel, Morgan, and many others, including epidemiologists showed) systematic, well-controlled inferential study designs were certainly known.

Good science or not, this is an example of how scientists, who are supposed to always be open to having their results falsified, indeed who (it is often taught in class) go out of their way to falsify their 'hypotheses', in fact tend to cling tightly to their conclusions in spite of contradictory results. Even in the era of 'more rigorous' science, if not significant, results are often described as 'nearly significant' or 'suggestive'. Outliers are discarded. Forced, post hoc reasoning is often offered to explain away weak results or exceptions, and of course the problem always warrants further study. But science is a human endeavor -- more often than scientists like to admit, results are taken on faith.