A lesson on lessening, from economics

By now we've all heard that an Excel spreadsheet error nearly brought down the world economy.  It has been reported by New York Magazine, for example, and the BBC, The Economist, among many other places.  It's a sobering story, a cautionary tale not only about economics but also about science and belief. It's an unwelcome caution, but one we should heed.

Confronting trying economic times, the question became whether governments should spend their way out of the crisis or cut spending to manage the crisis -- should they go the way of Hayek and embrace austerity or Keynes, and increase their spending.  Austerity was the way many countries chose to go -- too many, according to Keynesians, of course -- buoyed by a study done by Harvard economics professors, Carmen Reinhart and  Ken Rogoff, former chief economist of the International Monetary Fund, who delivered their results in a talk called 'Growth in a Time of Debt' at an economics meeting in 2010 (subsequently published here).  They reported that economic growth slows dramatically when a country's debt is more than 90 percent of it's gross domestic product.  Indeed, that there is a "non-linear response" to debt. 

That seemed clear enough, and strong justification for austerity.  In fact, there was a widespread near-panic about the catastrophe that would ensue if budgets were not cut drastically, and quickly, and of course the debate is ongoing as Greece, Spain, and other European countries struggle to right their economies.

But a graduate student at UMASS/Amherst, Thomas Herndon, tried to replicate the Reinhart/Rogoff study, and could not.  He repeatedly wrote to Reinhart and Rogoff to ask for their data, and to his surprise eventually they sent him the original spreadsheet in which they'd made their calculations.  That's when Herndon found that they'd mistakenly neglected to include five major nations in their figures and had selectively included data sets in their calculations in a way that seemed, to Herndon, to yield results that were, at best, disputable, and called their conclusions into question.  Herndon and colleagues wrote up their findings in a paper published on April 15, that has gotten huge play.

Here's the abstract from that paper:

Herndon, Ash and Pollin replicate Reinhart and Rogoff [RR] and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogo ff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower.

The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff 's claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.

They conclude the paper saying, "Specfically, RR's findings have served as an intellectual bulwark in support of austerity politics. The fact that RR's findings are wrong should therefore lead us to reassess the austerity agenda itself in both Europe and the United States."

From Herndon et al.

Reinhart and Rogoff have responded (e.g., the Monday 22 episode of BBC Radio 4's More or Less and at length here) thanking Herndon et al. for finding the error, but reiterating their conclusion that high debt and slowed growth go hand-in-hand.  "We do not...believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."  Though we must say that, to our eye, the figure above from the Herndon et al. paper suggests the correlation is weak, at best.

The Economist has published a comparison of the original and the revised figures.  You decide whether you think the differences are significant or not.  Note that for the 'Above 90' value, which is the issue at hand, the RR report showed a negative mean value, whereas the corrected value is strongly positive.  So the average growth rate for countries with debt above 90% should have been 2.2 rather than -0.1.  Whatever your interpretation of the critique and what it means about austerity measures around the globe, we'd bet that it has a lot to do with how you felt about austerity measures before the Herndon paper came out.  And correlates strongly with how you voted.
 

Published April 17, The Economist

This is clearly another nail in the coffin of the self-flattering myth that science is about objectively doing one's best to falsify his/her hypotheses. No one's going to change their mind.  Not politicians.  Not even scientists.

An analogy: The 9/12 Syndrome
On 9/11, the US was infamously attacked in what is still often described as the worst way in our history (well, that's if you don't count the Revolutionary War, the Civil War, or the genocidal wars we waged on the Native Americans).  We were attacked by rabid fundamentalists who somehow thought that killing people flying on business or vacation was a way to correct some political wrongs they imagined that we were doing.

Americans shared their shock and horror at these attacks.  But how was this tragedy explained?  On 9/12, the day after the attacks, the punditry crept out of the woodwork and.....not a single person's ideas were changed!  If you were a gun-toting right-winger, you said "See, I told you we needed to be getting tough with the rest of the world!"  But if you a dove-releasing left-winger, you said "See, I told you we should not have been being the world's bully!"

The events were used, after the fact, to reinforce polar opposite opinions by those who had been waging political battles to advance their views. The reason is that people simply have a hard time seeing somebody's view other than their own and that of their friends.  We in science are human (despite our occasional claims to superiority) and are vulnerable to exactly the same kind of complacency.  Bragging, not apoligizing, is rather too much our way of life.

In the case of recent economics, hugely negative effects have resulted, because politicians bought into convenient ideas, in part citing this influential 'research' in their support.  The word's in quotes because it's treated by the public, politicians, and scientists as if it were the same as 'gospel.  But who knows how many thousands--or millions--of people lost homes or jobs, were driven into crime, disease, divorce or dispair and the like, or even died because of lost access to affordable medical care, because government policy did not come to their rescue--because of a polarized commitment to some preconception?

The lesson is to lessen our claims, not just to adopt things uncritically if they fit our preconceptions.  It's the hardest kind of lesson to learn, in a society that does not reward modesty.  Still, we should do it.