Tuesday, April 23, 2013

A lesson on lessening, from economics

By now we've all heard that an Excel spreadsheet error nearly brought down the world economy.  It has been reported by New York Magazine, for example, and the BBC, The Economist, among many other places.  It's a sobering story, a cautionary tale not only about economics but also about science and belief. It's an unwelcome caution, but one we should heed.

Confronting trying economic times, the question became whether governments should spend their way out of the crisis or cut spending to manage the crisis -- should they go the way of Hayek and embrace austerity or Keynes, and increase their spending.  Austerity was the way many countries chose to go -- too many, according to Keynesians, of course -- buoyed by a study done by Harvard economics professors, Carmen Reinhart and  Ken Rogoff, former chief economist of the International Monetary Fund, who delivered their results in a talk called 'Growth in a Time of Debt' at an economics meeting in 2010 (subsequently published here).  They reported that economic growth slows dramatically when a country's debt is more than 90 percent of it's gross domestic product.  Indeed, that there is a "non-linear response" to debt. 

That seemed clear enough, and strong justification for austerity.  In fact, there was a widespread near-panic about the catastrophe that would ensue if budgets were not cut drastically, and quickly, and of course the debate is ongoing as Greece, Spain, and other European countries struggle to right their economies.

But a graduate student at UMASS/Amherst, Thomas Herndon, tried to replicate the Reinhart/Rogoff study, and could not.  He repeatedly wrote to Reinhart and Rogoff to ask for their data, and to his surprise eventually they sent him the original spreadsheet in which they'd made their calculations.  That's when Herndon found that they'd mistakenly neglected to include five major nations in their figures and had selectively included data sets in their calculations in a way that seemed, to Herndon, to yield results that were, at best, disputable, and called their conclusions into question.  Herndon and colleagues wrote up their findings in a paper published on April 15, that has gotten huge play.

Here's the abstract from that paper:
Herndon, Ash and Pollin replicate Reinhart and Rogoff [RR] and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogo ff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower.
The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff 's claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.
They conclude the paper saying, "Specfically, RR's findings have served as an intellectual bulwark in support of austerity politics. The fact that RR's findings are wrong should therefore lead us to reassess the austerity agenda itself in both Europe and the United States."

From Herndon et al.
Reinhart and Rogoff have responded (e.g., the Monday 22 episode of BBC Radio 4's More or Less and at length here) thanking Herndon et al. for finding the error, but reiterating their conclusion that high debt and slowed growth go hand-in-hand.  "We do not...believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."  Though we must say that, to our eye, the figure above from the Herndon et al. paper suggests the correlation is weak, at best.

The Economist has published a comparison of the original and the revised figures.  You decide whether you think the differences are significant or not.  Note that for the 'Above 90' value, which is the issue at hand, the RR report showed a negative mean value, whereas the corrected value is strongly positive.  So the average growth rate for countries with debt above 90% should have been 2.2 rather than -0.1.  Whatever your interpretation of the critique and what it means about austerity measures around the globe, we'd bet that it has a lot to do with how you felt about austerity measures before the Herndon paper came out.  And correlates strongly with how you voted.
Published April 17, The Economist
This is clearly another nail in the coffin of the self-flattering myth that science is about objectively doing one's best to falsify his/her hypotheses. No one's going to change their mind.  Not politicians.  Not even scientists.

An analogy: The 9/12 Syndrome
On 9/11, the US was infamously attacked in what is still often described as the worst way in our history (well, that's if you don't count the Revolutionary War, the Civil War, or the genocidal wars we waged on the Native Americans).  We were attacked by rabid fundamentalists who somehow thought that killing people flying on business or vacation was a way to correct some political wrongs they imagined that we were doing.

Americans shared their shock and horror at these attacks.  But how was this tragedy explained?  On 9/12, the day after the attacks, the punditry crept out of the woodwork and.....not a single person's ideas were changed!  If you were a gun-toting right-winger, you said "See, I told you we needed to be getting tough with the rest of the world!"  But if you a dove-releasing left-winger, you said "See, I told you we should not have been being the world's bully!"

The events were used, after the fact, to reinforce polar opposite opinions by those who had been waging political battles to advance their views. The reason is that people simply have a hard time seeing somebody's view other than their own and that of their friends.  We in science are human (despite our occasional claims to superiority) and are vulnerable to exactly the same kind of complacency.  Bragging, not apoligizing, is rather too much our way of life.

In the case of recent economics, hugely negative effects have resulted, because politicians bought into convenient ideas, in part citing this influential 'research' in their support.  The word's in quotes because it's treated by the public, politicians, and scientists as if it were the same as 'gospel.  But who knows how many thousands--or millions--of people lost homes or jobs, were driven into crime, disease, divorce or dispair and the like, or even died because of lost access to affordable medical care, because government policy did not come to their rescue--because of a polarized commitment to some preconception?

The lesson is to lessen our claims, not just to adopt things uncritically if they fit our preconceptions.  It's the hardest kind of lesson to learn, in a society that does not reward modesty.  Still, we should do it.


Anonymous said...

I'm genuinely curious, are there any societies that do reward/have rewarded modesty?

Another thought...although our society doesn't reward modesty, there is some respect in certain circles for non-modest skepticism. For the kind of person who confidently pokes holes in the work of other people. This UMASS student got a lot of attention didn't he.

Anne Buchanan said...

Actually rewarding modesty might be a bit oxymoronic. But there are certainly societies in which it's understood that it's best not to stand out. And self-crowing is scorned.

Agreed, non-modest skepticism has some cache. But this story has appeal from multiple angles -- these famous professors made a careless mistake, one that's simple to make and we all could have done it, it was caught by a mere student, and not even an Ivy League student at that, and it fits into a current, ongoing, emotional political debate. The student got a lot of attention, but he's appealingly modest about it (even if he did get a new PR photo).

Ken Weiss said...

Everyone has an ego and many at least want to be recognized, and societies have hierarchies. But even in our own society, going public to the extent at present is relatively new. This I can say from my own lifetime experience.

Many indigenous societies were known to anthropologists in which overt 'winning' and so on were discouraged.

Even in our Euro-American societies, leaders of business and even aristocrats didn't seek publicity. Professors published their work and wanted to be known for it, but they didn't generally call press conferences; journals didn't have gossip columns; journals were published in black and white without catchy article titles blazoned across their cover.

The exceptions stood out, in a way, because they were exceptions. Part of this was because we had less, and slower, media and fewer professors doing less research. But even the prominent ones didn't typically, I think, try to reach for attention from the general public.

This isn't to say our forebears were all without self-interest, of course.

Your final point is part of the same phenomenon: being an outspoken critic or claiming to have made a 'paradigm shifting' discovery that overturns all of prior history is also rewarded by the same system.

Anonymous said...

Thanks for the thoughtful responses.

Anne Buchanan said...

After the Herndon et al. paper appeared, Reinhart and Rogoff responded several times, saying, basically that Herndon et al. confirmed their results (links in our post). Herndon refutes this categorically in a piece in yesterday's Business Insider, here. He also explains in greater detail the ripple effect of R&R's Excel error on others of their calculations.

Finally, another piece in Business Insider (Joe Weisenthal) suggests that "the austerity movement is crumbling" as a result of Herndon et al.'s critique. "... the case for trying to cut debt just for the sake of cutting debt is looking a lot more tenuous." We'll see...

Ken Weiss said...

A way to look at this is as follows:
When causation is probabilistic, or at least we don't understand it in clear mechanistic terms, we use sampling and statistical tests to evaluate data.

The purpose of statistical tests is to change our behavior--to lead us to accept, or not accept, some hypothesis. The statistical cutoff ('significance') values are the line in the sand that we ourselves chose to affect our future behavior.

But if we do a study, and get results we like, we change our behavior accordingly. But if we don't like the results (or they change when an error is discovered), the proper response is to accept Nature's verdict and change our actions.

In this case, our actions would have to do with the politics of austerity.

But, of course, those who like austerity won't do that. Instead, as is already being seen, they'll rationalize the error as minor, and/or change the goal posts so their conclusions still score....

Anne Buchanan said...

And I didn't even link to the piece that says that nothing will change, nothing ever has and nothing ever will. I think I get the point (I suspect it's the point we make in this post but it was hard to tell), but of course things do in fact change. Explaining why, in response to what, or predicting what will motivate change is the issue. Just like in genetics.

Ken Weiss said...

A bit of a tempering note. A BBC Radio program called More or Less, that critiques numerical claims made by politicians and others, discussed the RR paper and its critics. They pointed out, and it's only fair for us to note that, that there are other aspects of evidence that suggest a relationship between debt and economic woes. The effects are less clear and certainly much smaller than the RR report.

The issue was the iconic way that the RR report was seized upon by supporters of debt reduction (who also are generally people against social nets or other forms of government programs or protections, or interference with one's attempt to make as much money as one can).

The science point is similar: there are people wedded to strong genetic determinism or who see natural selection under every bed. They, too, seize on any scrap of evidence and deny contrary uncertainties about causation. And those, like us, who favor a far more tempered view, need to beware that we don't seize on every uncertainty in a similarly polarized way. It's tough to do. In life, there are instances across the board, and the discussion needs to focus on their relative frequency or importance, and on developing a consistent theoretical understanding of life--and of economics.

Anne Buchanan said...

I agree that tempering is needed when one is tempted to let politics drive his/her interpretations of science, (or 'science' -- is economics a science?). But in this case I've read the papers and responses and I have to agree with Herndon that his paper unequivocally does not confirm R&R.

Anne Buchanan said...

And of course Stephen Colbert says it best.