Monday, February 23, 2015

When the methodology fails

Aop-ed piece by Nina Teicholz in Friday's NYTimes lays it on the line, chastising the government for its regular bulletins on dietary advice that, for 50 or so years have altered what we eat, what we fear to eat, and what the risks are.  Now, new studies tell us that what was bad is good and what was good is bad, and that the prior half-century of studies were wrong.  We've eliminated fats and cholesterol, and replaced them with carbohydrates, but, as Teicholz writes,
...recent science has increasingly shown that a high-carb diet rich in sugar and refined grains increases the risk of obesity, diabetes and heart disease — much more so than a diet high in fat and cholesterol.
But why should we believe these new studies?  Teicholz basically takes the underlying methodology to task, and yet she has written a book recommending that we eat more fats (“The Big Fat Surprise: Why Butter, Meat and Cheese Belong in a Healthy Diet"), but those recommendations are based on the very same faulty methodology as the recommendations with which she, and the current USDA advisory committee, find fault.

Embrace the fat! (Wikipedia)

The same, almost exactly the same, critiques are earned by many of the 'big data' genomics studies (and other long-term go-not-very-far megaprojects).  It is the statistical correlation methodology.  When many factors are studied at once (perhaps properly since many factors, genetic and environmental, are responsible for health or other traits), we can't expect simple answers.  We can't expect correlation to imply causation.  We can't expect replication.  We can't predict the risk factors that people, for whom risk advice is based on such studies, will face in the future.

The real conclusion is to shut down the nutrition megaprojects at Harvard (singled out by the op-ed) and the other genetics and public health departments that have been running them for decades, and do something different.  The megaprojects have become part of the entrenched System, with little or no real accountability.

Pulling the plug would be a major acknowledgment of failure, both by the feds for what they funded, the program officers for defending weak portfolios and their budgets, the universities defending their overhead and prestige projects and, of course, the investigators who are either simply unable to recognize what they're doing, or too dishonest and self-protecting to come clean about it.  And then they and their students could go on to do something actually productive.

Of course such a multi-million dollar threat will be resisted, and that's why the usual answer to the kinds of conflicting, confusing reports that so often come out of these megaprojects is to increase their size, length and, geez, what a surprise!, their cost.  To keep funding the same investigators and their protegés.  This is only to be expected, and many people's jobs are covered by the relevant grants, a genuine concern.  However, research projects are not supposed to be part of a welfare system, but to solve real problems.  And the same peoples' skills could be put to better use, addressing real problems in ways that might be more effective and accountable.

And we used to laugh at the Soviets' entrenched, never successful, Five Year Plans!

It is a public misappropriation that is taking place.  Yes, there are health problems we wish to avoid, and government and universities are set up to identify them and recommend changes.  But, for most of today's common chronic diseases, lifestyle changes would largely do the trick.

But then, that would just let people live longer to get diseases that might be worse, even if at older ages.  And meanwhile we aren't putting on a full court press for things that really are genetic, or really do have identifiable life-style causes.

Much of this research is being done at taxpayer expense.  We should let the people keep their money, or we should spend it more effectively.  We won't be able to do the latter until we admit, formally and fully, that we have a problem.  Given vested and entrenched interests, getting that to happen is a very hard trick to pull off.


Anonymous said...

Has anyone of the ecodevoevo crew commented on this latest GCTA from Plomin labs?

I know you've commented on previous GCTAs out of Scotland on the genetic underpinnings of cognitive ability, so I'm curious your views on this.

Ken Weiss said...

No we hadn't seen that. Plomin is determined to find that intelligence and so on are 'genetic' and this paper further documents that genes are contributing to the heritability. The paper acknowledges that the trait is polygenic. Whether the specific genes, or something else in the region, are responsible would be open, but there's no reason to doubt that identifiable genes are among the contributors.

However, the overall story is that these have little individual effect (the trait is polygenic) so, to me, the response is that to address issues or problems related to intelligence or cognitive achievement and success--the legitimate targets of public research--we should be moving past genomic studies towards something more effective for real people. Genetic studies can be done rather mechanically, and are easy to design, etc., and have newsworthy apparent panache; but they are not going any where and we can see that.

Addressing the net result of complex, individually unique genomes is the real problem and I think it is a real, hard problem.

So I don't question the quality of the paper's work, but how many of these papers do we need before someone is stimulated to think more deeply about the problem--or what 'problem' there really is?

Anonymous said...

I think this was most important in proving that individual differences in intelligence / cognitive ability, are genetic in background. This was known for some time, though. We can point out the SNPs now, which account for 29% of that.

Ken Weiss said...

Well, I guess I don't really agree. First, this is not the first intelligence GWAS, so if each one identifies some of the same genes (or genome locations), and we know that each genotype is different--clearly this is so, since no two people have all the same marker diploid genotypes. And different studies basically find different 'hit' locations. If you have even just a few locations each with just 2 variants at non-trivial frequencies, the probability of the same genotype is very small. We know that and we know that also from the failure of intelligence measures to segregate.

I dispute the 29% figure in the sense that different studies will have different hits accounting for different fractions, almost always small. And this also discounts other correlation causes, which we know are important, and secular trends in environmental factors like 'education' in its various forms.

So we can point to SNPs accounting for some of this in aggregate, but we have already known of many different genes whose mutant forms lead to cognitive impairment.

I guess I'm more of a nihilist in this context, in regard to the enumerability of 'causes' of such a complex trait. Or, to put it more positively, we have known this in essence for many decades. Where does identifying a potpourri of minor SNPs get us? Why not move on to better science or a more focused addressing of a real question?

Anonymous said...

Do you think this gets us closer to understanding to true genetic underpinnings of the g factor? If not, how can it be done?

Ken Weiss said...

I think it gets us only a miniscule bit closer in terms of specific genetic contributors. After all, what can we currently do (besides prenatal screening), about the many known contributing genes to coginitive function?

In my personal view, only if or when we are willing to acknowledge the genotypic non-specificity, and complexity, will we start thinking more creatively.

If enumerable genotypes are so complex that everyone is substantially unique (except for the clear pathogenic single locus instances), then we have to think of the problem differently.

But it won't be easy even then, because many behavioral/psychological traits are clear, severe, and manifest....and still 'map' in a similarly complex way.

We are not defining traits very appropriately. Too much isn't known. I don't think more and more of this sort of mapping is going to get very far, even if it will continue to be done (and, in my view, wasting funds that could be invested in more cogent science).

Jari Stengard said...


Few questions and a comment:

1) how many new statistical studies we need to reject a concensus of a research community, an opinion that is based on a data collected within last 50 years?

2) is research context free? We live totally different world than people lived when cholesterolo hypothsesi was settled. Can it be that cholesterol hypothesis was correct at that time and now, when industrialized food peoduction feed us with lots of carbohydrates (and hidden fats and transfats) carbohydrates are/look more dangerous than traditional fats.

3) you are also mixing here two things, science and practice. Scientists may find some facts about predictors of disease endpoints and/or genetic architecture of common complex disease but suchs facts do not tell whether it implies any practical acts and how and when and where suxh acts should be performed.


Ken Weiss said...

Your points are all right on the mark! Except for one thing: this isn't science, this is politics and largely money politics. You have to keep producing papers to keep getting funded, etc. This is the system we've built, and it's not just restricted to science.

For the same reason, while investigators do sincerely want to have their work lead to some practical good, the main objective is to keep the lab going. If you don't have a good idea, whatever idea you do have has to be promoted. Clinical good comes when it comes.

To be fairer, even in this very greed-based system, the problems are hard and only a few people can think innovatively. We just have to wait around for that to happen (and feel sorry that we, ourselves, aren't the ones with the Great idea!).

James Goetz said...

I feel like a guinea pig because I struggle with my blood pressure and more recently my blood sugar while I want to make medically informed lifestyle decisions that will keep me living healthy, but these changing trends in health science make this difficult and frustrating for me....

David J. Littleboy said...

"these changing trends in health science make this difficult and frustrating for me...."

It's worse than that. It turns out that the recommendations that we limit salt and fat consumption were _always_ wrong. (Note that there's no argument that trans fats are bad, though.)

Anonymous said...

"We can point out the SNPs now, which account for 29% of that."

That is demonstrably incorrect. That 29% and 28% was from GCTA in two different samples which only correlated genes that could account for 29-28% of the traits in those samples. They did not find the genes that caused said 28% which is an assumed estimate. All it says is that 28% could be caused by the genes they correlated.