Showing posts with label replication. Show all posts
Showing posts with label replication. Show all posts

Wednesday, November 28, 2018

Induction-deduction, and replicability: is there any difference?

In what sense--what scientific sense--does the future resemble the past?  Or perhaps, to what extent does it?  Can we know?  If we can't, then what credence for future prediction can we give to results of studies today, necessarily from the past experience of current samples?  Similarly, in what sense can we extrapolate findings on this sample to some other sample or population?  If these questions are not easily answerable (indeed if they are answerable at all!), then much of current, and currently very widespread and expensive science, is at best of unclear, questionable value.

We can look at these issues in terms of a couple of standard aspects of science: the relationship between induction and deduction; and the idea of replicability.  Induction and deduction basically come from the Enlightenment time in western history, when it was found in a formal sense that the world of western science--which at that time meant physical science--followed universal 'laws' of Nature.  At that time, life itself was generally excluded from this view, not least because it was believed to be the result of ad hoc creation events by God.

The induction--deduction problem
-----------------
Some terminology:  I will make an important distinction between two terms.  By induction I mean drawing a conclusion from specific observed data (e.g., estimating some presumed causal parameter's value).  Essentially, this means inferring a conclusion from the past, from events that have already occurred. But often what we want to do is to predict the future.  We do that, often implicitly, by equating observed past values as estimates of causal parameters, that apply generally and therefore to the future; I refer to that predictive process, derived from observed data, as deduction.  So, for example, if I flip a coin 10 times and get 5 Heads, I assume that this is somehow built into the very nature of coin-flipping so that the probability of Heads on any future flip is 0.5 (50%).
-----------------

If we can assume that induction implies deduction, then what we observe in our present or past observations will persist so that we can predict it in the future.  In a law-like universe, if we are sampling properly, this will occur and we generally assume this means with complete precision if we had perfect measurement (here I speculate, but I think that quantum phenomena at the appropriate scale have the same universally parametric properties).

Promises like 'precision genomic medicine', which I think amount to culpably public deceptions, effectively equate induction with deduction: we observe some genomic elements associated in some statistical way with some outcome, and assume that the same genome scores will similarly predict the future of people decades from now.  There is no serious justification for this assumption at present, nor quantification of by how much there might be errors in assuming the predictive power of past observations, in part because mutations and lifestyle clearly have major effects, but especially because these are unpredictable--even in principle.  Indeed, there is another, much deeper problem of a similar kind, that has gotten recent--but to me often quite naive attention: replicability.

The replicability problem
Studies, perhaps especially in social and behavioral fields, report findings that others cannot replicate.  This is being interpreted as suggesting that (ignoring the rare outright fraud), there is some problem with our decision-making criteria, other forms of bias, or poor study designs.  Otherwise, shouldn't studies of the same question agree?  There has been a call for the investigators involved to improve their statistical analysis (i.e., keep buying the same software!! but use it better), report negative results, and so on.

But this is potentially, and I think fundamentally, naive.  It assumes that such study results should be replicable.  It assumes, as I would put it, that at the level of interest, life = physics.  This is, I believe not just wrong but fundamentally so.

The assumption of replicability is not really different from equating induction to deduction, except in some subtle way applied to a more diverse set of conditions.  Induction of genomic-based disease risk is done on a population like, say, case-control samples, and then applied to the same population in terms of its current members' future disease risks.  But we know very well that different genotypes are found in different populations, so it is not clear what degree of predictability we should, or can, assume.

Replicability is similar except that in general a result is assumed to apply across populations or samples, not just to the same sample's future.  That is, I think, an even broader assumption than the genomics-precision promise that does, at least nominally, now recognize population differences.

The real, the deeper problem is that we have absolutely no reason to expect any particular degree of replicability between samples for these kinds of things.  Evolution is about variation, locally responsive and temporary, and that applies to social behavior as well.  We know that 'distance' or difference accumulates (generally) gradually over time and separation as a property of cultural as well as biological evolution.  The same obviously applies even more to psychological and sociological samples and inferences from them.

Not only is it silly to think that samples of, say, this year's college seniors at X University will respond to questionnaires in the same way as samples of some other class or university or beyond.  Of course, college students come cheap to researchers, and they're convenient.  But they are not 'representative' in the replicability sense except by some sort of rather profound assumption.  This is obvious, yet it is a tacit concept of very much research (biological, psychological, and sociological).

Even social scientists acknowledge the local and temporary nature of many of the things they investigate, because the latter are affected by cultural and historical patterns, fads, fashions, and so much more.  Indeed, the idea of replicability is to me curious to begin with.  Thus, a study that fails to replicate some other study may not reflect failings in either, and the idea that we should replicate in this kind of way is a carryover of physics envy.  Perhaps in many situations, a replication result is what should be examined most closely! The social and even biological realms are simply not as 'Newtonian', or law-like, as is the real physical realm in which our notions of science--especially the very idea of a law-like replicability, arose. Not only is failure to replicate not necessarily suspect at all, but replicability should not generally be assumed.  Or, put an other way, a claim that replicability is to be expected is a strong claim about Nature that requires very strong evidence!

This raises the very deep problem that in the absence of replicability assumptions, we don't know what to expect of the next study, after we've done the first.....or is this a justification for just keeping the same studies going (and funded) indefinitely?  That's of course the very rewarding game being played in genomics.

Wednesday, July 9, 2014

Wait, isn't it genes that make us vote the way we do?

A story in the Tuesday NYTimes reports that our votes for president in the US is influenced by our year of birth.  Or rather, who whites vote for is influenced by when they were born, whites who don't live in the South.  The story describes a model constructed by Yair Ghitza and Andrew Gelman at Columbia and published as a working paper titled "The Great Society, Reagan's Revolution, and Generations of Presidential Voting."  The Times piece has a nice graphic showing the effect of birth year on a lifetime of voting.

Ghitza and Gelman write
The political events of a voter’s teenage and early adult years, centered around the age of 18, are enormously important in the formation of these long- term partisan preferences. The model is shown to be powerful, explaining a substantial amount of the macro-level voting trends of the last half century, especially for white voters and non-Southern whites in particular.
Their model is a way to explain the non-monotonic pattern of voting by age.  That is, the proportion of Republican votes doesn't increase steadily as people age, for example, and young people do not overwhelmingly vote Democratic, as some analysts would suggest.  Instead, the graphs of votes by age jump around, as below, and Ghitza and Gelman believe their model explains why.

Republican vote by age; Ghitza and Gelman, 2014

So, while people tend to vote as their parents voted, the stronger influence according to the model is formative events when the voter is in his or her teens and early twenties.  If the country was going through good times, with a president with high approval ratings, those who were teenagers at that time will vote that president's party, on average, for the rest of their lives.  And vice versa.  So, people who were teenagers during the Reagan years tend to vote Republican, and those who were teenagers during the Clinton years tend to vote Democratic.  The authors note that the model works best for whites in general, and non-Southern whites in particular.  African Americans tend to always vote Democratic, and the data weren't good enough in early years to enable the authors to treat Hispanics separately.

So, this is sort of interesting.  Does it mean that candidates are wasting their money on political advertising, and it doesn't matter whether corporations are people or not?  Or is the non-trend flexibility enough to swing an election and hence worth all the PAC and other sketchy dealings candidates engage in?

But wait a minute.  Weren't we told not too long ago that how we vote is genetic?  Fixed before birth -- yes, we take our voting patterns from our parents, but it's hard-wired, not learned.  An associate professor of political science and microbiology here at Penn State, Peter Hatemi, along with a collaborator published a paper in Trends in Genetics in 2012 reviewing evidence for genes that influence our political leanings, and how we vote.  So who's right?  Is it our genes, or pivotal events in our adolescence?

Does science require replication?
Replication of results is one of the standard criteria for science: if you did your work right, the idea (taught to all science students) is that anyone else who wishes to should be able, at least in principle, to replicate your result.  That is routinely given as the way science automatically polices itself against bad work or fraud.

The importance of replication is currently a major issue in psychology, with heated debate about whether failures to replicate original results show that they were wrong and should be reported to the authors, and what it means for the careers of the non-replicators, and so on. But, there's less discussion about what failure to replicate might really mean.

Jason Mitchell, a social psychologist at Harvard, has written on this subject, and his paper is getting torn apart by scientists of many stripes.  In essence, his argument is that scientists should not attempt to replicate other's results because failure to do so is meaningless, largely because the replicators won't be as expert at the science as the original investigators.  It's understandable that scientists who routinely rely on replicability, at least in principle, would criticize Mitchell's point of view.

But there are some central, yet poorly recognized issues here.  And this is not to say that we agree with Mitchell's take, just that there are issues that we don't see being discussed.  First, social science is not physics, or chemistry or even biology; replication is not as straightforward when humans are the subject, not atoms that are all alike, and methodology is, well, soft.  A bottle of gozillions of oxygen molecules may represent replicable phenomena since each molecule is identical, and statistical methods were largely developed for and suitable to such truly replicable phenomena.  Indeed, much of chemistry and physics depend on this because the behavior of individual elements, like atoms or electrons, can't be observed directly.  But this is by no means so clearly the case for social or genetic or evolutionary phenomena.  We've blogged before on the imperfectability of the social sciences (e.g., here), so enough said about that.

More importantly, researchers routinely try to replicate studies in epidemiology, to identify the cause of obesity or heart disease or asthma -- and what we now know is that obesity is caused by energy imbalance or processed foods or sugar or high fructose corn syrup or the not-Mediterranean diet or gluten and much much more, including an ever-growing list of genes.  And, asthma is caused by breast feeding, bottle feeding, poverty, wealth, prenatal stress, postnatal stress, antibiotics, acetaminophen and much much more, as well as an ever-growing list of genes.  A tentative finding is followed-up (a favored grant strategy, in fact) with larger samples to try to refine the estimates of effects -- assuming the effects are still there!

In fact, genetic and epidemiological studies of complex diseases often fail to replicate earlier studies, which is why we so often see science news stories that, shockingly, overturn accepted beliefs.  But it's usually not clear why a new study doesn't find the same as an earlier one, and sometimes can't be clear. It's too easy to say that things weren't measured as accurately etc. in the previous study -- or the replicating study -- but that is often not a very convincing explanation.  There are many pathways to complex diseases, and people can get to the same place in their own way.  So, two studies, though they yield different results, might well be correct.  But how would we know?  Replicate again?  Two out of three?  Toss a coin?  Take a vote of 'experts'?

A sample involving complex living organisms -- mice, humans, ferrets -- is never possible to completely replicate. Because we're the result of evolution, which is about diversity, we are simply never the same.  Even the same investigators, using the same equipment run by the same technicians in the same lab may fail to replicate their own earlier results if they are analyzing samples from different study populations.  Or even a second sample of the same people at a later time, or different people from the same population. People aren't atoms.  If an effect is small, its detectability or net effect may depend on the presence and frequency of other causal factors in a given sample.  Even a positive replication may be for differing reasons (more than one cause for a similar outcome, a different mix in different samples).  So failure to replicate need not mean a suspected cause isn't a cause after all -- it may reflect a poor understanding of evolution and variation.  We do not have an adequate body of theory for understanding the specifics of given cases, even in genetics and certainly in social sciences.

P.S. Our vote on the voting studies
For what it's worth, we wouldn't bet on either of the explanations for how we vote, replicated or not.  The pivotal-events hypothesis is as unlikely to actually explain voting as is the political-genetics hypothesis, largely because both are attempting to reduce a complex, population-level trait to simple, single explanations.  And then apply them to individuals -- there's a name for this; it's called the ecological fallacy.  As with complex diseases, there surely isn't a one-to-one relationship (Ghitza and Gelman no doubt would agree, given that they don't find the same results for African Americans, and the effect is stronger outside the Southern US); there are many paths to how people vote, and how people vote can change over time.

And despite what behavioral geneticists claim, we don't know nearly enough about genetics to be able to attribute a culturally laden trait like voting preference to genes.  Political preference can cluster in families and appear genetic for a variety of reasons, and we know that voting patterns are transitory.

So, pardon us if we don't take seriously either model of what makes us vote the way we do.  Maybe we need to fix up social science, and maybe some people who do 'genetics' ought to have a better understanding of genetics, before we start making assertions about cause, by confusing correlation with causation, and temporary patterns as if they were inherent.

Monday, August 19, 2013

Genes 'for' XLID -- the replication/falsifiability fallacy

How do we know when a particular gene variant actually causes a trait?  
A recent paper in the American Journal of Human Genetics (XLID-Causing Mutations and Associated Genes Challenged in Light of Data From Large-Scale Human Exome Sequencing, Piton et al.) raises some important questions about identifying genes for variants associated with X-linked intellectual disabilities (XLID).  But in doing so, the paper raises questions about identifying causal variants in general.

The incidence of intellectual disability (ID) is 1-2% in children.  Patterns of occurrence have long suggested X chromosome-linked causation, and numerous X-linked variants have been found to be associated with ID.  FMR1, associated with fragile X syndrome, was one of the first found, but the number has grown since then, "from only 11 in 1992 to 43 in 2002 and over 100 genes now". That is, 100 genes on the X chromosome thought to be associated with intellectual disability.

Some XLID is syndromic, and some not -- identifying causal gene variants is easier for syndromic forms because unrelated individuals with some or all of a specific phenotype can be more easily matched, and the assumption made that it's the same mutation that causes multiple symptoms.  But, as Piton et al. point out, variation in the phenotype, with milder forms of a syndrome, or incomplete penetrance, not to mention that there can be multiple genetic pathways to a given trait, can mask -- or create -- similarities, thus making it harder to match unrelated probands (people affected with the trait you're interested in explaining).

Gene: Wikimedia Commons
Function is hard to demonstrate
But, let's say a potentially causal mutation has been identified in one or more people or families.  Now it has to be validated, and its function deciphered.  Mark Wanner at Jackson Labs recently interviewed leading human geneticist Aravinda Chakravarti on determining gene function. Finding a gene is the easy part, Aravinda said.
But while finding and localizing genes isn’t a big challenge any more, it is still very difficult to figure out what they actually do and how their malfunctions lead to disease. Clearly, one aim is to understand their normal function and another to figure out how changes in that normal genetic program leads to disease. For chronic diseases of humankind—cancer, heart disease, neurological diseases, anything—we still are largely ignorant of how genetic abnormalities lead to disease. That is still mostly a black box. That’s the part that many labs, including my own, are focusing on more closely and looking to solve. Unfortunately, this may require a disease-by-disease solution.
When the variant has been found in only one or a handful of individuals, this makes it even more difficult.  And, as Chakravarti said,
...typically a single gene has many alternative functional forms with many different functions. Understanding these functions across development and aging remains challenging since the universe of possibilities is so large. Moreover, few genes function by themselves and many functions are only evident by one gene interacting with another . . . figuring out this aspect is still in its infancy since the universe of these possibilities is even larger. One has then also to consider that the gene may have different functions in different cell types and tissues. That's why it’s hard.
Piton et al. note that even if a functional study shows that a candidate variant has an effect at the protein or cellular level, this doesn't necessarily demonstrate that this effect is responsible for the trait. This is further complicated by the observation that people may carry a supposed causal gene variant but not have the trait.  Does that mean the variant is not causal after all? 

Given the difficulties in validating a gene variant's involvement in a trait, Piton et al. wanted to assess the likelihood that the genes currently reported to be associated with XLID in fact are.  They used the National Heart, Lung and Blood Institute's database of 6,500 sequenced exomes, collected for genetic studies of traits unrelated to ID, as a set of controls in which to search for the 104 variants currently thought to be causal. 

Most of the validating is done statistically, though this can be ambiguous.  E.g., when a variant is present in a single affected individual, this can mean either that it's a false positive or that it's a rare causal variant, but it's impossible to determine which with a single observation.  It can also be impossible to determine which with multiple observations, because some people can have the trait and not be affected, or multiple affected related individuals might have the variant simply because they have shared ancestry.

If a variant is also found in the NHLBI database of individuals of whom perhaps 50 males might have XLID, based on population prevalence coupled with the terms of inclusion in the database (being able to sign informed consent forms, e.g.), it becomes statistically more likely that it's a false positive, though there are numerous reasons why it might not be. 

So, even addressing the question head on, aware of the issues, in many cases it's still hard if not impossible to definitively determine function.  Piton et al. found 22 of the many mutations identified in the 104 genes associated with XLID in the NHLBI control group.  But, if the variants are truly causal, they might have expected to find at most 1 or two given the expected prevalence of XLID in the sample.  So, there were more than expected, and all but two were detected in males and females, which means they were indeed unlikely to be causal, given that XLID is overwhelmingly seen in males, as are most X-linked traits.

They further determined that 10 of the genes previously considered to be associated with XLID are most likely not, and another 15 are doubtful. But their results are only suggestive.  They ranked other genes on the list from 'highly questionable' to 'likely' to 'needs verification', based on what is known about the gene, the variants and traits.

The more we know about genetics, the more it seems we don't know.  As with many other traits, candidate genes and variants thought to be associated with XLID keep getting added to the list of possible causal genes, but they are rarely validated.  Why are they considered causal?  Most often for statistical reasons, but also including the sometimes deceptive fact that if a variant is found in a gene already on the list of genes associated with a trait, it's considered to be a very likely candidate.

Cystic fibrosis is another example -- thousands of variants in the CFTR gene have been found in patients with CF -- again as with many traits, most of the rare variants are assumed to be causal, not demonstrated, based on knowledge that other variants in the gene are causal.  Similarly, Piton et al. have determined that some variants in genes known to be associated with XLID are apparently innocuous.  To their surprise, they also found a mutation in unaffected males that had previously been demonstrated to affect the protein encoded by the gene and therefore was assumed to be associated with XLID.

Another cautionary tale appeared in Cell not long ago.  A paper called "Exome Sequencing of Ion Channel Genes Reveals Complex Profiles Confounding Personal Risk Assessment in Epilepsy", Klassen et al., from Jeffrey Noebels' lab, reported the results of sequencing the exomes of 237 ion channel genes in people with and without sporadic idiopathic epilepsy.  They found that "rare missense variation in known Mendelian disease genes is prevalent in both groups at similar complexity, revealing that even deleterious ion channel mutations confer uncertain risk to an individual depending on the other variants with which they are combined." That is, the same ion channel variants assumed to cause epilepsy are found in people without the trait. 

But causation is genomic!
This all raises an important issue that is often overlooked.  Perhaps the important issue, as the hunt continues for genes for disease.  Causation is genomic, and background and environment matter.  Marginal gene effects really may often depend more on the sampled backgrounds, therefore, than on the candidate gene itself. 

This means that if we assume genes are causal, but that they don't always act that way, neither 'positive' nor 'negative' findings can undermine that theory.  It allows us to keep our genomic causation axiom, and impedes conceptual progress.  The underlying reason, consistent with the above sentiments, is that we do not have a theory for how genes affect traits.  We would say that the nature of life is an evolutionary historical process that proliferates ad hoc diversity, with no prior constraint on which or how many or how genes produce which trait.  In this sense, we can say that genomics is not like chemistry, in which every carbon atom is identical.

This is also why genomic risk rather than single-gene risk may be better, and why we've argued that single-site GWASsy results are suspect, but also even if one uses background, the backgrounds are so complex and varied they really don't help much except in relatively unusual circumstances.

Wednesday, May 30, 2012

Magical science: now you see it, now you don't. Part II: How real is 'risk'?

Why is it that after countless studies, we don't know whether to believe the latest hot-off-the-press pronouncement of risk factors, genetic or environmental, for disease, or of assertions about the fitness history of a given genotype?  Or in social and behavioral science....almost anything!  Why are scientific studies, if they really are science, so often not replicated when the core tenet of science is that causes determine outcomes?  Why should we have to have so many studies of the same thing, even decade after decade?  Why do we still fund more studies of the same thing?  Is there ever a time when we say Enough!?

That time hasn't come yet, and partly that's because professors have to have new studies to keep our grants and our jobs, and we do what we know how to do.  But there are deeper reasons, without obvious answers, and they're important to you if you care about what science is, or what it should be--or what you should be paying for.

Last Thursday, we discussed some aspects of the problem when a set of causes that we suspect work only by affecting the probability of an outcome we're interested in.  The cause may truly be deterministic, but we just don't understand it well enough, so must view its effect in probability terms.  That means we have to study a sample, of repeated individuals exposed to the risk factor we're interested in, in the same way you have to flip a coin many times to see if it's really fair--if its probability of coming up Heads is really 50%.  You can't just look at the coin or flip it once.

Nowadays, reports are often of meta-analysis, in which, because it is believed that no single study is definitive (i.e., reliable), we pool them and analyze the lot, that is, the net result of many studies, to achieve adequate sample sizes to see what risk really is associated with the risk factor. It should be a warning in itself that the samples of many studies (funded because they claimed and reviewers expected them to be adequate to the task) are now viewed as hopelessly inadequate.  Maybe it's a warning that the supposed causes are weak to begin with--too weak for this kind of approach to be very meaningful?

Why, among countless examples, after having done many studies don't we know if HDL cholesterol does or doesn't protect from heart disease, or antioxidants from cancer, or coffee is a risk factor, or obesity is, or how to teach language or math, or avoid misbehavior of students, or whether criminality is genetic (or is a 'disease'), and so on--so many countless examples from the daily news, and you are paying for this, study after study without conclusive results, every day!

There are several reasons.  These are serious issues, worthy of the attention of anyone who actually cares about understanding truth and the world we live in, and its evolution.  The results are important to our society as well as to our basic understanding of the world.

So, then, why are so many results not replicable?
Here are at least some reasons to consider:
1.  If no one study is trustworthy, why on earth would pooling them be?
2.  We are not defining the trait of interest accurately
3.  We are always changing the definition of the trait or how we determine its presence or absence
4.  We are not measuring the trait accurately
5.  We have not identified the relevant causal risk factors
6.  We have not measured the relevant risk factors accurately
7.  The definition of the risk factors is changing or vague
8.  The individual studies are each accurate, and our understanding of risk is in error
9.  Some of the studies being pooled are inaccurate
10.  The first study or two that indicated risk were biased (see our post on replication), and should be removed from meta-analysis....and if that were done the supposed risk factor would have little or no risk.
11.  The risk factor's effects depend on its context:  it is not a risk all by itself
12.  The risk factor just doesn't have an inherent causal effect:  our model or ideas are simply wrong
13.  The context is always changing, so the idea of a stable risk is simply wrong
14.  We have not really collected samples that are adequate for assessing risk (they may not be representative of the population at-risk)
15.  We have not collected large enough samples to see the risk through the fog of measurement error and multiple contributing factors
16. Our statistical models of probability and sampling are not adequate or are inappropriate for the task at hand (usually, the models are far too simplified, so that at best they can be expected only to generate an approximate assessment of things)
17.  Our statistical criteria ('significance level') are subjective but we are trying to understand an objective world
18.  Some causes that are really operating are beyond what we know or are able to measure or observe (e.g., past natural selection events)
19. Negative results are rarely published, and so meta-analyses cannot include them, so a true measure of risk is unattainable
20. The outcome has numerous possible causes; each study picks up a unique, real one (familial genetic diseases, say), but it won't be replicable in another population (or family) with a different cause that is just as real
21. Population-based studies can never in fact be replicated because you can never study the same population--same people, same age, same environmental exposures--at the same time, again
22. The effect of risk factors can be so small--but real--that it is swamped by confounding, unmeasured variables. 
Overall, when this is the situation, the risk factor is simply not a major one!
This situation--and our list is surely not exhaustive--is typical and pervasive in observational rather than experimental science.  (In the same kinds of problems, lists just as long exist to explain why some areas even of experimental science don't do much better!)

A recent Times commentary and post of ours discussed these issues.  The commentary says that we need to make social science more like experimental physical science with better replications and study designs and the like.  But that may be wrong advice.  It may simply lead us down an endless, expensive path that simply fails to recognize the problem.  Social sciences already consider themselves to be real science.  And presenting peer-reviewed work that way, they've got their fingers as deeply entrenched into the funding pot as, say genetics does.

Whether coffee is a risk factor for disease, or certain behaviors or diseases are genetically determined, or why some trait has evolved in our ancestry...these are all legitimate questions whose non-answers show that there may be something deeply wrong without current methods and ideas about science.  We regularly comment on the problem.  But there seems to be no real sense that there's an issue being recognized, in opposition to the forces that pressure scientists to continue business as usual---which means that we continue to do more and more and more-expensive studies of the same things.

One highly defensible solution would be to cut support for such non-productive science until people figure out a better way to view the world, and/or that we require scientists to be accountable for their results.  No more, "I write the significance section of my grants with my fingers crossed behind my back" because I know that I'm not telling the truth (and the reviewers, who do the same themselves, know that you are doing that).

As it is, resources go to more and more and more studies of the same that yield basically little, students flock to large university departments that teach them how to do it, too, journals and funders make their careers reporting their research results, and policy makers follow the advice.  Every day on almost any topic you will see in the news "studies show that....."

This is no secret: we all know the areas in which the advice goes little if anywhere.  But politically, we haven't got the nerve to make such cuts and in a sense we would be lost if we had nobody assessing these issues.  What to do is not an easy call, even if there were the societal will to act.