N=1 drug trials: yet another form of legerdemain?

The April 30 issue of Nature has a strange commentary ("Personalized medicine: time for one-person trials," by Nicholas Schork) arguing for a new approach to clinical trials, in which individuals are the focus of entire studies, the idea being that personalized medicine is going to be based on what works for individuals rather than on what works for the average person.  This, we would argue, shows the mental tangles and gyrations being undertaken to salvage something that is, for appropriate reasons, falling far short of expectation, and threatening big business as usual.  The author is a properly highly regarded statistical geneticist, and the underlying points are clearly made.

A major issue is that the statistical evidence shows that many important and costly drugs are now known to be effective in only a small fraction of those patients who take them.  That is shown in this figure from Schork's commentary.  For each of 10 important drugs, the blue icons are persons with positive results, the red icons are the relative number of people who do not respond successfully to the drug.

Schork calls this 'imprecision medicine', and asks how we might improve our precision.  The argument is that large-scale sampling is too vague or generic to provide focused results.  So he advocates samples of size N=1!  This seems rather weird, since you can hardly find associations that are interpretable from a single observation; did a drug actually work, or would the person's health have improved despite the drug, e.g.? But the idea is at least somewhat more sensible: it is to measure every possible little thing on one's chosen guinea pig and observe the outcome of treatment.

"N-of-1" sounds great and, like Big Data, is sure to be exploited by countless investigators to glamorize their research, make their grant applications sound deeply insightful and innovative, and draw attention to their profound scientific insights.  There are profound issues here, even if it's too much yet another PR-spinning way to promote one's research.  As Schork points out, major epidemiological research, like drug trials, uses huge samples with only very incomplete data on each subject.  His plea is for far more individually intense measurements on the subjects.  This will lead to more data on those who did or didn't respond.  But wait.....what does it mean to say 'those'?

In fact, it means that we have to pool these sorts of data to get what will amount to population samples.  Schork writes that "if done properly, claims about a person's response to an intervention could be just as well supported by a statistical analysis" as standard population-based studies. However, it boils down to replication-based methods in the end, and that means basically standard statistical assumptions.  You can check the cited reference yourself if you don't agree with our assessment.

That is, even while advocating N-of-1 approaches, the conclusion is that patterns will arise when a collection of such person-trials are looked at jointly.  In a sense, this really boils down to collecting more intense information on individuals rather than just collecting rather generic aggregates. It makes sense in that way, but it really does not get around the problem of population sampling and the statistical gerrymandering typically needed to find signals that are strong or reliable enough to be important and generalizable.

While better and more focused data may be an entirely laudable goal, if quality control and so on can in some way be ensured, but beyond this, N-of-1 seems more like a shell game or an illusion in important ways.  It's a sloganized way to get around the real truth, of causal complexity, that the scientific community (including us, of course) simply have not found adequate ways of understanding--or, if we have, then we've been dishonorably ignoring what we know in making false promises to the public who support our work and who seem to believe what scientists say.

It's a nice idea, or perhaps one should say  'nice try'?  But it really strikes one as more wishful than novel thinking, ways to keep on motoring along with the same sorts of approaches to look for associations without good theoretical or prior functional knowledge.  And, of course, it's another way to get in on the million genome, 'precision medicine'© gravy train. It's a different sort of plea for the usual view that intensified reductionism, enumeration of every scrap of data one can find, will lead to an emerging truth.  Sometimes, for sure, but how often is that likely?

We often don't have such knowledge, but whether there is or isn't a conceptually better way, rather than a kind of 'trick' to work around the problem, is the relevant question.  There will always be successes, both lucky and because of appropriately focused data.  The plea for more detailed knowledge, and treatment adjustments, for individual patients goes back to Hippocrates and should not be promoted as a new idea.  Medicine is still largely an art and still involves intuition (ask any thoughtful physician if you doubt that).

However, retrospective claims usually stress the successes, even if they are one-off rather than general, at the neglect of the lack of overall effectiveness of the approach--as an excuse to avoid facing fully up to the problem of causal complexity.  What we need is not more slogans, but better ideas, questions, more realistic expectations, or really new thinking.  The best way of generating the latter is to stop kidding ourselves by encouraging investigators, especially young investigators, to dive into the very crowded reductionist pool.

Personalized medicine and Fragile X: complexity, disease and economics

Fragile X is a genetic disorder that typically causes distinct facial features as well as moderate to severe mental impairment and social withdrawal, and as such it is considered to be the most common single-gene cause of autism.  The disorder was determined some time ago to be  "caused by the unstable expansion of a CGG repeat in the FMR1 gene and abnormal methylation, which results in suppression of FMR1 transcription and decreased protein levels in the brain" (Lutz et al., 1998, cited in OMIM).  That this repeat expansion is causal, for whatever reason, it well established by a strong association between the genetic variant and the trait. As with nearly all genetic disorders, even when 'the' cause has been identified as a variant or variants in a single gene, the manifestation of this disorder is a spectrum of severity.

There is currently no drug treatment for the "core symptoms" of fragile X, though several drugs that treat various behavioral symptoms are available.  A recent story in the New York Times describes the experiences of several families with a drug that was developed to reduce social withdrawal in children with fragile X or autism, manufactured by Seaside Therapeutics, but was still undergoing clinical trial.  Why autism should have any physiological connection with fragile X is not known, but there is an overlap of symptoms.  The drug, arbaclofen, or STX209, affects neurotransmitter signaling and hence, brain function and its behavioral implications.

A Phase 2 clinical trial was carried out over three years ago, and the results announced in 2011.  Seaside said

...clinically meaningful improvements on global and specific neurobehavioral outcomes were observed in the general study population. The improvements were statistically significant in pediatric patients with more severe impairments in sociability -- a core symptom of fragile X syndrome.

Following the trial, Seaside Therapeutics continued to supply the drug to some children for whom the drug was considered to be beneficial.  According to the NYT, the drug made an enormous difference in the lives of a number of children.  Some were able to speak for the first time, others halted aggressive behaviors, and so forth.  This seemed at least statistically, as well as subjectively, to be a cause-effect relationship.  There were parents, of children with autism as well as fragile X, who considered arbaclofen a miracle drug.

In 2012 Seaside initiated a Phase 3 trial, to further test the drug.  In announcing this trial, the president of Seaside said,

“In our Phase 2 study, we were very excited to observe clinically meaningful improvements in social impairment in patients receiving STX209—marking the first time a drug candidate has positively impacted a core symptom of fragile X syndrome. The Phase 3 study is the most comprehensive study ever undertaken in patients with fragile X syndrome and represents the first time that a drug candidate will be evaluated for a core symptom of fragile X syndrome as the primary endpoint.”

But, production is now being halted because the drug has not proven to be a successful enough treatment for social withdrawal in children with autism in the Phase 3 study.  This seems to be a bureaucratic and financial issue rather than a scientific one.  According to a Seaside vice president, quoted on the "Age of Autism" website,

The FDA requires companies to pick one, and only one, assessment as the “primary endpoint” of the study. In their eyes, the result on that one pre-selected endpoint makes or breaks the study. In our recent autism study, STX209 did not show an advantage over placebo (see above) on the primary endpoint of social withdrawal, so the FDA and some news reporters regard it as a negative study. In fact, STX209 did show advantages over placebo on a number of other assessments. Some of these secondary endpoints are just as meaningful as the social withdrawal assessment, or even more so, but in the FDA’s eyes, they don’t make the study a positive study, because they were not pre-selected as the “primary endpoint.”

The pharmaceutical giant, Roche, had been backing development of the drug, but they've recently pulled their support as they cut back on research and development in general, and it seems that Seaside cannot afford to continue manufacturing the drug alone.

Seaside released this statement:

"The study termination is due to resource limitations at seaside, and is not due to any safety issue associated with STX209. We know that this termination will be disruptive and disappointing for many. We are planning to complete phase 3 and the results of this study should be available late summer. If the results are positive, seaside will discuss with FDA the required next steps for approval of STX209. 

"This means the drug could potentially become available in the future but there is still nothing guaranteed."

Parents who saw tremendous improvement in their children on this medication are devastated.

Well, we started this post after reading the piece in the New York Times, thinking that we knew where we were going with it -- like most other genetic disorders, fragile X is a complex disorder, and only a subset of children in the studies would respond favorably to the drug because of the heterogeneity of the phenotype and variation in genetic background, and so of course it's understandable that it's difficult to ascertain the drug's efficacy.  By implication, genetic background determines whether a child responds poorly or well to STX209, and which children respond well and which respond poorly hasn't been determined.

While that may well be true, that's not the real story here.  Instead it's economics.  And that's heartbreaking.  So here is a case when 'personalized genomic medicine' might be a correct way to identify cause or predict effect, and here, where the association actually works (regardless of why), we have other issues interfering with effective care.  Life is complex, but our culture often advances things that don't do good, and impedes ones that actually do.

Crowdsourcing: large-scale, peoples' epidemiology.

Crowdsourcing is the outsourcing of a task previously performed by a small group or organization, to a much larger community.  As if you didn't already know.  People all over the UK, for example, have been painting garden snails with nail polish, swapping them with snails from their neighbors, and figuring out if the little guys know how to get back home.  (Yep, they do, it turns out, within 100 meters -- we know this because thousands of people have sent their data to the researchers for analysis.)

Of course, the Audubon Society in the US has been enlisting birders in the US to help in their annual Christmas bird count for decades, so the idea isn't new, it's just a lot easier to contact masses of people, and collate their data with computers.

And SETI, the search for extra-terrestial intelligence, and other astronomical things, has used many many people's home computers.  Crowdsourcing is cheaper than outsourcing, and involves the public in general rather than relying on experts who may or may not have blind-spots, vested interests, or rigid institutionalized structures.

So, we wonder, could crowdsourcing replace some clinical trials?

What if you want to know, say, the best possible treatment for acne?  You've taken your dermatologist's advice for years, religiously using that retin A, or taking those antibiotics, but maybe you're worried about contributing to antibiotic resistance problems, or you're just not happy with the results.

So you log onto the web, poke around a bit and eventually come to rest at a site called Acne.org.  Where you find some guy's homegrown regimen for treating acne, and you're tempted to dismiss it as yet another get-rich-quick-on-the-internet scheme, but hold on.  It turns out that this guy has spent years experimenting with and adjusting his use of both prescription and over-the-counter remedies, and based on his own experience, has found one that works.

And it works not just for him but for tens of thousands of other people as well, which we know from the message boards hosted by his website and from the many compelling before and after pictures posted in the gallery. 

This guy may be onto something. 

And in no small part because of the informal testing and testimonials of thousands of others who took the time to contribute their experience with both this guy's own regimen and other treatments they've tried as well.  If you want to solve your acne problem, you could do worse than start here. 

But of course acne isn't the only affliction for which you'll find this kind of coming together of people and information on the web.  There's a website -- at least one website -- for almost any disease or disorder, real or imagined, that you can think up, collecting stories, dispensing advice, selling products, and much more.  Granted, much of it will be the modern-day equivalent of snake-oil, but surely there's at least some gold in them thar hills.

But, how to separate the real from the fool's gold?  After all, what professional scientists are supposed to bring to the research table is training in formal study design and statistical analysis, and if that's missing, what's left? 

Indeed, the most obvious problem with many of these sites is that the contributors who write in tend to be sicker, more unhappy, non-responders, etc. than the general population, or even the population of people with that disorder. People who recovered just fine don't write in.  They've moved on.  Crackpots can participate and may be hard to identify.  People, including companies, could pose as jus' folks (this happens, for example, in Amazon's and various other companies' customer reviews). 

That is, there's rampant selection bias going on, so it is in fact often quite difficult or impossible to learn much that makes sense about cause, effect, or effectiveness of therapeutic regimens.  These kinds of sites would in no sense be replacements for traditional clinical trials.

So, there are serious constraints on the instances when this could work, but we can imagine that, if a disease or disorder is one that can be easily defined, possible treatments are readily available, and effectiveness can be easily documented, this kind of crowdsourcing approach to data collection could be brilliant.  There's no limit to the number of people who can contribute, the experiments are free to conduct and free to access, and we can all make up our own minds.  What's not to like?

In fact, established academia has an area of work called SEDA, for structured exploratory data analysis, which bears some resemblance to public science. The idea is to take less-than-ideal data or crude methods, to see whether there is evidence of an interesting signal.  If there is, then formal studies would be called for.  Or, SEDA can suggest causal hypotheses one might not have thought of.

Given the cost and also great size limitations of Establishment science, and the fact that vested interests of all sorts can have their fingers in the pie, crowdsourcing could become a very important, very inexpensive, very effective way to understand complex causal problems----and then to exert political pressure to respond more quickly than stodgy rules normally allow, to results.

Meanwhile, if you know someone with a stubborn case of acne, send 'em Acne.org's way.

Clinical trials on trial

So, how do pharmaceutical companies figure out whether a new drug is effective or not?  Or whether it saves lives?  The Sunday New York Times has a front page story about a drug trial of a new treatment for melanoma (skin cancer) and answers these questions, while also asking about the morality of drug testing as it's currently carried out.

Generally, drugs are tested in double-blind case-control studies.  Some subjects receive the treatment being tested and others, the controls, are given either the existing therapy or a placebo, and in a double-bline study neither the subjects nor the physicians know which group a patient is enrolled in.

Two young cousins had the extreme misfortune of being diagnosed with melanoma at about the same time.  Both were enrolled in a drug trial, but one was given the new therapy and the other the comparison state-of-the-art therapy.  The question being asked is apparently not whether the new drug 'cures' melanoma, but whether it lengthens the lives of those who are treated with it.  It was already known that the new drug improves quality of life, but the apparently the tumors of those treated with this drug in an early study returned.

As it turned out, the tumors of the cousin on the new drug shrank immediately and dramatically, while  those of the cousin in the control group did not.  In fact, he has now died from his cancer, while the first cousin has survived past the average longevity for people diagnosed with the advanced melanoma with which both he and his cousin were diagnosed, that is, 8 months.  He's now been alive for 9 months.

One man was 24 and the other 22 when diagnosed, one had children and the other, the one in the control group, did not.  When it was clear that the cousin on the new therapy was in much better shape than the one in the control group, his mother begged the researchers to switch him from the old therapy to the new.  They refused, saying that would make their test results uninterpretable.  Indeed, subjects in their control group need to die at a higher rate than those given the new therapy if the new treatment is to be deemed effective.

Two things are troubling about this story, if it's an accurate portrayal of this drug trial.  One, it seems that it was already known that the new drug for melanoma produces dramatically better results than any previous therapy -- at least in the short term.

The standard chemotherapy used in melanoma, dacarbazine, slowed tumor growth in 15 percent of patients for an average of two months. By contrast, PLX4032 had halted tumor growth in 81 percent of patients for an average of eight.

That is, quality of life is greatly improved for those whose tumors are genetically disposed to respond (apparently 80% of people with melanoma) on this drug.  The thing that isn't yet known is whether the new drug allows patients to live longer.

Even so, it's ethically troubling to many, including many oncologists, that answering this question requires allowing patients -- the control group -- to suffer unnecessarily.  Should longevity remain the only outcome of interest, when it's clear that a new medication can reduce suffering -- even if it doesn't improve longevity, which it may?  We think not, and there is plenty of precedent for halting clinical trials when it's clear that the drug being tested is either significantly more harmful or more beneficial than the state-of-the-art treatment.  If the NYTimes story is accurately portraying this trial, it should be halted now.

Second and more troubling, according to the story it's already known that patients diagnosed at the stage at which these two young men were diagnosed survive, on average, for eight more months.  Why isn't that a good enough comparison statistic for this new drug?  Why subject more people to the pain and suffering the state-of-the-art therapy entails in order to confirm this statistic?

The stainless steel answer is that without a 'gold standard' double-blind clinical trial, one cannot accurately assess if, or to what extent, a new therapy improves on the existing approach.  This is a gold standard that often falls short for various reasons, and one of the general standards, we thought, was that as soon as you have good reason to think one approach (new drug) or the other (current treatment)  is clearly better, you stop the trial and put everyone on the better treatment.  Why this is not routine, or applied in this case isn't clear (to us).  Granted, the results may be premature, and as the story itself reports,

One of the melanoma field’s senior clinicians, Dr. Chapman had lived through trial after trial of drugs that failed to live up to early promise. Almost every oncologist knew, too, of a case nearly 20 years earlier when bone marrow transplants appeared so effective thatbreast cancer patients demanded their immediate approval, only to learn through a controlled trial that the transplants were less effective than chemotherapy and in some cases caused death.

So perhaps there is reason to be cautious.  But if 'all' the patients get out of this is improved quality of the last months of their lives, isn't that something?


Of course, all the usual questions about evidence arise here, but so do societal questions.  Clinical trials are very expensive, and results often show no effect after many years and much expense.  How easily could we be convinced that some new treatment really works better, if vested interests learn that with any sort of evidence the costly clinical trial can be abandoned?  In other words, if the standard is relaxed, will it be abused?  Yet, if not, are patients being abused who suffer or die because, so to speak, some professor wants to finish and publish a trial, or some government agency wants to fill out a 'satisfactory' grade on some form?

To cause or not to cause, that is the question.  The answer isn't clear.

Squeezing the scientific lemon

James Lind, Scottish naval surgeon, in 1747 conducted the first known clinical trial, to test the curative effects of lemon juice on scurvy. Scurvy was the scourge of the navy, causing more deaths than actual combat, or so Lind was said to have said. Although the curative effects of citrus had apparently been known for centuries, Lind was the first to do a controlled test of its efficacy. As described in Wikipedia,

In his experiment he divided twelve scorbutic sailors into six groups. They all received the same diet, and in addition group one was given a quart of cider daily, group two twenty-five drops of elixir of vitriol (sulfuric acid), group three six spoonfuls of vinegar, group four half a pint of seawater, group five received two oranges and one lemon and the last group a spicy paste plus a drink of barley water. The treatment of group five stopped after six days when they ran out of fruit, but by that time one sailor was fit for duty and the other had almost recovered. Apart from that, only group one also showed some effect of its treatment.

BBC Radio's World Service program, Science in Action, has a running series called "Moments of Genius", and last week's (July 23) contributor chose Lind's invention of the clinical trial as his important moment. Raymond Tallis, philosopher and retired physician, said that not only did Lind's trial mark the beginning of the science of vitamins, but clinical trials are at the very heart of so-called "evidence-based medicine" (so-called when medical decisions are based on evidence collected with the scientific method; some advocates want doctors to use particular kinds of evidence in particular ways).

As Tallis puts it, we all are tempted to deceive ourselves and others, believing in what we think works even without any evidence, so that the advent of clinical trials was essential to modern medicine. For example, the theory of the four humors, that was formally accepted at least as far back as Galen, certainly seemed to be valid according to informal impressions of doctors -- or they talked themselves into it because it was considered to be the proper standard and/or they had no better ideas. Some patients did, or course, recover after being bled, but the evidence supporting this cure was not systematic or controlled by modern standards.

Clinical trials are controlled studies of the effects of a single treatment or other kinds of intervention. So, in the case of bleeding, for example, a group of people with colds would be bleed and the time to recovery would be compared with people with colds who weren't bled.  If they recovered significantly faster (significance defined statistically), than bleeding would be said to be effective.  If not, bleeding would be considered a non-treatment -- and insurance companies wouldn't cover it.  As Tallis said, "Humility is built into the very idea of a clinical trial," and is the opposite of the 'argument from authority' that is another -- very tempting -- way we have of convincing others of the efficacy of our ideas. For centuries, Galen was authority.

Though, he admits that we've still got the High Street chemist selling homeopathic medicines (see, for example, Holly's story of her adventure with homeopathy in an earlier post), evidence-based medicine has made a big difference.

Like Mendel and his peas, Lind was lucky in his choice of treatment. The effect he hypothesized -- recovery from scurvy -- was large and easily demonstrated, and led to a massive difference in the health of sailors who were on the oceans for months at a time. And certainly clinical trials are indeed fundamental to modern medicine, but as we've blogged about many times, when effects are small, or the outcomes are the accumulation of many small factors, or are complex and can be arrived at in numerous ways, experimental methods like clinical trials -- or, say, genomewide association studies (GWAS) -- aren't necessarily so straightforward. Yes, they are better than argument from authority, just on principle, but they don't necessarily lead to medicine based on evidence.

The problem is simple and it is the same one that applies to GWAS and even to evolutionary explanations: it is the idea that only one relevant factor -- be it a drug under test, or a gene, or a purportedly favored trait -- is varying between those given the drug (or cases, or organisms bearing a particular genetic variant thought to be favored by selection) and those not (or controls, or organisms without the genetic variant).

The number of relevant factors dictates the way that these kinds of statistical associations, between supposed cause and its effect, are found. More variables means larger studies that have to identify, and take into account, those variables. Thus, clinical trials are important, but not unambiguous. And here we don't refer to conflicts of interest or subtle biases in diagnosis, case and control choice, and so on, even though we know those only-too-human foibles can enter into such studies.

Even the idea of natural selection is the idea that exposure to a genetic variant, the logical equivalent to a test drug, is good for the individual. Thus evolution is viewed as a kind of clinical trial where those who have the beneficial 'drug' do well, and the others expire.

There are problems with clinical trials, and there have been recent scandals about cover-ups of relevant data, and so on. Journals have seemed complicit, investigators sometimes sponsored by a company that wants to market a drug, and so on.

But even if scandals are avoided, there are other issues that again have to do with the other, unmeasured but relevant variables. Genomic background of cases vs controls, and lifestyle exposures are among them. Lifestyle exposures of people tested in a study today may be relevant to a drug's effectiveness. But those exposures may change so that, in a decade or two, the same drug will have different effects. Likewise, as is well known, racial background can be highly relevant, and sometimes notoriously hard to control in a clinical trial study.

Nonetheless, limes and lemons do prevent scurvy, and Lind is the insightful person whose insight about how to gather persuasive evidence saved countless sailors, and many others.