More on the potential of crowdsourcing

Maybe the Office of Rare Diseases at the NIH could do with some crowdsourcing.  A place for patients to go when all else has failed, this office is a repository for unsolved medical mysteries.  Even with a record of more unsolved than solved cases, this clinic is a source of hope for many, because they're in the business of mining all the information they can on rare conditions.

The patients who go there are desperately hoping for answers, but the clinicians who try desperately to find them are facing a major limitation of science.  Science is about causal regularity in Nature, and does rather poorly with unique observations.  Repetition and prediction are gold-standard criteria for understanding.  Yet these cases are essentially unique!

A story on the CNN website tells of two patients with undiagnosed diseases who spent five days at the NIH, the recipients of exhaustive testing for every possible thing that could be wrong.  The problem with rare diseases, of course, is that they can be so rare that only one case has ever been seen.  Or, at least rare enough that there just isn't enough data in the medical literature, or collective experience with the disease, to facilitate a diagnosis.  

Here's where crowdsourcing could be -- and often is -- useful, as we suggested about last week.  The huge numbers of people who populate the web, and perhaps intensely when they have a medical dilemma that concerns them, provides a potentially unique way in which repetition can be ascertained.  People seem to find each other, and relate their stories.   Of course, these are informal and not rigorous by normal clinical standards, but experience shows that sense can be made of the data, at least to some useful extent.

Doctors try as best they can to pigeon-hole their patients into categories for which there are treatments.  Most doctors work with small sets of patients, or in small communities, where they simply can't be expected to see very rare traits more than once or twice, if at all, and then not at the same time.

Taking advantage of huge population samples, assembling rare information to reveal patterns.  Crowdsourcing.

Crowdsourcing: large-scale, peoples' epidemiology.

Crowdsourcing is the outsourcing of a task previously performed by a small group or organization, to a much larger community.  As if you didn't already know.  People all over the UK, for example, have been painting garden snails with nail polish, swapping them with snails from their neighbors, and figuring out if the little guys know how to get back home.  (Yep, they do, it turns out, within 100 meters -- we know this because thousands of people have sent their data to the researchers for analysis.)

Of course, the Audubon Society in the US has been enlisting birders in the US to help in their annual Christmas bird count for decades, so the idea isn't new, it's just a lot easier to contact masses of people, and collate their data with computers.

And SETI, the search for extra-terrestial intelligence, and other astronomical things, has used many many people's home computers.  Crowdsourcing is cheaper than outsourcing, and involves the public in general rather than relying on experts who may or may not have blind-spots, vested interests, or rigid institutionalized structures.

So, we wonder, could crowdsourcing replace some clinical trials?

What if you want to know, say, the best possible treatment for acne?  You've taken your dermatologist's advice for years, religiously using that retin A, or taking those antibiotics, but maybe you're worried about contributing to antibiotic resistance problems, or you're just not happy with the results.

So you log onto the web, poke around a bit and eventually come to rest at a site called Acne.org.  Where you find some guy's homegrown regimen for treating acne, and you're tempted to dismiss it as yet another get-rich-quick-on-the-internet scheme, but hold on.  It turns out that this guy has spent years experimenting with and adjusting his use of both prescription and over-the-counter remedies, and based on his own experience, has found one that works.

And it works not just for him but for tens of thousands of other people as well, which we know from the message boards hosted by his website and from the many compelling before and after pictures posted in the gallery. 

This guy may be onto something. 

And in no small part because of the informal testing and testimonials of thousands of others who took the time to contribute their experience with both this guy's own regimen and other treatments they've tried as well.  If you want to solve your acne problem, you could do worse than start here. 

But of course acne isn't the only affliction for which you'll find this kind of coming together of people and information on the web.  There's a website -- at least one website -- for almost any disease or disorder, real or imagined, that you can think up, collecting stories, dispensing advice, selling products, and much more.  Granted, much of it will be the modern-day equivalent of snake-oil, but surely there's at least some gold in them thar hills.

But, how to separate the real from the fool's gold?  After all, what professional scientists are supposed to bring to the research table is training in formal study design and statistical analysis, and if that's missing, what's left? 

Indeed, the most obvious problem with many of these sites is that the contributors who write in tend to be sicker, more unhappy, non-responders, etc. than the general population, or even the population of people with that disorder. People who recovered just fine don't write in.  They've moved on.  Crackpots can participate and may be hard to identify.  People, including companies, could pose as jus' folks (this happens, for example, in Amazon's and various other companies' customer reviews). 

That is, there's rampant selection bias going on, so it is in fact often quite difficult or impossible to learn much that makes sense about cause, effect, or effectiveness of therapeutic regimens.  These kinds of sites would in no sense be replacements for traditional clinical trials.

So, there are serious constraints on the instances when this could work, but we can imagine that, if a disease or disorder is one that can be easily defined, possible treatments are readily available, and effectiveness can be easily documented, this kind of crowdsourcing approach to data collection could be brilliant.  There's no limit to the number of people who can contribute, the experiments are free to conduct and free to access, and we can all make up our own minds.  What's not to like?

In fact, established academia has an area of work called SEDA, for structured exploratory data analysis, which bears some resemblance to public science. The idea is to take less-than-ideal data or crude methods, to see whether there is evidence of an interesting signal.  If there is, then formal studies would be called for.  Or, SEDA can suggest causal hypotheses one might not have thought of.

Given the cost and also great size limitations of Establishment science, and the fact that vested interests of all sorts can have their fingers in the pie, crowdsourcing could become a very important, very inexpensive, very effective way to understand complex causal problems----and then to exert political pressure to respond more quickly than stodgy rules normally allow, to results.

Meanwhile, if you know someone with a stubborn case of acne, send 'em Acne.org's way.