tag:blogger.com,1999:blog-1812431336777691886.post7063022829179881857..comments2024-02-29T03:57:00.088-05:00Comments on The Mermaid's Tale: Genomic scientists wanted: Healthy skepticism requiredAnne Buchananhttp://www.blogger.com/profile/09212151396672651221noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-1812431336777691886.post-3358985681427357742012-07-31T20:58:17.957-04:002012-07-31T20:58:17.957-04:00Thanks for your comment, Daniel. But ack, caught ...Thanks for your comment, Daniel. But ack, caught out on my flip lead-in! I agree with you, and certainly hope I didn't imply that this was what you'd said in your piece. Public data bases are indeed a plus in genomics, being open to correction and so on, but as our post said, should be approached with healthy skepticism. <br /><br />Yes, epidemiology is another field with a high error rate, much of it due to statistical noise but also a fundamental difficulty dealing with complexity. Among other things. Public databases would be a plus!Anne Buchananhttps://www.blogger.com/profile/09212151396672651221noreply@blogger.comtag:blogger.com,1999:blog-1812431336777691886.post-19367055270847341982012-07-31T20:44:40.681-04:002012-07-31T20:44:40.681-04:00Hi Anne,
Thanks for the write-up of the piece - g...Hi Anne,<br /><br />Thanks for the write-up of the piece - glad it's causing such constructive discussion!<br /><br />I have just one minor correction:<br /><br />"...but geneticists make them more often."<br /><br />I don't actually think that's true. Genomics makes for easy pickings because (1) nearly all raw data is made public in the field, and (2) there's an active community of people willing to reanalyze and critique published findings. However, I suspect the underlying rates of error are similar or substantially higher in other fields - for instance, cell biology is a boiling mess of unreplicated, small-scale, hypothesis-driven studies, many or most of which are based on experimental and statistical noise.<br /><br />Indeed, part of the reason I focused on genomics is because I'm optimistic that we can actually go some way towards fixing the problems in our field. For cell biology, on the other hand, I have less hope. :-)Danielhttps://www.blogger.com/profile/07276690118219000204noreply@blogger.comtag:blogger.com,1999:blog-1812431336777691886.post-65526981442816977412012-07-27T09:52:41.380-04:002012-07-27T09:52:41.380-04:00Anne makes another, if much deeper, point. We wan...Anne makes another, if much deeper, point. We want to use Enlightenment criteria of replicability, which is the underlying rationale for study designs and statistical analysis. Yet we have theory in both evolution and genetic mechanism that clearly shows why replicability is a problematic concept here.<br /><br />There is no real incentive to absorbing that awareness. Eventually, we'll have to come to grips with the fact that causes can be real but not statistically documentable by the usual criteria.<br /><br />This aspect of causal complexity is also a, if not, the major implication of evolution.<br /><br />Some clever or lucky person will one day provide a better way to think. Today, we know the problem but choose to ignore it, preferring to chase rainbows. A fraction of biological causation is due to strong effects--that, too, is expected from evolutionary principles and what we know of genetic mechanisms.<br /><br />So, whenever we have a substantial success, we proclaim it from the Journal-tops as if it's revolutionary, ignoring the majority of effort that isn't very successful. We do that in part for understandable venal reasons, but never ask whether the same investment of funds, talent, and effort might have had greater achievements.Ken Weisshttps://www.blogger.com/profile/02049713123559138421noreply@blogger.comtag:blogger.com,1999:blog-1812431336777691886.post-45396390666888297962012-07-27T09:33:51.366-04:002012-07-27T09:33:51.366-04:00Good question, Holly. There are or have been vario...Good question, Holly. There are or have been various databases and even journals for publishing negative results, which most journals are rarely willing to publish but which are obviously important. The paper last year in the reputable psychology journal showing that ESP is a real phenomenon is a case in point -- at least 2 studies unable to confirm those results were rejected by the same journal. <br /><br />But negative results aren't the same as replicating or confirmatory results. These are often published in 'lesser' journals. But as for a repository, I don't know of any -- which doesn't mean there aren't any. The genomics databases are open-ended and people can publish anything, including duplicate data, in those. But this doesn't address the problem of errors in the data.<br /><br />Submissions to the genetics databases are reviewed before they are published, but sequences aren't replicated then and catching sequencing errors is up to the scientist submitting the data. <br /><br />The question of replicating results in genetics or epidemiology is not always straightforward though. Study populations can never be completely replicated, so non-replication can mean the data are wrong, due to technical errors or poor study design, say, or the data might be correct but the failure to replicate because of the inclusion of different subjects. So, 'gene for' studies very commonly don't replicate previous ones. All the studies might accurately represent their study populations but the 'truth' is different in each one. So failure to replicate can be as informative as replication. Or not.Anne Buchananhttps://www.blogger.com/profile/09212151396672651221noreply@blogger.comtag:blogger.com,1999:blog-1812431336777691886.post-36165719036233324062012-07-27T09:10:19.904-04:002012-07-27T09:10:19.904-04:00NIH has tried to have _negative_ results reported ...NIH has tried to have _negative_ results reported by mandating reporting of results of all clinical trials. I heard directly not too long ago that Science (that august science magazine soon to be at a checkout counter near you) doesn't print rebuttals of its papers. They sometimes bury them in some online site. I can't remember if arsenic evolution paper was so ludicrous that they published refutations, but I think those were published in Science.<br /><br />But confirmatory results rarely would be published by themselves (too boring!), though relevant types of papers get to appear in the minor journals that libraries have to subscribe to but that nobody reads.<br /><br />There need to be stronger criteria for major findings to account for the various biases that lead to their over-interpretation, and less hyping of the results. <br /><br />The problem is that in the frenetic world today, the journals have no real incentive to be tempered, and they (and the 24/7 media) are so hungry for stories that we'll not see this being addressed.Ken Weisshttps://www.blogger.com/profile/02049713123559138421noreply@blogger.comtag:blogger.com,1999:blog-1812431336777691886.post-26494198785082530952012-07-27T08:57:47.842-04:002012-07-27T08:57:47.842-04:00Is there a good, go-to repository on-line for post...Is there a good, go-to repository on-line for posting confirming (but not new) results that people don't send to journals but that establish and reestablish fact over and over with maybe differing labs, machines, human subjects, human researchers? Is that what these genetic databases are? Or do their data have to go through journal peer-review first too?Holly Dunsworthhttps://www.blogger.com/profile/05260104967932801186noreply@blogger.com