Three days ago, I asked whether irreproducible research is a problem or a crisis. I asked on Twitter, consulted Ivan Oransky of RetractionWatch, and discussed over e-mail with many.

We have a reproducibility problem. But is it a crisis? http://t.co/mqOlXnezp8 (I am genuinely curious about the answer. Please RT.)

— Lenny Teytelman (@lteytelman) April 6, 2015

The overwhelming consensus is that it is indeed an old problem rather than a new crisis. Most have heard that Bayer and Amgen could not reproduce 75-90 percent of published research. This has been covered extensively in the media (Economist, Wall Street Journal, NY Times, Reuters, Boston Globe). But Bayer and Amgen haven’t done their reproducibility studies 20 years ago to be able to say that what we have today is significantly different, and there is simply no evidence today that the situation is dramatically worse.

And if this isn’t a new crisis but an old problem, that has huge implications for how we address it. There have been some extreme reactions to the reports of the irreproducible research (let’s require independent validation of results before publishing, refund grants for retractions, or insitute random audits of labs). Let’s calm down and keep in mind that this same inefficient, messy, hard-to-reproduce research enterprise has given us extraordinary breakthroughs throughout the last century and on economic terms gives phenomenal returns on taxpayer funding.

At the same time, we shouldn’t just pat ourselves on the back and ignore the problems. Just because the return on the taxpayer investment is good, doesn’t mean we shouldn’t strive to make it better. I am against heavy new regulations and creating bureaucratic overhead at a huge cost, siphoning away money from new research into audits and validation of old results. I am against efforts that throw extra burdens on the scientist when doing the work and publishing it; the scientist is overwhelmed and stressed enough as is. But there is so much we can do if we acknowledge the problem rather than panic about the non-existent crisis.

If we acknowledge as the Bayer/Amgen reports show and as John Ionnidis argues that Most Published Research Findings Are False, perhaps the best way to deal with it is to make it easier to correct the research rather than obsess with making it perfect in the first place? Our whole company exists to make it easier for scientists to share and discover corrections and optimization of published methods on protocols.io (great post from Mike Loukides on why this is so critical for reproducibility). Here are some of the many other terrific efforts that address the reproducibility problem in biomed while making scientists’ lives easier:

– Open Peer Review -if we see the concerns of the reviewers and the responses of the authors, that makes it easier to determine what we can and cannot trust in the published work. F1000Research, BMJ, Publons

– Post-publication reviews/discussion/annotation – it is a foolish to think that peer review can predict whether or not a given work will stand the test of time. Peer review is not designed for that. But as the authors and other researchers extend on the published work and invariably find mistakes and corrections, we need to make it easy to correct! PubMed Commons, PubPeer, F1000Research, protocols.io

– Pre-print servers – one contributing factor to irreproducible research is the pressure to publish in a high impact journal and the desire to do everything possible to see your work finally in print. The pre-print servers relieve so much of this pressure by letting you instantly make your manuscript public. bioRxiv,PeerJ, F1000Research

– Discovery services – Since the nature of science is dynamic and evolving and most publications need corrections and optimizations, it’s important to automatically keep the researchers informed when these changes happen. That’s why our PubChase users instantly get notified if a paper in their library is being discussed on PubMed/PubPeer or has been retracted. Our protocols.io users are alerted right away if a protocol they use has been annotated. And services such asReadCube, Mendeley, Papers, Zotero are in a great position to implement as many of such auto-notifications as possible.

The very nature of science research, because it’s pushing the boundaries, is that much of what is published is inherently wrong. Just as software released by startups cannot be free of bugs, striving to publish research that is 100% verified and correct would undermine the progress of science. Good startups fix these bugs quickly. Let’s make it easier for the scientists to do the same.

————————

[UPDATE 4/11/15: Just came across Stephen Heard’s excellent post from a few days ago, making a similar argument.]

This article was originally published on The Spectroscope by Career Advisory Council member Lenny Teytelman. The Career Advisory Council is comprised of leaders in the biomedical science community of San Antonio who are a resource of insights and expertise for UT Health Science Center at San Antonio trainees.