NIH ponders on Reproducibility Rules
“Nullius in Verba”
“Take Nobody’s word for it”
After having hosted two workshops on the issues reproducibility in 2012, the National Institutes of Health (NIH) are just beginning to take active measures to restore the proper practice of scientific research.
Lawrence Tabak, the agency’s principal deputy director, stated
“There is certainly sufficient information now that the NIH
feels it’s appropriate to look at this at a central-agency level,”
Wadman reports that
This summer, Tabak and other senior NIH officials, including Story Landis, director at the neurology institute, and Harold Varmus, director at the cancer institute, are assessing input gathered from the directors of the agency’s 27 institutes and centres. They will then confer with NIH director Francis Collins, who will decide what steps to take.
This is on the context of recent findings, that Wadman captures in the following list:
- September 2011 Bayer HealthCare finds inconsistencies between in-house and published data in almost two-thirds of 67 projects.
- March 2012 Amgen publication shows that the findings from only 11% of 53 landmark papers can be replicated by company scientists.
- January 2013 Elizabeth Iorns, chief executive of Science Exchange, asks more than 22,000 study authors if they want her company to arrange verification of their experiments; nearly 2,000 say yes.
- May 2013 A survey at the MD Anderson Cancer Center finds that more than half of its respondents have tried and failed to reproduce published data.
- July 2013 Science Exchange launches a verification programme for commercially sold antibodies.
“If the premise isn’t validatable,
then we’re done;
it doesn’t matter how well you wrote the grant,”
Wadman states also that:
Agency officials are also considering a requirement
that independent labs validate the results of important
preclinical studies as a condition of receiving grant funding.
There is of course, the question of who will pay for reproducibility and how much would that be. This is what makes so significant that finally the conversation has escalated to the level of the largest US Federal funding agency.
It has been estimated that the cost of a reproducibility study will be in the range of $25,000. This is about 5% of the original cost of doing the research, and when put in context, it is a small cost to pay for ensuring that the initial social investment in scientific research truly leads to results that stand to inspection.
Scientific reproducibility is therefore emerging as a Market Service, whose customers are any of the R&D organizations that take as input the results of basic research funded by public agencies.
The pioneer in this field is Science Exchange,
a start-up that has taken a free-market approach to bring the benefits of reproducible research to those who actually care about the research being more than a flashy paper published in journals where reviewers do not make any effort for independent verification.
Other key players in the space of Reproducible Research are
- Run My Code: a site that allows researchers to replicate scientific results in order to check their validity and robustness.
- Figshare: a place where data, negative results and true verification materials are shared publicly.
- PLoS ONE: the progressive mega-journal that is require software-intensive papers to include source code under open source licenses.
- Mendeley: The platform for managing and sharing research papers.
These progressive organizations are taking upon themselves to restore the honest practice of the scientific method, that is anchored on the practice of reproducibility verification.
The Renaissance of True Science is now Unstoppable.