We recently had the honor of contributing a chapter to the book
“Implementing Reproducible Research“
edited by Victoria Stodden, Friedrich Leisch, and Roger D. Peng.
The book features contributions under three major themes of
- “Practices and Guidelines” and
Certainly three areas we are passionate about at Kitware!
summarizes our experience working at Kitware and with collaborators on the practical aspects of performing reproducible research on a daily basis while developing open source frameworks for scientific research, such as ParaView, VTK, ITK, CMake and MIDAS.
In particular we addressed the following topics:
- Open Data
- Open Source
- Open Access
- Open Standards
- Open Science Platform
The book is already available from Amazon, and chapters can be viewed online, and downloaded.
We hope that this book will be a valuable contribution to current and future scientists who want to practice the true scientific method, which is rooted in the principle of reproducibility verification.
“Nullius in Verba”
“Take Nobody’s word for it”
To be counted as true scientific work, the reports of experiments ought to be replicated first by independent groups. Such a task requires that the independent researchers have access to all the elements defining the experiment; today these elements must include software, data and documentation. Therefore, Open Science is a necessity to enable the practice of Reproducible Research.
There are a host of subjects covered in other book chapters, authored by strongly committed advocates and practitioners of reproducible research and open science.
We enthusiastically invite you to check them out.
2 comments to Reproducible Research Book – Practicing Open Science
Hi Luis etc.
Great book chapter.
I attended ISBI recently as a participant in one of the Grand Challenges. I talked to a lot of PHd candidates and it seemed there is still tremendous pressure on them to produce fractional dB SNR improvements on a few standard test images… I remember those days, 16 hour days in the computer lab trying to get fractional improvements on Lena.
ISBI has a practical track…
“Particular topics of interest for this track include: Recent practical developments/advances of optical, MRI, or other imaging instrumentation, imaging protocols, or biochemical tools as well as recent biological/clinical findings obtained primarily via the use of existing imaging algorithms or software tools.”
but it seems to be a second class citizen…
“Only 1-page abstract submissions will be considered and accepted submissions will be presented as posters at the meeting, but will be neither published in the conference proceedings nor in IEEE Xplore.”
As one of the sponsors maybe Kitware could bring up the possibility of having the poster abstracts published. It would be nice to see the “Findings obtained primarily via the use of existing imaging algorithms” still published.
Thanks for pointing this out.
It is unfortunately true that most conferences, and almost of journals disregard the practical reality of scientific research, which is that 90% of the work is software development. Those who do not know this, are probably too far removed from the practical work of scientific research, that they can pretend to ignore this reality.
As you mentioned, they treat software (and data) as second class citizens. I speculate that this is one more of the deleterious effects of how the “Publish-or-Perish” corruption has destroyed the essential ethics in scientific research.
On the bright side, confronted with the poor record or reproducibility,
several funding agencies are turning their attention to the basic contributions of (real) data sharers, and (open) software development, and starting to recognize how essential they are for actual scientific work.
NIH is revising reproducibility rules:
and NSF is exploring the mechanisms required to make software and data citable elements, so that they can contribute to the professional record of researcher, at a similar level of the traditional papers:
To make this work, we must apply pressure in multiple directions.
1) As authors, we must build the culture of “showing data and software” as a cultural norm. As to make hiding data and hiding software, an ethically unacceptable practice.
2) As reviewers, we must request access to data and software. Otherwise, the reviewer is a head figure that simply glance over papers and write unfounded opinions of no scientific value.
3) As readers, we must request access to data and software. Because the paper is never sufficient by itself to make the case for a result.
4) As taxpayers, we must request government agencies to ensure that the public gets its money’s worth, by enforcing that publicly funded research is required to make public any acquired data and develop software.
A new generation of true publishers is emerging. These “New Publishers” are restoring the essential meaning of the word “Publish” which is “To Make Public”.
A true “Publication” is then, by definition, one where all the elements of the work are made public.
Along those lines, true “Scientific Publishers”, are those who make “Reproducible Work” public.
Anything else, is in my opinion, incomplete work, and false advertisement, and shall not be confused with scientific work.