Skip to content

Reproducibility definition

What is reproducibility ?

The definition may not be the same depending on the scientific area or even context. Reproducibility is an "umbrella" term covering different aspects such as reproducibility, replication and repeatability, which may co-exist in different disciplinary fields. In this document we will focus on reproducibility and briefly compare it to replicability in the context of software and computational science.

Different kinds of reproducibility

We can distinguish different kinds of reproducibility, such as:

  • Observational reproducibility This is the case, for example, with research in ethology or ecology, which focuses on the behaviour of species or the functioning of ecosystems, which are by definition unique. Observing the same ecosystem, researchers should be able to reach same conclusions.
  • Experimental reproducibility Experimental reproducibility refers to the ability to reproduce the results of a scientific experiment using the same experimental protocol, but with different data or samples.
  • Statistical reproducibility Statistical reproducibility refers to the reproduction of the results of a study under statistical equivalent conditions. The results should be statistically compatible but not identical.
  • Computational reproducibility This is the case for research involving more or less important calculus on (super-)computers and including complete software environnements. With same code and same data, a researcher should be able to reproduce the same calculus.

Our context

Our context is that of computational science, i.e. around codes, software and software environnement.

Reproducibility vs replicability

Lorena A. Barba wrote in 2018 an interesting study about the history of the vocabulary and the difference between reproducibility and replicability : * Reproducible research Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results * Replication A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses

Initially, the Association for Computing Machinery(ACM), had chosen different terminology as can be read in this document). More recently (in 2020), they decided to align their terminology with the broader use of this term as described in this document.

As a consequence we seem to have a consensus for the definitions presented by Lorena A. Barba, as can be seen in different publications. As an example, the Rescience C journal introduces in its presentation the choices made to clarify their use of definition here.

And finally, The Turing Way, a publication from the Turing institute, illustrates with various figures and drawing the meaning of different terms: definitions.

In our context, we choose to define the computational reproducibility with these terms:

Définitions

Computational reproducibility

Computational reproducibility means "obtaining the same results when using the same analysis software applied to the same input data". Although this may seem straightforward, the difficulty resides in defining precisely what "the same" means for each of the terms in the definition.

"Same results" may vary from bit-wise reproducible results to confidence intervals or any other metric appropriate for the scientific use-case (e.g., number of significant digits, Dice coefficient, Hausdorff distance, etc.)

"Same analysis software" may refer to the same software version, same git commit, same executable, etc... Same analysis software may or may not take into account dependencies versions or the more general computing environment.

Last but not least, "same input data" may refer to large data bases or specific files that may undergo a certain number of pre-processing transformations before the main analysis software.

To go further

Computational reproducibility requires transparency and good practices:

  • access to source code and data
  • source code and data corectly identified (identifiers + metadata)
  • being able to re-execute the code and re-obtain the results
  • ensuring a valid computing environment
  • knowledge/documentation on how to run the software

Bibliography

[1] Lorena A. Barba. Terminologies for Reproducible Research. 2018. url: https://arxiv.org/pdf/1802.03311

[2] Jon F. Claerbout and Martin Karrenbach. ‘Electronic documents give reproducible research a new meaning’. In: SEG Technical Program Expanded Abstracts 1992. 2005, pp. 601–604. doi: 10.1190/1.1822162

[3] Roger D. Peng. ‘Reproducible research in computational science’. In: Science (Dec. 2011), pp. 1226–1227

[4] Victoria Stodden. Reproducibility. 2014. url: https://www.edge.org/response-detail/25340

[5] Fabien C. Y. Benureau and Nicolas P. Rougier. ‘Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions’. In: Frontiers in Neuroinformatics 11 (Jan. 2018). doi: 10.3389/fninf.2017.00069, url: https://doi.org/10.3389/fninf.2017.00069

[6] Olivier Colliot, Elina Thibeau-Sutre, Ninon Burgos. ‘Reproducibility in Machine Learning for Medical Imaging’. In Reproducibility in Machine Learning for Medical Imaging (2023). doi: 10.1007/978-1-0716-3195-9_21, url: https://doi.org/10.1007/978-1-0716-3195-9_21