Reproducible Research

Biostat M280

Reproducible research in statistics/data science

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

Buckheit and Donoho (1995)

Non-reproducible research | Duke Potti scandal

---

• Potti et al. (2006): https://www.nature.com/articles/nm1491

• Baggerly and Coombes (2009): https://projecteuclid.org/euclid.aoas/1267453942

• Simply Statistics Blog: The Duke Saga Starter Set

Non-reproducible research | Microarray studies

Nature Genetics (2015 Impact Factor: 31.616). 20 articles about microarray profiling published in Nature Genetics between Jan 2005 and Dec 2006.

Non-reproducible research | Bible code

• Witztum, Rips, and Rosenberg (1994)

• McKay et al. (1999)

Why reproducible research

• Reproducibility has been the foundation of science. It helps accumulate scientific knowledge.

• Greater research impact.

• Better work habit boosts quality of research.

• Better teamwork. For you as graduate students, it means better communication with your advisor.

  while true  
    Student: "that idea you told me to try - it doesn't work!"  
    Professor: "ok. how about trying this instead."
  end

  Unless you reproduce the computing environment (algorithms, dataset, tuning parameters), others cannot help you.

How to be reproducible in statistics/data science?

When we publish articles containing figures which were generated by computer, we also publish the complete software environment which generates the figures.

Buckheit and Donoho (1995)

• A good example: http://stanford.edu/~boyd/papers/admm_distr_stats.html

• I highly recommend the book Reproducible Research with R and RStudio by Christopher Gandrud.

  • Amazon

  • GitHub repo

Tools for reproducible research

• Version control: Git+GitHub.

• Distribute method implementation, e.g., R packages, on GitHub or bitbucket.

• Dynamic document: RMarkdown for R or Jupyter for Julia/Python/R.

• Docker container for reproducing a computing environment.

• Cloud computing tools.

• We are going to practice reproducible research now. That is to make your homework reproducible using Git, GitHub, and RMarkdown.

References

Baggerly, Keith A., and Kevin R. Coombes. 2009. “Deriving Chemosensitivity from Cell Lines: Forensic Bioinformatics and Reproducible Research in High-Throughput Biology.” Ann. Appl. Stat. 3 (4). The Institute of Mathematical Statistics: 1309–34. doi:10.1214/09-AOAS291.

Buckheit, JonathanB., and DavidL. Donoho. 1995. “WaveLab and Reproducible Research.” In Wavelets and Statistics, edited by Anestis Antoniadis and Georges Oppenheim, 103:55–81. Lecture Notes in Statistics. Springer New York. doi:10.1007/978-1-4612-2544-7_5.

McKay, Brendan, Dror Bar-Natan, Maya Bar-Hillel, and Gil Kalai. 1999. “Solving the Bible Code Puzzle.” Statist. Sci. 14 (2). The Institute of Mathematical Statistics: 150–73. doi:10.1214/ss/1009212243.

Potti, Anil, Holly K. Dressman, Andrea Bild, and Richard F. Riedel. 2006. “Genomic signatures to guide the use of chemotherapeutics.” Nature Medicine 12 (11). [1] Center for Applied Genomics; Technology, Duke Institute for Genome Sciences; Policy, Duke University, Box 3382, Durham, North Carolina 27710, USA. [2] Department of Medicine, Duke University Medical Center, Box 31295, Durham, North Carolina 27710, USA.: Nature Publishing Group: 1294–1300. doi:10.1038/nm1491.

Witztum, Doron, Eliyahu Rips, and Yoav Rosenberg. 1994. “Equidistant Letter Sequences in the Book of Genesis.” Statist. Sci. 9 (3). The Institute of Mathematical Statistics: 429–38. doi:10.1214/ss/1177010393.