Abstract
This chapter discusses some of the pitfalls encountered when performing biomedical research involving high-throughput “omics” data and presents some strategies and guidelines that researchers should follow when undertaking such studies. We discuss common errors in experimental design and data analysis that lead to irreproducible and non-replicable research and provide some guidelines to avoid these common mistakes so that researchers may have confidence in study outcomes, even if the results are negative. We discuss the importance of ranking and prespecifying hypotheses, performing power analysis, careful experimental design, and preplanning of statistical analyses in order to avoid the “fishing expedition” data analysis strategy, which is doomed to fail. The impact of multiple testing on false-positive rates is discussed, particularly in the context of the analysis of high-throughput data, and methods to correct for it are presented, as well as approaches to detect and correct for experimental biases and batch effects, which often plague high-throughput assays. We highlight the importance of sharing data and analysis code to facilitate reproducibility and present tools and software that are appropriate for this purpose.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jager LR, Leek JT (2014) An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15:1–12
Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2:e124
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR (1991) Publication bias in clinical research. Lancet 337:867–872
Goodman S, Greenland S (2007) Why most published research findings are false: problems in the analysis. PLoS Med 4:e168
von Elm E, Egger M (2004) The scandal of poor epidemiological research. BMJ 329:868–869
Humphrey LL, Chan BKS, Sox HC (2002) Postmenopausal hormone replacement therapy and the primary prevention of cardiovascular disease. Ann Intern Med 137:273–284
Pocock SJ, Collier TJ, Dandreo KJ, de Stavola BL, Goldman MB, Kalish LA et al (2004) Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 329:883
Hutson S (2010) Data handling errors spur debate over clinical trial. Nat Med 16:618
Baggerly KA, Coombes KR (2011) What information should be required to support clinical “omics” publications? Clin Chem 57:688–690
Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227
Ioannidis JPA, Allison DB, Ball CA, Coulibaly I, Cui X, Culhane AC et al (2009) Repeatability of published microarray gene expression analyses. Nat Genet 41:149–155
Macarthur D (2012) Methods: face up to false positives. Nature 487:427–428
Sebastiani P, Solovieff N, Puca A, Hartley SW, Melista E, Andersen S et al (2011) Retraction. Science 333:404
Hunt KA, Smyth DJ, Balschun T, Ban M, Mistry V, Ahmad T et al (2012) Rare and functional SIAE variants are not associated with autoimmune disease risk in up to 66,924 individuals of European ancestry. Nat Genet 44:3–5
Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10:405–408
McNutt M (2014) Journals unite for reproducibility. Science 346:679
Principles and Guidelines for Reporting Preclinical Research—About NIH—National Institutes of Health (NIH) [Internet]. [cited 10 Sep 2015]. http://www.nih.gov/about/reporting-preclinical-research.htm
Noble WS (2009) How does multiple testing correction work? Nat Biotechnol 27:1135–1137
Storey JD (2002) A direct approach to false discovery rates. J R Stat Soc B Stat Methodol 64:479–498
Hochberg Y (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75:800–802
Hommel G (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75:383–386
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B Stat Methodol 57:289–300
Yekutieli D, Benjamini Y (1999) Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plan Infer 82:171–196
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand Stat Theory Appl 6:65–70
Dudoit S, Shaffer JP, Boldrick JC (2003) Multiple hypothesis testing in microarray experiments. Stat Sci 18:71–103
Shaffer JP (1995) Multiple hypothesis testing. Annu Rev Psychol 46:561–584
Sarkar SK (1998) Some probability inequalities for ordered MTP2 random variables: a proof of the Simes conjecture. Ann Stat 26:494–504
Sarkar SK, Chang C-K (1997) The Simes method for multiple hypothesis testing with positively dependent test statistics. J Am Stat Assoc 92:1601–1608
Wright SP (1992) Adjusted P-values for simultaneous inference. Biometrics 48:1005–1013
Ihaka R, Gentleman R (1996) R: a language for data analysis and graphics. J Comput Graph Stat 5:299–314
Leek JT, Scharpf RB, Bravo HC, Simcha D, Langmead B, Johnson WE et al (2010) Tackling the widespread and critical impact of batch effects in high-throughput data. Nat Rev Genet 11:733–739
Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C et al (2001) Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat Genet 29:365–371
Lee JA, Spidlen J, Boyce K, Cai J, Crosbie N, Dalphin M et al (2008) MIFlowCyt: the minimum information about a flow cytometry experiment. Cytometry A 73:926–930
The Functional Genomics Data Society. Minimum Information about a high-throughput SeQuencing Experiment—MINSEQE (Draft Proposal) [Internet]. http://www.mged.org/minseqe/
Thomas L, Krebs CJ (1997) A review of statistical power analysis software. Bull Ecol Soc Am 78:126–138
Champely S (2009) pwr: basic functions for power analysis. R package version 1.1. 1. The R Foundation, Vienna, Austria
Scherer A (2009) Sources and solutions. Wiley, Chichester
Johnson WE, Li C, Rabinovic A (2007) Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8:118–127
Chen C, Grennan K, Badner J, Zhang D, Gershon E, Jin L et al (2011) Removing batch effects in analysis of expression microarray data: an evaluation of six batch adjustment methods. PLoS One 6:e17238
Hahne F, Khodabakhshi AH, Bashashati A, Wong C-J, Gascoyne RD, Weng AP et al (2010) Per-channel basis normalization methods for flow cytometry data. Cytometry A 77:121–131
Finak G, Jiang W, Krouse K, Wei C, Sanz I, Phippard D et al (2014) High-throughput flow cytometry data normalization for clinical trials. Cytometry A 85:277–286
Jones DC, Ruzzo WL, Peng X, Katze MG (2012) A new approach to bias correction in RNA-Seq. Bioinformatics 28:921–928
Buonaccorsi JP (2009) Models, methods, and applications. Chapman & Hall/CRC, New York
Maecker HT, Rinfret A, D’Souza P, Darden J, Roig E, Landry C et al (2005) Standardization of cytokine flow cytometry assays. BMC Immunol 6:13
Huang Y, Moodie Z, Li S, Self SG (2012) Comparing and combining assay measurements across laboratories via integration of paired-sample data to correct for measurement error. Stat Med 31(28):3748–3759
Bland JM, Altman DG (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Report No.: 0140-6736 (Print)r0140-6736 (Linking). pp 307–310
Dudoit S, Yang YH, Callow MJ, Speed TP (2002) Statistical methods for identifying differentially expressed genes in replicated cDNA microarray experiments. Stat Sin 12:111–140
Maecker HT, McCoy JP, Nussenblatt R (2012) Standardizing immunophenotyping for the Human Immunology Project. Nat Rev Immunol 12:191–200
Finak G, Langweiler M, Malekesmaeili M, Stanton R, Ramey J, Jaimes M et al (2014) Standardizing flow cytometry immunophenotyping: automated gating recapitulates central manual analysis with low variability. Cyto 2014. p Parallel Session 17–Flow Cytometry Data Analysis
Robinson MD, McCarthy DJ, Smyth GK (2010) edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics 26:139–140
Anders S, Huber W (2012) Differential expression of RNA-Seq data at the gene level—the DESeq package
McDavid A, Finak G, Chattopadyay PK, Dominguez M, Lamoreaux L, Ma SS et al (2013) Data exploration, quality control and testing in single-cell qPCR-based gene expression experiments. Bioinformatics 29:461–467
McDavid A, Dennis L, Danaher P, Finak G, Krouse M, Wang A et al (2014) Modeling bi-modality improves characterization of cell cycle on gene expression in single cells. PLoS Comput Biol 10:e1003696
Hicks SC, Teng M, Irizarry RA (2015) On the widespread and critical impact of systematic bias and batch effects in single-cell RNA-Seq data. bioRxiv. http://dx.doi.org/10.1101/025528
Finak G, McDavid A, Yajima M, Deng J, Gersuk V, Shalek AK et al (2015) MAST: a flexible statistical framework for assessing transcriptional changes and characterizing heterogeneity in single-cell RNA-seq data. Genome Biol 16:278
Shalek AK, Satija R, Shuga J, Trombetta JJ, Gennert D, Lu D et al (2014) Single-cell RNA-seq reveals dynamic paracrine control of cellular variation. Nature 510:263–269
Imholte GC, Sauteraud R, Korber B, Bailer RT, Turk ET, Shen X et al (2013) A computational framework for the analysis of peptide microarray antibody binding data with application to HIV vaccine profiling. J Immunol Methods 395:1–13
Finak G, Frelinger J, Jiang W, Newell EW, Ramey J, Davis MM et al (2014) OpenCyto: an open source infrastructure for scalable, robust, reproducible, and automated, end-to-end flow cytometry data analysis. PLoS Comput Biol 10:e1003806
Hahne F, LeMeur N, Brinkman RR, Ellis B, Haaland P, Sarkar D et al (2009) flowCore: a Bioconductor package for high throughput flow cytometry. BMC Bioinformatics 10:106
Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S et al (2004) Bioconductor: open software development for computational biology and bioinformatics. Genome Biol 5:R80
Cock PJA, Antao T, Chang JT, Chapman BA, Cox CJ, Dalke A et al (2009) Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics 25:1422–1423
Stajich JE, Block D, Boulez K, Brenner SE, Chervitz SA, Dagdigian C et al (2002) The Bioperl toolkit: Perl modules for the life sciences. Genome Res 12:1611–1618
Reich M, Liefeld T, Gould J, Lerner J, Tamayo P, Mesirov JP (2006) GenePattern 2.0. Nat Genet 38:500–501
Spidlen J, Barsky A, Breuer K, Carr P, Nazaire M-D, Hill BA et al (2013) GenePattern flow cytometry suite. Source Code Biol Med 8:14
Nelson EK, Piehler B, Eckels J, Rauch A, Bellew M, Hussey P et al (2011) LabKey Server: an open source platform for scientific data integration, analysis and collaboration. BMC Bioinformatics 12:71
Brusic V, Gottardo R, Kleinstein SH, Davis MM, HIPC Steering Committee (2014) Computational resources for high-dimensional immune analysis from the Human Immunology Project Consortium. Nat Biotechnol 32:146–148
Di Tommaso P, Palumbo E, Chatzou M, Prieto P, Heuer ML, Notredame C (2015) The impact of Docker containers on the performance of genomic pipelines. PeerJ 3:e1428
Boettiger C (2014) An introduction to Docker for reproducible research, with examples from the R environment. arXiv [cs.SE]
Mesirov JP (2010) Accessible reproducible research. Science 327:415–416
Leisch F (2002) Sweave, Part I: Mixing R and LaTeX. R News 2:28–31
Gentleman R, Lang DT (2004) Statistical analyses and reproducible research. Available at: http://biostats.bepress.com/bioconductor/paper2/
Allaire J, Cheng J, Xie Y, McPherson J, Chang W, Allen J et al (2015) rmarkdown: dynamic documents for R. R package version 0.5
Acknowledgments
This work was supported by a Bill and Melinda Gates Foundation grant, the Vaccine Immunology Statistical Center, and NIH grants U01 AI068635-01, U19 AI089986-01, and R01 EB00840-08.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Science+Business Media New York
About this protocol
Cite this protocol
Finak, G., Gottardo, R. (2016). Promises and Pitfalls of High-Throughput Biological Assays. In: Carugo, O., Eisenhaber, F. (eds) Data Mining Techniques for the Life Sciences. Methods in Molecular Biology, vol 1415. Humana Press, New York, NY. https://doi.org/10.1007/978-1-4939-3572-7_12
Download citation
DOI: https://doi.org/10.1007/978-1-4939-3572-7_12
Published:
Publisher Name: Humana Press, New York, NY
Print ISBN: 978-1-4939-3570-3
Online ISBN: 978-1-4939-3572-7
eBook Packages: Springer Protocols