Skip to main content
SpringerLink
Log in

Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

The goal of this manuscript is to describe strategies for maximizing the yield of data from small samples in prevention research. We begin by discussing what “small” means as a description of sample size in prevention research. We then present a series of practical strategies for getting the most out of data when sample size is small and constrained. Our focus is the prototypic between-group test for intervention effects; however, we touch on the circumstance in which intervention effects are qualified by one or more moderators. We conclude by highlighting the potential usefulness of graphical methods when sample size is too small for inferential statistical methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Our focus on statistical power assumes a traditional null hypothesis statistical testing (NHST) approach to data analysis. We recognize the shortcomings of this approach and its frequent misuse; however, because it remains the primary approach to the analysis of data from prevention trials, it is the approach on which our analysis and recommendations focus. For readers interested in concerns about NHST and potential alternatives, Nickerson (2000) and Harlow et al. (1997) provide balanced, largely nontechnical presentations.

  2. See von Hippel (2013) for potential problems and solutions for use of these methods with small samples.

  3. An informative discussion of the use of covariates to increase statistical power is provided by Dennis et al. (2009).

  4. Information about approaches to data visualization can be found in Young (1996) and a collection of papers edited by Post et al. (2003).

References

  • Bacchetti, P., Deeks, S. G., & McCune, J. M. (2011). Breaking free of sample size dogma to perform innovative translational research. Science Translational Medicine, 3, 87. doi:10.1126/scitranslmed.3001628.

    Article  Google Scholar 

  • Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burkhart, G., Vigna-Taglianti, F., Howe, G., Masyn, K., Wang, W., Muthén, B., Stephens, P., Grey, S., & Perrino, T. (2011). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.1007/s11121-011-0207-8.

    Article  Google Scholar 

  • Carrig, M., Wirth, R. J., & Curran, P. J. (2004). A SAS macro for estimating and visualizing individual growth curves. Structural Equation Modeling: An Interdisciplinary Journal, 11, 132–149. doi:10.1207/S15328007SEM1101_9.

    Article  Google Scholar 

  • Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah: Erlbaum.

    Google Scholar 

  • Collins, L. M., Schafer, J. L., & Kam, C. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351. doi:10.1037/1082-989X.6.4.330.

    Article  CAS  PubMed  Google Scholar 

  • Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202–224. doi:10.1037/a0015826.

    Article  PubMed Central  PubMed  Google Scholar 

  • Collins, L. M., Baker, T. D., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Christiansen, B. A., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208–226. doi:10.1007/s12160-010-9253-x.

    Article  PubMed Central  PubMed  Google Scholar 

  • Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi:10.1037/a0015914.

    Article  PubMed Central  PubMed  Google Scholar 

  • Dennis, M., Francis, D. J., Cirino, P. T., Schachar, R., Barnes, M. A., & Fletcher, J. M. (2009). Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders. Journal of the International Neuropsychological Society, 15, 331–343. doi:10.1017/S1355617709090481.

    Article  PubMed Central  PubMed  Google Scholar 

  • Dumas, J. E., Lynch, A. M., Laughlin, J. E., Phillips Smith, E., & Prinz, R. J. (2001). Promoting intervention fidelity. Conceptual issues, methods, and preliminary results from the EARLY ALLIANCE prevention trial. American Journal of Preventive Medicine, 20, 38–47. doi:10.1016/S0749-3797(00)00272-5.

    Article  CAS  PubMed  Google Scholar 

  • Embretson, S. E. (1996). Item response theory models and spurious interaction effect in factorial ANOVA designs. Applied Psychological Measurement, 20, 201–212. doi:10.1177/014662169602000302.

    Article  Google Scholar 

  • Enders, C. K. (2010). Applied missing data analysis. New York: Guildford Press.

    Google Scholar 

  • Fok, C. C. T., Henry, D., & Allen, J. A (2015). Maybe small is too small a term: Introduction to advancing small sample prevention science. Prevention Science, in press.

  • Friendly, M. (1995). Exploratory and graphical methods of data analysis [Online short course]. Retrieved from http://www.datavis.ca/courses/eda/. Accessed 6 Aug 2013.

  • Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. doi:10.1146/annurev.psych.58.110405.085530.

    Article  PubMed  Google Scholar 

  • Hansen, W. B., & Collins, L. M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph. In L. M. Collins & L. A. Seitz (Eds.), Advances in data analysis for prevention intervention research (NIDA Research Monograph 142, NIH Publication No. 94–3599, pp. 184–195). Rockville, MD: National Institutes of Health.

  • Hansen, W. B., Tobler, N. S., & Graham, J. W. (1990). Attrition in substance abuse prevention research: A meta-analysis of 85 longitudinally followed cohorts. Evaluation Review, 14, 677–685. doi:10.1177/0193841X9001400608.

    Article  Google Scholar 

  • Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Mahwah: Erlbaum.

    Google Scholar 

  • Hawkins, N. G., Sanson-Fisher, R. W., Shakeshaft, A., D’Este, C., & Green, L. W. (2007). The multiple baseline design for evaluating population-based research. American Journal of Preventive Medicine, 33, 162–168. doi:10.1016/j.amepre.2007.03.020.

    Article  PubMed  Google Scholar 

  • Hoyle, R. H. (Ed.). (1999). Statistical strategies for small sample research‬. Thousand Oaks: Sage Publications.

    Google Scholar 

  • Hoyle, R. H., & Gottfredson, N. C. (2015). Sample size considerations in prevention research applications of multilevel modeling and structural equation modeling. Prevention Science. doi:10.1007/s11121-014-0489-8.

  • Kang, S., & Waller, G. (2005). Moderated multiple regression, spurious interaction effects, and IRT. Applied Psychological Measurement, 29, 87–105. doi:10.1177/0146621604272737.

    Article  Google Scholar 

  • Kratochwill, T. R., & Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144. doi:10.1037/a0017736.

    Article  PubMed  Google Scholar 

  • Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037//1082-989X.S.2.241.

    Article  CAS  PubMed  Google Scholar 

  • Post, F. H., Nielson, G. M., & Bonneau, G.-P. (Eds.) (2003). Data visualization: The state of the art. Boston: Kluwer Academic Publishers. Retrieved from http://www.springer.com/computer/image+processing/book/978-1-4020-7259-8?otherVersion=978-1-4613-5430-7

  • Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.

    Article  Google Scholar 

  • Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.

    Book  Google Scholar 

  • Supplee, L. H., Kelly, B. C., MacKinnon, D. M., & Barofsky, M. Y. (2013). Subgroup analysis in prevention and intervention science [Special issue]. Prevention Science 14(2).

  • von Hippel, P. T. (2013). The bias and efficiency of incomplete-data estimators in small univariate normal samples. Sociological Methods & Research, 42, 531–558. doi:10.1177/0049124113494582.

    Article  Google Scholar 

  • Wang, R., & Ware, J. H. (2013). Detecting moderator effects using subgroup analyses. Prevention Science, 14, 111–120. doi:10.1007/s11121-011-0221-x.

    Article  PubMed Central  PubMed  Google Scholar 

  • Young, F. W. (1996). ViSta: The visual statistics system. Chapel Hill: Thurstone Psychometric Laboratory Research Memorandum 94-1(c). Retrieved from http://forrest.psych.unc.edu/research/index.html

  • Young, F. W., & Bann, C. M. (1996). ViSta: A visual statistics system. In R. A. Stine & J. Fox (Eds.), Statistical computing environments for social research (pp. 207–236). Thousand Oaks: Sage Publications.

    Google Scholar 

  • Zand, D., Thomson, N. R., Dugan, M., Braun, J. A., Holterman-Hommes, P., & Hunter, P. L. (2006). Predictors of retention in an alcohol, tobacco, and other drug prevention study. Evaluation Review, 30, 209–222. doi:10.1177/0193841X05281160.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

During the writing of this manuscript, the authors were supported by National Institute on Drug Abuse (NIDA) Grant P30 DA023026. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NIDA.

Conflict of Interest

The authors declare that they have no conflict of interest.

Author information

Authors and Affiliations

  1. Department of Psychology and Neuroscience, Duke University, Durham, NC, 27708, USA

    Cameron R. Hopkin & Rick H. Hoyle

  2. Center for Developmental Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA

    Nisha C. Gottfredson

Authors
  1. Cameron R. Hopkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rick H. Hoyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nisha C. Gottfredson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rick H. Hoyle.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hopkin, C.R., Hoyle, R.H. & Gottfredson, N.C. Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices. Prev Sci 16, 950–955 (2015). https://doi.org/10.1007/s11121-014-0542-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-014-0542-7

Keywords

Search

Navigation