t-Test and ANOVA for data with ceiling and/or floor effects

Behavior Research Methods volume 53, pages264–277(2021)Cite this article

Abstract

Ceiling and floor effects are often observed in social and behavioral science. The current study examines ceiling/floor effects in the context of the t-test and ANOVA, two frequently used statistical methods in experimental studies. Our literature review indicated that most researchers treated ceiling or floor data as if these data were true values, and that some researchers used statistical methods such as discarding ceiling or floor data in conducting the t-test and ANOVA. The current study evaluates the performance of these conventional methods for t-test and ANOVA with ceiling or floor data. Our evaluation also includes censored regression with regard to its capacity for handling ceiling/floor data. Furthermore, we propose an easy-to-use method that handles ceiling or floor data in t-tests and ANOVA by using properties of truncated normal distributions. Simulation studies were conducted to compare the performance of the methods in handling ceiling or floor data for t-test and ANOVA. Overall, the proposed method showed greater accuracy in effect size estimation and better-controlled Type I error rates over other evaluated methods. We developed an easy-to-use software package and web applications to help researchers implement the proposed method. Recommendations and future directions are discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Notes

  1. 1.

    F* can be systematically smaller than F given large samples with small variances where F is too liberal. F* can be systematically larger than F given large samples with large variances where F is too conservative. Moreover, when cell sample sizes are equal, F and F* are identical, but the denominator degrees of freedom are different. See Maxwell et al., (2018) for more details.

References

  1. Aitkin, M. A. (1964). Correlation in a singly truncated bivariate normal distribution. Psychometrika, 29(3), 263–270. https://doi.org/10.1007/BF02289723

    Article  Google Scholar 

  2. Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31(2), 144–152. https://doi.org/10.1111/j.2044-8317.1978.tb00581.x

    Article  Google Scholar 

  3. Brown, M. B., & Forsythe, A. B. (1974). Robust Tests for the Equality of Variances. Journal of the American Statistical Association, 69(346), 364. https://doi.org/10.2307/2285659

    Article  Google Scholar 

  4. Chiu, Y.-C., & Egner, T. (2015). Inhibition-Induced Forgetting: When More Control Leads to Less Memory . Psychological Science , 26(1), 27–38. https://doi.org/10.1177/0956797614553945

    Article  PubMed  Google Scholar 

  5. Cohen, A. C. J. (1959). Simplified estimators for the normal distribution when samples are single censored or truncated. Technometrics, 1(3), 217–237. https://doi.org/10.2307/1266442

    Article  Google Scholar 

  6. Coman, A., & Berry, J. N. (2015). Infectious Cognition: Risk Perception Affects Socially Shared Retrieval-Induced Forgetting of Medical Information . Psychological Science , 26(12), 1965–1971. https://doi.org/10.1177/0956797615609438

    Article  PubMed  Google Scholar 

  7. Delacre, M., Lakens, D., & Leys, C. (2017). Why Psychologists Should by Default Use Welch’s t-test Instead of Student’s t-test. International Review of Social Psychology, 30(1), 92–101. https://doi.org/10.5334/irsp.82

    Article  Google Scholar 

  8. Dompnier, B., Darnon, C., Meier, E., Brandner, C., Smeding, A., & Butera, F. (2015). Improving Low Achievers’ Academic Performance at University by Changing the Social Value of Mastery Goals. American Educational Research Journal, 52(4), 720–749. https://doi.org/10.3102/0002831215585137

    Article  Google Scholar 

  9. Fantuzzo, J. W., Gadsden, V. L., & McDermott, P. A. (2011). An Integrated Curriculum to Improve Mathematics, Language, and Literacy for Head Start Children. American Educational Research Journal, 48(3), 763–793. https://doi.org/10.3102/0002831210385446

    Article  Google Scholar 

  10. Greene, W. H. (2002). Econometric Analysis. In Econometric Analysis.

  11. Henningsen A. (2011). Censreg: Censored Regression (Tobit) Models. R package version 0.5, http://CRAN.R-project.org/package=censReg

  12. Jennings, M. A., & Cribbie, R. A. (2016). Comparing Pre-Post Change Across Groups: Guidelines for Choosing between Difference Scores, ANCOVA, and Residual Change Scores. Journal of Data Science, 14, 205–230.

    Google Scholar 

  13. Kim, R., Peters, M. A. K., & Shams, L. (2012). 0 + 1 > 1: How Adding Noninformative Sound Improves Performance on a Visual Task . Psychological Science , 23(1), 6–12. https://doi.org/10.1177/0956797611420662

    Article  PubMed  Google Scholar 

  14. Liu, Q., & Wang, L. (2018). DACF: Data Analysis with Ceiling and/or Floor Data. CRAN

  15. Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). Designing Experiments and Analyzing Data: A Model Comparison Perspective (3rd ed.). New York: Routledge.

  16. Miller GA. (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63(2):81–97. https://doi.org/10.1037/h0043158

  17. Muthen, B. (1990). Moments of the censored and truncated bivariate normal distribution. British Journal of Mathematical and Statistical Psychology, 43(1), 131–143.

    Article  Google Scholar 

  18. Muthén, L. K., & Muthén, B. O. (2002). How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 599–620. https://doi.org/10.1207/S15328007SEM0904_8

    Article  Google Scholar 

  19. Olsen, M. K., & Schafer, J. L. (2001). A Two-Part Random-Effects Model for Semicontinuous Longitudinal Data. Journal of the American Statistical Association, 96(454), 730–745. https://doi.org/10.1198/016214501753168389

    Article  Google Scholar 

  20. Piccinin, A. M., Muniz-Terrera, G., Clouston, S., Reynolds, C. A., Thorvaldsson, V., Deary, I. J., … Spiro, A. (2013). Coordinated analysis of age, sex, and education effects on change in MMSE scores. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 68(3), 374–390.

    Article  Google Scholar 

  21. Priebe, K., Kleindienst, N., Zimmer, J., Koudela, S., Ebner-Priemer, U., & Bohus, M. (2013). Frequency of intrusions and flashbacks in patients with posttraumatic stress disorder related to childhood sexual abuse: An electronic diary study. Psychological Assessment, 25(4), 1370–1376. https://doi.org/10.1037/a0033816

    Article  PubMed  Google Scholar 

  22. Salthouse, T. A. (2004). Localizing age-related individual differences in a hierarchical structure. Intelligence, 32(6), 541–561. https://doi.org/10.1016/j.intell.2004.07.003

    Article  Google Scholar 

  23. Schweizer, K. (2016). A confirmatory factor model for the investigation of cognitive data showing a ceiling effect: an example. In Quantitative Psychology Research (pp. 187–197). Springer International Publishing.

  24. Sokol-Hessner, P., Lackovic, S. F., Tobe, R. H., Camerer, C. F., Leventhal, B. L., & Phelps, E. A. (2015). Determinants of Propranolol’s Selective Effect on Loss Aversion. Psychological Science, 26(7), 1123–1130. https://doi.org/10.1177/0956797615582026

    Article  PubMed  PubMed Central  Google Scholar 

  25. Timeo, S., Farroni, T., & Maass, A. (2017). Race and Color: Two Sides of One Story? Development of Biases in Categorical Perception. Child Development, 88(1), 83–102. https://doi.org/10.1111/cdev.12564

    Article  PubMed  Google Scholar 

  26. Tobin, J. (1958). Estimation of Relationships for Limited Dependent Variables. Econometrica, 26(1), 24–36. https://doi.org/10.2307/1907382

    Article  Google Scholar 

  27. Ulber, J., Hamann, K., & Tomasello, M. (2016). Extrinsic Rewards Diminish Costly Sharing in 3-Year-Olds. Child Development, 87(4), 1192–1203. https://doi.org/10.1111/cdev.12534

    Article  PubMed  Google Scholar 

  28. Uttl, B. (2005). Measurement of Individual Differences. Psychological Science, 16(6), 460–467. https://doi.org/10.1111/j.0956-7976.2005.01557.x

    Article  PubMed  Google Scholar 

  29. Wang, L., & Zhang, Z. (2011). Estimating and Testing Mediation Effects with Censored Data. Structural Equation Modeling: A Multidisciplinary Journal, 18(1), 18–34. https://doi.org/10.1080/10705511.2011.534324

    Article  Google Scholar 

  30. Wang, L., Zhang, Z., McArdle, J. J., & Salthouse, T. A. (2008). Investigating Ceiling Effects in Longitudinal Data Analysis. Multivariate Behav Res, 43(3), 476–496. https://doi.org/10.1080/00273170802285941

    Article  Google Scholar 

  31. Welch, B. L. (1947). The generalisation of student’s problems when several different population variances are involved. Biometrika, 34(1–2), 28–35. https://doi.org/10.1093/BIOMET/34.1-2.28

    Article  PubMed  Google Scholar 

Download references

Author information

Affiliations

  1. Department of Psychology and Human Development, Vanderbilt University, Nashville, TN, 37203, USA

    Qimin Liu

  2. Department of Psychology, University of Notre Dame, 390 Debartolo Hall, Notre Dame, IN, 46556, USA

    Lijuan Wang

Authors
  1. Qimin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lijuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Qimin Liu or Lijuan Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Lijuan Wang is grateful for the support from NIH 1R01HD091235, NIH 1R01HD087319, and NIH 1R01HD088482.

Electronic supplementary material

ESM 1

(PDF 178 kb)

ESM 2

(PDF 206 kb)

ESM 3

(PDF 1273 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, Q., Wang, L. t-Test and ANOVA for data with ceiling and/or floor effects. Behav Res 53, 264–277 (2021). https://doi.org/10.3758/s13428-020-01407-2

Download citation

Keywords

  • Ceiling effect
  • Floor effect
  • t-Test
  • ANOVA