Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review
Meta-analyses are used to make educational decisions in policy and practice. Publication bias refers to the extent to which published literature is more likely to have statistically significant results and larger sample sizes than studies that do not make it through the publication process. The purpose of the present study is to estimate the extent to which publication bias is present in a broad set of education and special education journals. We reviewed 222 meta-analyses to describe the prevalence of publication bias tests, and further identified 29 that met inclusion criteria for effect size extraction. Descriptive data reveal that 58% of meta-analyses (n = 128) documented no effort to test for possible publication bias, and analyses of 72 difference statistics revealed that published studies were associated with significantly larger effect sizes than unpublished studies (d = 0.64). Exploratory moderator analyses revealed that effect size metric was a significant predictor of the difference between published and unpublished studies.
KeywordsMeta-review Meta-analysis Publication bias
References preceded by asterisks were included in the meta-review.
- *Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: a meta-analysis. Journal of Educational Psychology, 104(1), 250. https://doi.org/10.1037/a0026147.
- *Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017.
- *Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: a meta-analytic review. Educational Psychologist, 48(2), 87–113. https://doi.org/10.1080/00461520.2013.775712.
- *Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom management self-efficacy and burnout: a multivariate meta-analysis. Educational Psychology Review, 26(1), 101–126. https://doi.org/10.1007/s10648-013-9244-0.
- Becker, B. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 35–48). Chichester: Wiley.Google Scholar
- Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088–1101. https://doi.org/10.2307/2533446.
- *Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400. https://doi.org/10.1037/a0031084.
- Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders. https://doi.org/10.1177/0198742917709473.
- *Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: a meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8.
- Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origin and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Boren- stein (Eds.), Publication bias in metaanalysis: Prevention, assessment and adjustments (pp.11–33). Chichester: England, Wiley.Google Scholar
- *Dunst, C. J. (2014). Meta-analysis of the effects of puppet shows on attitudes toward and knowledge of individuals with disabilities. Exceptional Children, 80(2), 136–148. https://doi.org/10.1177/001440291408000201.
- Every Student Succeeds Act of 2015 (2015). Pub. L. No. 114-95 § 114 Stat. 1177.Google Scholar
- *Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53(3), 199–208. https://doi.org/10.1177/004005998401600318.
- *Gajda, A., Karwowski, M., & Beghetto, R. A. (2016). Creativity and academic achievement: a meta-analysis. Journal of Educational Psychology. Advance online publication: https://doi.org/10.1037/edu0000133.
- *Ginns, P., Martin, A. J., & Marsh, H. W. (2013). Designing instructional text in a conversational style: a meta-analysis. Educational Psychology Review, 25(4), 445–472. https://doi.org/10.1007/s10648-013-9228-0.
- Higgins, JPT, Green S (editors). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.
- Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Library.Google Scholar
- Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. Publication bias in meta-analysis: prevention, assessment and adjustments, 48–72.Google Scholar
- Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library.Google Scholar
- *Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: a meta-analysis. Journal of Educational Psychology, 104(1), 48–73. https://doi.org/10.1037/a0026223.
- Individuals with Disabilities Education Act (IDEA) of 2004 (2004) 20 U.S.C. § 1401 et seq.Google Scholar
- *Klassen, R. M., Tze, V. M., & Hannok, W. (2011). Internalizing problems of adults with learning disabilities: a meta-analysis. Journal of Learning Disabilities, 46(4), 317–327. https://doi.org/10.1177/0022219411422260.
- Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66(3), 357–373.Google Scholar
- Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration. https://doi.org/10.4073/cmg.2016.1.
- *Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19–26. https://doi.org/10.1037/0022-06188.8.131.52.
- *Lee, J., & Yoon, S. Y. (2015). The effects of repeated reading on reading fluency for students with reading disabilities: a meta-analysis. Journal of Learning Disabilities. https://doi.org/10.1177/0022219415605194.
- *Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243. https://doi.org/10.1007/s10648-010-9125-8.
- Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.Google Scholar
- *Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037222.
- *Maeda, Y., & Yoon, S. Y. (2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT: R). Educational Psychology Review, 25(1), 69–94. https://doi.org/10.1007/s10648-012-9215-x.
- *Murawski, W. W., & Swanson, H. L. (2001). A meta-analysis of co-teaching research where are the data? Remedial and Special Education, 22(5), 258–267. https://doi.org/10.1177/074193250102200501.
- *Nelson, J. M., & Harwood, H. (2011). Learning disabilities and anxiety: a meta-analysis. Journal of Learning Disabilities, 44(1), 3–17. https://doi.org/10.1177/0022219409359939.
- *Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: a meta-analysis. School Psychology Review, 44(2), 224–240. https://doi.org/10.17105/spr-14-0008.1.
- *Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: a systematic review and meta-analysis. Educational Psychologist, 51(2), 129–145. https://doi.org/10.1080/00461520.2016.1151793.
- *Pinquart, M. (2016). Associations of parenting styles and dimensions with academic achievement in children and adolescents: a meta-analysis. Educational Psychology Review, 28(3), 475–493. https://doi.org/10.1007/s10648-015-9338-y.
- Rothstein, H. R., Bornstein, M., & Sutton, A. J. (2006). Publication bias in meta-analysis. Prevention, assessment, and adjustments. Chichester: Wiley.Google Scholar
- *Schwinger, M., Wirthwein, L., Lemmer, G., & Steinmayr, R. (2014). Academic self-handicapping and achievement: a meta-analysis. Journal of Educational Psychology, 106(3), 744–761. https://doi.org/10.1037/a0035832.
- Spaulding, L. S. (2009). Best practices and interventions in special education: how do we know what works? Teaching Exceptional Children Plus, 5(3), 1–13.Google Scholar
- *Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. https://doi.org/10.1037/a0032447.
- Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 145–174). Chichester: Wiley.Google Scholar
- Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.Google Scholar
- *Swanson, H. L. (1999). Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532. https://doi.org/10.1177/002221949903200605.
- *Tenenbaum, H. R., & Ruck, M. D. (2007). Are teachers’ expectations different for racial minority than for European American students? A meta-analysis. Journal of Educational Psychology, 99(2), 253–273. https://doi.org/10.1037/0022-06184.108.40.206.
- *Theule, J., Wiener, J., Tannock, R., & Jenkins, J. M. (2012). Parenting stress in families of children with ADHD a meta-analysis. Journal of Emotional and Behavioral Disorders, 21(1), 3–17. https://doi.org/10.1177/1063426610387433.
- *Vasquez, A. C., Patall, E. A., Fong, C. J., Corrigan, A. S., & Pine, L. (2016). Parent autonomy support, academic achievement, and psychosocial functioning: a meta-analysis of research. Educational Psychology Review, 28(3), 605–644. https://doi.org/10.1007/s10648-015-9329-z.
- *Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. https://doi.org/10.1037/a0031311.