Improving the Power of an Efficacy Study of a Social and Emotional Learning Program: Application of Generalizability Theory to the Measurement of Classroom-Level Outcomes
- 1.2k Downloads
Social and emotional learning programs are designed to improve the quality of social interactions in schools and classrooms in order to positively affect students’ social, emotional, and academic development. The statistical power of group randomized trials to detect effects of social and emotional learning programs and other preventive interventions on setting-level outcomes is influenced by the reliability of the outcome measure. In this paper, we apply generalizability theory to an observational measure of the quality of classroom interactions that is an outcome in a study of the efficacy of a social and emotional learning program called The Recognizing, Understanding, Labeling, Expressing, and Regulating emotions Approach. We estimate multiple sources of error variance in the setting-level outcome and identify observation procedures to use in the efficacy study that most efficiently reduce these sources of error. We then discuss the implications of using different observation procedures on both the statistical power and the monetary costs of conducting the efficacy study.
KeywordsGeneralizability theory Setting-level outcomes Interventions Social and emotional learning
This research was supported by grants from the William T. Grant Foundation (#8364 and #11456). The authors wish to thank J. Patrick Meyer, Howard Bloom, and Steven Raudenbush for their comments about this manuscript.
- Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 115–172). New York: Sage.Google Scholar
- Cairns, R. B., & Green, J. A. (1979). How to assess personality and social patterns: Observations or ratings? In R. B. Cairns (Ed.), The analysis of social interactions: Methods, issues, and illustrations (pp. 209–226). Hillsdale: Lawrence Erlbaum.Google Scholar
- Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.Google Scholar
- Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students' social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x.CrossRefPubMedGoogle Scholar
- Greenberg, M. T., Weissberg, R. P., O'Brien, M. U., Zins, J. E., Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58, 466–474. doi: 10.1037/0003-066X.58.6-7.466.CrossRefPubMedGoogle Scholar
- Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.Google Scholar
- Liu, X., Spybrook, J., Congdon, R., Martinez, A., & Raudenbush, S. W. (2009). Optimal design for longitudinal and multilevel research (version 2) [computer software]. Available at http://www.wtgrantfoundation.org.
- MacKinnon, D. P. (1994). Analysis of mediating variables in prevention intervention studies. In A. Cazares & L. A. Beatty (Eds.), Scientific methods for prevention intervention research. NIDA research monograph 139, DHHS pub. 94-3631 (pp. 127–153). Washington, DC: US Department of Health and Human Services.Google Scholar
- Murray, D. M. (1998). Design and analysis of group-randomized trials. New York: Oxford University Press.Google Scholar
- Pianta, R. C., La Paro, K., & Hamre, B. K. (2008). Classroom assessment scoring system (CLASS). Baltimore: Paul H. Brookes.Google Scholar
- Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park: Sage.Google Scholar
- Suen, H. K. (1990). Principles of test theories. Hillsdale: Lawrence Erlbaum.Google Scholar
- Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale: Lawrence Erlbaum.Google Scholar
- Zins, J. E., Bloodworth, M. R., Weissberg, R. P., & Walberg, H. J. (2004). The scientific base linking social and emotional learning to school success. In J. E. Zins, R. P. Weissberg, M. C. Wang, & H. J. Walberg (Eds.), Building academic success on social and emotional learning: What does the research say? (pp. 3–22). New York: Teachers College Press.Google Scholar