Prevention Science

, Volume 15, Issue 2, pp 146–155 | Cite as

Improving the Power of an Efficacy Study of a Social and Emotional Learning Program: Application of Generalizability Theory to the Measurement of Classroom-Level Outcomes

  • Andrew J. Mashburn
  • Jason T. Downer
  • Susan E. Rivers
  • Marc A. Brackett
  • Andres Martinez


Social and emotional learning programs are designed to improve the quality of social interactions in schools and classrooms in order to positively affect students’ social, emotional, and academic development. The statistical power of group randomized trials to detect effects of social and emotional learning programs and other preventive interventions on setting-level outcomes is influenced by the reliability of the outcome measure. In this paper, we apply generalizability theory to an observational measure of the quality of classroom interactions that is an outcome in a study of the efficacy of a social and emotional learning program called The Recognizing, Understanding, Labeling, Expressing, and Regulating emotions Approach. We estimate multiple sources of error variance in the setting-level outcome and identify observation procedures to use in the efficacy study that most efficiently reduce these sources of error. We then discuss the implications of using different observation procedures on both the statistical power and the monetary costs of conducting the efficacy study.


Generalizability theory Setting-level outcomes Interventions Social and emotional learning 



This research was supported by grants from the William T. Grant Foundation (#8364 and #11456). The authors wish to thank J. Patrick Meyer, Howard Bloom, and Steven Raudenbush for their comments about this manuscript.


  1. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. doi: 10.1111/j.1745-3984.1977.tb00045.x.CrossRefGoogle Scholar
  2. Brown, J. L., Jones, S. M., LaRusso, M., & Aber, J. L. (2010). Improving classroom quality: Teacher influences and experimental impacts of the 4Rs program. Journal of Educational Psychology, 102, 153–167. doi: 10.1037/a0018160.CrossRefGoogle Scholar
  3. Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 115–172). New York: Sage.Google Scholar
  4. Cairns, R. B., & Green, J. A. (1979). How to assess personality and social patterns: Observations or ratings? In R. B. Cairns (Ed.), The analysis of social interactions: Methods, issues, and illustrations (pp. 209–226). Hillsdale: Lawrence Erlbaum.Google Scholar
  5. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.Google Scholar
  6. Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students' social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x.CrossRefPubMedGoogle Scholar
  7. Erlich, O., & Borich, C. (1979). Occurrence and generalizability of scores on a classroom interaction instrument. Journal of Educational Measurement, 16, 11–18. doi: 10.1111/j.1745-3984.1979.tb00081.x.CrossRefGoogle Scholar
  8. Greenberg, M. T., Weissberg, R. P., O'Brien, M. U., Zins, J. E., Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58, 466–474. doi: 10.1037/0003-066X.58.6-7.466.CrossRefPubMedGoogle Scholar
  9. Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.Google Scholar
  10. Judd, C. M., & Kenny, D. A. (1981). Process analysis: Estimating mediation in treatment evaluations. Evaluation Review, 5, 602–619. doi: 10.1177/0193841X8100500502.CrossRefGoogle Scholar
  11. Liu, X., Spybrook, J., Congdon, R., Martinez, A., & Raudenbush, S. W. (2009). Optimal design for longitudinal and multilevel research (version 2) [computer software]. Available at
  12. MacKinnon, D. P. (1994). Analysis of mediating variables in prevention intervention studies. In A. Cazares & L. A. Beatty (Eds.), Scientific methods for prevention intervention research. NIDA research monograph 139, DHHS pub. 94-3631 (pp. 127–153). Washington, DC: US Department of Health and Human Services.Google Scholar
  13. Marcoulides, G. (1989). The application of generalizability analysis to observational studies. Quality and Quantity, 23, 115–127. doi: 10.1007/BF00151898.CrossRefGoogle Scholar
  14. Mashburn, A. J., Pianta, R. C., Hamre, B. K., Downer, J. T., Barbarin, O., et al. (2008). Measures of pre-k quality and children’s development of academic, language and social skills. Child Development, 79, 732–749. doi: 10.1111/j.1467-8624.2008.01154.x.CrossRefPubMedGoogle Scholar
  15. Meyer, J. P., Henry, A. E., & Mashburn, A. J. (2011). Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis. Educational Assessment Journal, 16, 227–243. doi: 10.1080/10627197.2011.638884.CrossRefGoogle Scholar
  16. Murray, D. M. (1998). Design and analysis of group-randomized trials. New York: Oxford University Press.Google Scholar
  17. Pianta, R. C., Belsky, J., Vandergrift, N., Houts, R., & Morrison, F. J. (2008). Classroom effects on children's achievement trajectories in elementary school. American Educational Research Journal, 45, 365–397. doi: 10.3102/0002831207308230.CrossRefGoogle Scholar
  18. Pianta, R. C., La Paro, K., & Hamre, B. K. (2008). Classroom assessment scoring system (CLASS). Baltimore: Paul H. Brookes.Google Scholar
  19. Raudenbush, S. W., & Sadoff, S. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154. doi: 10.1080/19345740801982104.CrossRefGoogle Scholar
  20. Rivers, S. E., & Brackett, M. A. (2011). Achieving standards in the English language arts (and more) using The RULER Approach to social and emotional learning. Reading & Writing Quarterly, 27, 75–100. doi: 10.1080/10573569.2011.532715.CrossRefGoogle Scholar
  21. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park: Sage.Google Scholar
  22. Snyder, J., Reid, J., Stoolmiller, M., Howe, G., Brown, H., Dagne, G., & Cross, W. (2006). The role of behavior observation in measurement systems for randomized prevention trials. Prevention Science, 7, 43–56. doi: 10.1007/s11121-005-0020-3.CrossRefPubMedGoogle Scholar
  23. Suen, H. K. (1990). Principles of test theories. Hillsdale: Lawrence Erlbaum.Google Scholar
  24. Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale: Lawrence Erlbaum.Google Scholar
  25. Zins, J. E., Bloodworth, M. R., Weissberg, R. P., & Walberg, H. J. (2004). The scientific base linking social and emotional learning to school success. In J. E. Zins, R. P. Weissberg, M. C. Wang, & H. J. Walberg (Eds.), Building academic success on social and emotional learning: What does the research say? (pp. 3–22). New York: Teachers College Press.Google Scholar

Copyright information

© Society for Prevention Research 2013

Authors and Affiliations

  • Andrew J. Mashburn
    • 1
  • Jason T. Downer
    • 2
  • Susan E. Rivers
    • 3
  • Marc A. Brackett
    • 3
  • Andres Martinez
    • 4
  1. 1.Psychology DepartmentPortland State UniversityPortlandUSA
  2. 2.University of VirginiaCharlottesvilleUSA
  3. 3.Yale UniversityNew HavenUSA
  4. 4.University of MichiganAnn ArborUSA

Personalised recommendations