The success of a comprehensive and integrated school mental health system is dependent on data to inform decision-making. Universal screening is one such method utilized by schools to inform treatment across universal, targeted, and individual levels of service. However, the use of multiple informants (teacher and student) remains elusive due to a lack of practical and research based-aggregation guidance. The purpose of the current study is to identify the optimal numbers of rater agreement profiles, determine demographic predictors of profiles membership, and if profiles differentially predict distal academic outcomes. Nearly 24,000 teacher-student dyads were included in this study. Results suggested four profiles emerged with significant differences in academic outcomes. Considerations for research and practice are presented.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
We also conducted Little’s MCAR tests: χ2(317) = 746.88, p < .001 for student and teacher SAEBRS scores; χ2(317) = 700.51, p < .001 with gender and race covariates additionally; and χ2(507) = 3645.74, p < .001 with reading and math outcomes additionally. However, given large sample size in this study, we majorly interpreted the relationship between missingness and other variables with correlation coefficients for the missing mechanism.
As reported earlier, the missingness of gender and race were not associated with any variables in the model and thus possibly happened completely at random. When we compared the results of a model with special education status and grade which were complete (N = 24,094) to those of a model with all four predictors (N = 16,710), general findings in terms of special education status and grade remained consistent although their coefficients changed by controlling for the other predictors. Hence, we concluded that listwise deletion less likely made any notable differences in the findings. In addition, we did not consider multiple imputation which generally assumes multivariate normality that is not relevant for binary variables, gender and race.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioural and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.
Adelman, H. S., & Taylor, L. (2007). Systemic change for school improvement. Journal of Educational and Psychological Consultation, 17(1), 55–77. https://doi.org/10.1080/10474410709336590.
Auerbach, E. R., Chafouleas, S. M., Briesch, A. M., & Long, S. J. (2020). Exploring the alignment of behavior screening policies and practices in US public school districts. Journal of School Health, 90, 264–270. https://doi.org/10.1111/josh.12872.
Barrett, S., Eber, L., & Weist, M. (Eds.) (2013). Advancing educational effectiveness: Interconnecting school mental health and school-wide positive behavior support. Retrieved from http://www.pbis.org/common/cms/files/Current%20Topics/Final-Monograph.pdf.
Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. https://doi.org/10.1037/a0032475.
Berlin, K. S., Williams, N. A., & Parra, G. R. (2014). An introduction to latent variable mixture modeling (part 1): Overview and cross-sectional latent class and latent profile analyses. Journal of Pediatric Psychology, 39(2), 174–187. https://doi.org/10.1093/jpepsy/jst084.
Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K–12 schools. Education and Treatment of Children, 37(4), 611–634. https://doi.org/10.1353/etc.2014.0039.
Christ, T. J., Aranas, Y., Johnson, L., Kember, J., Kilgus, S., Kiss, A., et al. (2014). Formative Assessment System for Teachers Technical Manual. Minneapolis, MN: FastBridge Learning.
Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. John Wiley & Sons, Inc.
Cunningham, J. M., & Suldo, S. M. (2014). Accuracy of teachers in identifying elementary school students who report at-risk levels of anxiety and depression. School Mental Health, 6, 237–250. https://doi.org/10.1007/s12310-014-9125-9.
Curhan, A. L., Rabinowitz, J. A., Pas, E. T., & Bradshaw, C. P. (2020). Informant discrepancies in internalizing and externalizing symptoms in an at-risk sample: The role of parenting and school engagement. Journal of Youth and Adolescence, 49, 311–322. https://doi.org/10.1007/s10964-019-01107-x.
De Los Reyes, A. (2013). Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Development and Psychopathology, 25(3), 669–682.
De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16(3), 330–334. https://doi.org/10.1037/1040-3518.104.22.1680.
De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A. G., & Burgers, D.,& Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498.
DiStefano, C. (2012). Cluster analysis and latent class clustering techniques. In B. Laursen, T. D. Little, & N. A. Card (Eds.), Handbook of developmental research methods (pp. 645–666). Guilford Publications: New York.
Domitrovich, C. E., Bradshaw, C. P., Greenberg, M. T., Embry, D., Poduska, J. M., & Ialongo, N. S. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47(1), 71–88. https://doi.org/10.1002/pits.20452.
Dowdy, E., Dever, B. V., Raines, T. C., & Moffa, K. (2016). A preliminary investigation into the added value of multiple gates and informants in universal screening for behavioral and emotional risk. Journal of Applied School Psychology, 32(2), 178–198. https://doi.org/10.1080/15377903.2016.1165327.
Dowdy, E., & Kim, R. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.
Dowdy, E., Ritchey, K., & Kamphaus, R. W. (2010). School-based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. https://doi.org/10.1007/s12310-010-9036-3.
Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.
Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Special Issue: Raising Healthy Children, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x.
Eklund, K., & Dowdy, E. (2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40–49. https://doi.org/10.1007/s12310-013-9109-1.
Eklund, K., Kilgus, S., von der Embse, N., Beardmore, M., & Tanner, N. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29(5), 486–499. https://doi.org/10.1037/pas0000355.
Elliott, S. N., & Gresham, F. (2008). Social Skills Improvement System (SSiS)—Performance Screening Guide. San Antonio, TX: PsychCorp Pearson Education.
Evans, S. W., Stephan, S. H., & Sugai, G. (2014). Advancing research in school mental health: Introduction of a special issue on key issues in research. School Mental Health, 6, 63–67. https://doi.org/10.1007/s12310-014-9126-8.
Franklin, C. G. S., Kim, J. S., Ryan, T. N., Kelly, M. S., & Montgomery, K. L. (2012). Teacher involvement in school mental health interventions: A systematic review. Children and Youth Services Review, 34(5), 973–982. https://doi.org/10.1016/j.childyouth.2012.01.027.
Goolsby, C., Jacobs, L., Hunt, R. C., Goralnick, E., Singletary, E. M., Levy, M. J., Goodloe, J. M., Epstein, J. L., Strauss-Riggs, K., Seitz, S. R., Krohmer, J. R., Nemeth, I., Rowe, D. W., Bradley, R. N., Gestring, M. L., & Kirsch, T. D. (2018). Stop the bleed education consortium. Journal of Trauma and Acute Care Surgery, 84(1), 205–210. https://doi.org/10.1097/TA.0000000000001732.
Herman, K. C., Cohen, D., Reinke, W. M., Ostrander, R., Burrell, L., McFarlane, E. K., & Duggan, A. K. (2018). Using latent profile and transition analyses to understand patterns of informant ratings of child depressive symptoms. Journal of School Psychology, 69, 84–99. https://doi.org/10.1016/j.jsp.2018.05.004.
Holzinger, K. J., & Swineford, F. (1937). The bi-factor model. Psychometrika, 2, 41–54. https://doi.org/10.1007/BF02287965.
Iaccarino, S., von der Embse, N. P., & Kilgus, S. P. (2018). Interpretation and use of the social, academic, and emotional behavior risk screener: A latent transition approach. Journal of Psychoeducational Assessment, 37, 486–503. https://doi.org/10.1177/0734282918766650.
Kahlberg, J. R., Lane, K. L., Driscoll, S., & Wehby, J. (2011). Systematic screening for emotional and behavioral disorders at the high school level: A formidable and necessary task. Remedial and Special Education, 32(6), 506–520. https://doi.org/10.1177/0741932510362508.
Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6(4), 89–97.
Kilgus, S. P., Bowman, N. A., Christ, T. J., & Taylor, C. N. (2017). Predicting academics via behavior within an elementary sample: An evaluation of the social, academic, and emotional behavior risk screener (SAEBRS). Psychology in the Schools, 54, 246–260.
Kilgus, S. P., Chafouleas, S. M., Riley-Tillman, T. C., & von der Embse, N. P. (2014). Social, academic, and emotional behavior risk screener (SAEBRS). Minneapolis: Theodore J. Christ & Colleagues.
Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C., & Sims, W. A. (2016). Psychometric defensibility of the social, academic, and emotional behavior risk screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. https://doi.org/10.1016/j.jsp.2016.07.001.
Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. https://doi.org/10.1037/spq0000087.
Kilgus, S. P., & von der Embse, N. P. (2014). Unpublished technical manual of the social, academic, and emotional behavior risk screener.
Kilgus, S. P., Bonifay, W., von der Embse, N. P., & Taylor, C. (2018). Evidence for the interpretation of Social, Academic, and Emotional, Behavior Risk Screener (SAEBRS) scores: An argument-based approach to screener validation. Journal of School Psychology, 68, 129–141. https://doi.org/10.1016/j.jsp.2018.03.002.
Kim, E. S., Joo, S. -H., Lee, P., Wang, Y., & Stark, S. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870–887. https://doi.org/10.1080/10705511.2016.1196108.
Kim, E. S., & Wang, Y. (2019). Investigating sources of heterogeneity with three-step multilevel factor mixture modeling: Beyond testing measurement invariance in cross-national studies. Structural Equation Modeling: A Multidisciplinary Journal, 26(2), 165–181. https://doi.org/10.1080/10705511.2018.1521284.
King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the validity of the behavioral and emotional screening system in a rural elementary school: Validity of the BESS. Journal of Psychoeducational Assessment, 30(6), 527–538. https://doi.org/10.1177/0734282912440673.
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(9), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566.
Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have tier 2 and tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182. https://doi.org/10.1080/1045988X.2014.895573.
Laursen, B., & Hoff, E. (2006). Person-centered and variable-centered approaches to longitudinal data. Merrill-Palmer Quarterly, 52(3), 377–389.
Lubke, G., & Neale, M. C. (2008). Distinguishing between latent classes and continuous factors with categorical outcomes: Class invariance of parameters of factor mixture models. Multivariate Behavioral Research, 43(4), 592–620. https://doi.org/10.1080/00273170802490673.
Lyon, A. R., Maras, M. A., Pate, C. M., Igusa, T., & Stoep, A. V. (2016). Modeling the impact of school-based universal depression screening on additional service capacity needs: A system dynamics approach. Administration and Policy in Mental Health and Mental Health Services Research, 43, 168–188. https://doi.org/10.1007/s10488-015-0628-y.
Makol, B. A., Youngstrom, E. A., Racz, S. J., Qasmieh, N., Glenn, L. E., & De Los Reyes, A. (2020). Integrating multiple informants’ reports: How conceptual and measurement models may address long-standing problems in clinical decision-making. Clinical Psychological Science.
Masyn, K. E. (2013). 25 latent class analysis and finite mixture modeling. The Oxford handbook of quantitative methods, 551
Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling, 16, 191–225.
Meyer, J. P., & Morin, A. J. S. (2016). A person-centered approach to commitment research: Theory, research, and methodology. Journal of Organizational Behavior, 37, 584–612. https://doi.org/10.1002/job.2085.
Pastor, D. A., Barron, K. E., Miller, B. J., & Davis, S. L. (2007). A latent profile analysis of college students’ achievement goal orientation. Contemporary Educational Psychology, 32, 8–47.
Pendergast, L., von der Embse, N. P., Kilgus, S. P., & Eklund, K. (2017). Measurement equivalence in school psychology research: A primer and illustrated example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82.
Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., Hedden, S. L., Crosby, A. E., Visser, S. N., Schieve, L. A., Parks, S. E., Hall, J. E., Brody, D., Simile, C. M., Thompson, W. W., Baio, J., Avenevoli, S., Kogan, M. D., & Huang, L. N. (2013). Mental health surveillance among children--United States, 2005-2011. Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report, 62(2), 1–35.
Romer, N., von der Embse, N., Eklund, K., Kilgus, S., Perales, K., Splett, J. W., Sudlo, S., Wheeler, D. (2020). Best Practices in Social, Emotional, and Behavioral Screening: An Implementation Guide. Version 2.0. From www.smhcollaborative.org/universalscreening
Shelton, A. J., Wang, C. D., & Zhu, W. (2017). Perceived social support and mental health: Cultural orientations as moderators. Journal of College Counseling, 20(3), 194–207.
Skiba, R. J., Michael, R. S., Nardo, A. C., & Peterson, R. L. (2002). The color of discipline: Sources of racial and gender disproportionality in school punishment. The Urban Review, 34(4), 317–342.
Sklad, M., Diekstra, R., Ritter, M., Ben, J., & Gravesteijn, C. (2012). Effectiveness of school-based universal social, emotional, behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment? Psychology in the Schools, 49(9), 892–909. https://doi.org/10.1002/pits.21641.
Substance Abuse and Mental Health Services Administration. (2014). National Survey on Drug Use and Health: Summary of Methodological Studies, 1971–2014 [Internet]. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2014 Nov. Available from: https://www.ncbi.nlm.nih.gov/books/NBK519735/
Taylor, C., Allen, A., Kilgus, S. P., von der Embse, N. P., & Garbacz, S. A. (2018). Development and validation of a parent version of the social, academic, and emotional behavior risk screener (SAEBRS) in an elementary sample. Behavioral Disorders., 44, 205–214. https://doi.org/10.1177/0198742918798561.
Tein, J., Coxe, S., & Cham, H. (2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling, 20(4), 640–657. https://doi.org/10.1080/10705511.2013.824781.
Vermunt, J. K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469. https://doi.org/10.1093/pan/mpq025.
von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017). Development and factor structure of the Social, Academic, and Emotional Behavior Risk Screener Student Rating Scale (SAEBRS-SRS). Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.
von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017a). Development and validation of the social, academic, and emotional behavior risk screener-student rating scale. Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.
von der Embse, N. P., Kilgus, S. P., Iaccarino, S., & Levi-Nielsen, S. (2017b). Screening for student mental health risk: Diagnostic accuracy, measurement invariance, and predictive validity of the social, academic, and emotional behavior risk screener-student rating scale (SAEBRS-SRS). School Mental Health, 9(3), 273–283. https://doi.org/10.1007/s12310-017-9214-7.
von der Embse, N. P., Kilgus, S. P., Eklund, K., Ake, E., & Levi-Neilsen, S. (2018). Training teachers to facilitate early identification of mental and behavioral health risks. School Psychology Review, 47(4), 372–384. https://doi.org/10.17105/SPR-2017-0094.V47-4.
von der Embse, N., Kim, E. S., Kilgus, S., Dedrick, R., & Sanchez, A. (2019). Multi-informant universal screening: Evaluation of rater, item, and construct variance using a trifactor model. Journal of School Psychology, 77, 52–66. https://doi.org/10.1016/j.jsp.2019.09.005.
von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. (2016). Evaluating the applied use of a mental health screener: Structural validity of the social, academic, and emotional behavior risk screener (SAEBRS). Psychological Assessment, 28(10), 1265–1275. https://doi.org/10.1037/pas0000253.
Youngstrom, E. A., Halverson, T. F., Youngstrom, J. K., Lindhiem, O., & Findling, R. L. (2018). Evidence-based assessment from simple clinical judgments to statistical learning: Evaluating a range of options using pediatric bipolar disorder as a diagnostic challenge. Clinical Psychological Science, 6(2), 243–265. https://doi.org/10.1177/2167702617741845.
Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038–1050. https://doi.org/10.1037//0022-006X.68.6.1038.
Availability of Data and Material
No funding was used to support the present investigation.
Conflicts of Interest/Competing Interests
The authors report no conflicts of interest or competing interests.
The authors report no experiment participants as these data are from a de-identified existing dataset
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
von der Embse, N., Kim, E., Jenkins, A. et al. Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes. J Psychopathol Behav Assess (2021). https://doi.org/10.1007/s10862-021-09869-0
- Universal screening
- School mental health
- Latent profile analysis