Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes

Abstract

The success of a comprehensive and integrated school mental health system is dependent on data to inform decision-making. Universal screening is one such method utilized by schools to inform treatment across universal, targeted, and individual levels of service. However, the use of multiple informants (teacher and student) remains elusive due to a lack of practical and research based-aggregation guidance. The purpose of the current study is to identify the optimal numbers of rater agreement profiles, determine demographic predictors of profiles membership, and if profiles differentially predict distal academic outcomes. Nearly 24,000 teacher-student dyads were included in this study. Results suggested four profiles emerged with significant differences in academic outcomes. Considerations for research and practice are presented.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    We also conducted Little’s MCAR tests: χ2(317) = 746.88, p < .001 for student and teacher SAEBRS scores; χ2(317) = 700.51, p < .001 with gender and race covariates additionally; and χ2(507) = 3645.74, p < .001 with reading and math outcomes additionally. However, given large sample size in this study, we majorly interpreted the relationship between missingness and other variables with correlation coefficients for the missing mechanism.

  2. 2.

    As reported earlier, the missingness of gender and race were not associated with any variables in the model and thus possibly happened completely at random. When we compared the results of a model with special education status and grade which were complete (N = 24,094) to those of a model with all four predictors (N = 16,710), general findings in terms of special education status and grade remained consistent although their coefficients changed by controlling for the other predictors. Hence, we concluded that listwise deletion less likely made any notable differences in the findings. In addition, we did not consider multiple imputation which generally assumes multivariate normality that is not relevant for binary variables, gender and race.

References

  1. Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioural and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.

  2. Adelman, H. S., & Taylor, L. (2007). Systemic change for school improvement. Journal of Educational and Psychological Consultation, 17(1), 55–77. https://doi.org/10.1080/10474410709336590.

    Article  Google Scholar 

  3. Auerbach, E. R., Chafouleas, S. M., Briesch, A. M., & Long, S. J. (2020). Exploring the alignment of behavior screening policies and practices in US public school districts. Journal of School Health, 90, 264–270. https://doi.org/10.1111/josh.12872.

    Article  Google Scholar 

  4. Barrett, S., Eber, L., & Weist, M. (Eds.) (2013). Advancing educational effectiveness: Interconnecting school mental health and school-wide positive behavior support. Retrieved from http://www.pbis.org/common/cms/files/Current%20Topics/Final-Monograph.pdf.

  5. Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. https://doi.org/10.1037/a0032475.

  6. Berlin, K. S., Williams, N. A., & Parra, G. R. (2014). An introduction to latent variable mixture modeling (part 1): Overview and cross-sectional latent class and latent profile analyses. Journal of Pediatric Psychology, 39(2), 174–187. https://doi.org/10.1093/jpepsy/jst084.

    Article  PubMed  Google Scholar 

  7. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.

  8. Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K–12 schools. Education and Treatment of Children, 37(4), 611–634. https://doi.org/10.1353/etc.2014.0039.

    Article  Google Scholar 

  9. Christ, T. J., Aranas, Y., Johnson, L., Kember, J., Kilgus, S., Kiss, A., et al. (2014). Formative Assessment System for Teachers Technical Manual. Minneapolis, MN: FastBridge Learning.

    Google Scholar 

  10. Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. John Wiley & Sons, Inc.

  11. Cunningham, J. M., & Suldo, S. M. (2014). Accuracy of teachers in identifying elementary school students who report at-risk levels of anxiety and depression. School Mental Health, 6, 237–250. https://doi.org/10.1007/s12310-014-9125-9.

    Article  Google Scholar 

  12. Curhan, A. L., Rabinowitz, J. A., Pas, E. T., & Bradshaw, C. P. (2020). Informant discrepancies in internalizing and externalizing symptoms in an at-risk sample: The role of parenting and school engagement. Journal of Youth and Adolescence, 49, 311–322. https://doi.org/10.1007/s10964-019-01107-x.

    Article  PubMed  Google Scholar 

  13. De Los Reyes, A. (2013). Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Development and Psychopathology, 25(3), 669–682.

    Article  Google Scholar 

  14. De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16(3), 330–334. https://doi.org/10.1037/1040-3590.16.3.330.

    Article  PubMed  Google Scholar 

  15. De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A. G., & Burgers, D.,& Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498.

    Article  PubMed  PubMed Central  Google Scholar 

  16. DiStefano, C. (2012). Cluster analysis and latent class clustering techniques. In B. Laursen, T. D. Little, & N. A. Card (Eds.), Handbook of developmental research methods (pp. 645–666). Guilford Publications: New York.

    Google Scholar 

  17. Domitrovich, C. E., Bradshaw, C. P., Greenberg, M. T., Embry, D., Poduska, J. M., & Ialongo, N. S. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47(1), 71–88. https://doi.org/10.1002/pits.20452.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Dowdy, E., Dever, B. V., Raines, T. C., & Moffa, K. (2016). A preliminary investigation into the added value of multiple gates and informants in universal screening for behavioral and emotional risk. Journal of Applied School Psychology, 32(2), 178–198. https://doi.org/10.1080/15377903.2016.1165327.

    Article  Google Scholar 

  19. Dowdy, E., & Kim, R. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.

    Google Scholar 

  20. Dowdy, E., Ritchey, K., & Kamphaus, R. W. (2010). School-based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. https://doi.org/10.1007/s12310-010-9036-3.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.

    Article  Google Scholar 

  22. Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Special Issue: Raising Healthy Children, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x.

    Article  Google Scholar 

  23. Eklund, K., & Dowdy, E. (2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40–49. https://doi.org/10.1007/s12310-013-9109-1.

    Article  Google Scholar 

  24. Eklund, K., Kilgus, S., von der Embse, N., Beardmore, M., & Tanner, N. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29(5), 486–499. https://doi.org/10.1037/pas0000355.

    Article  PubMed  Google Scholar 

  25. Elliott, S. N., & Gresham, F. (2008). Social Skills Improvement System (SSiS)—Performance Screening Guide. San Antonio, TX: PsychCorp Pearson Education.

    Google Scholar 

  26. Evans, S. W., Stephan, S. H., & Sugai, G. (2014). Advancing research in school mental health: Introduction of a special issue on key issues in research. School Mental Health, 6, 63–67. https://doi.org/10.1007/s12310-014-9126-8.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Franklin, C. G. S., Kim, J. S., Ryan, T. N., Kelly, M. S., & Montgomery, K. L. (2012). Teacher involvement in school mental health interventions: A systematic review. Children and Youth Services Review, 34(5), 973–982. https://doi.org/10.1016/j.childyouth.2012.01.027.

    Article  Google Scholar 

  28. Goolsby, C., Jacobs, L., Hunt, R. C., Goralnick, E., Singletary, E. M., Levy, M. J., Goodloe, J. M., Epstein, J. L., Strauss-Riggs, K., Seitz, S. R., Krohmer, J. R., Nemeth, I., Rowe, D. W., Bradley, R. N., Gestring, M. L., & Kirsch, T. D. (2018). Stop the bleed education consortium. Journal of Trauma and Acute Care Surgery, 84(1), 205–210. https://doi.org/10.1097/TA.0000000000001732.

    Article  Google Scholar 

  29. Herman, K. C., Cohen, D., Reinke, W. M., Ostrander, R., Burrell, L., McFarlane, E. K., & Duggan, A. K. (2018). Using latent profile and transition analyses to understand patterns of informant ratings of child depressive symptoms. Journal of School Psychology, 69, 84–99. https://doi.org/10.1016/j.jsp.2018.05.004.

    Article  PubMed  Google Scholar 

  30. Holzinger, K. J., & Swineford, F. (1937). The bi-factor model. Psychometrika, 2, 41–54. https://doi.org/10.1007/BF02287965.

  31. Iaccarino, S., von der Embse, N. P., & Kilgus, S. P. (2018). Interpretation and use of the social, academic, and emotional behavior risk screener: A latent transition approach. Journal of Psychoeducational Assessment, 37, 486–503. https://doi.org/10.1177/0734282918766650.

    Article  Google Scholar 

  32. Kahlberg, J. R., Lane, K. L., Driscoll, S., & Wehby, J. (2011). Systematic screening for emotional and behavioral disorders at the high school level: A formidable and necessary task. Remedial and Special Education, 32(6), 506–520. https://doi.org/10.1177/0741932510362508.

    Article  Google Scholar 

  33. Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6(4), 89–97.

    Google Scholar 

  34. Kilgus, S. P., Bowman, N. A., Christ, T. J., & Taylor, C. N. (2017). Predicting academics via behavior within an elementary sample: An evaluation of the social, academic, and emotional behavior risk screener (SAEBRS). Psychology in the Schools, 54, 246–260.

    Article  Google Scholar 

  35. Kilgus, S. P., Chafouleas, S. M., Riley-Tillman, T. C., & von der Embse, N. P. (2014). Social, academic, and emotional behavior risk screener (SAEBRS). Minneapolis: Theodore J. Christ & Colleagues.

    Google Scholar 

  36. Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C., & Sims, W. A. (2016). Psychometric defensibility of the social, academic, and emotional behavior risk screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. https://doi.org/10.1016/j.jsp.2016.07.001.

    Article  PubMed  Google Scholar 

  37. Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. https://doi.org/10.1037/spq0000087.

    Article  PubMed  Google Scholar 

  38. Kilgus, S. P., & von der Embse, N. P. (2014). Unpublished technical manual of the social, academic, and emotional behavior risk screener.

  39. Kilgus, S. P., Bonifay, W., von der Embse, N. P., & Taylor, C. (2018). Evidence for the interpretation of Social, Academic, and Emotional, Behavior Risk Screener (SAEBRS) scores: An argument-based approach to screener validation. Journal of School Psychology, 68, 129–141. https://doi.org/10.1016/j.jsp.2018.03.002.

    Article  PubMed  Google Scholar 

  40. Kim, E. S., Joo, S. -H., Lee, P., Wang, Y., & Stark, S. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870–887. https://doi.org/10.1080/10705511.2016.1196108.

  41. Kim, E. S., & Wang, Y. (2019). Investigating sources of heterogeneity with three-step multilevel factor mixture modeling: Beyond testing measurement invariance in cross-national studies. Structural Equation Modeling: A Multidisciplinary Journal, 26(2), 165–181. https://doi.org/10.1080/10705511.2018.1521284.

    Article  Google Scholar 

  42. King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the validity of the behavioral and emotional screening system in a rural elementary school: Validity of the BESS. Journal of Psychoeducational Assessment, 30(6), 527–538. https://doi.org/10.1177/0734282912440673.

    Article  Google Scholar 

  43. Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(9), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566.

    Article  Google Scholar 

  44. Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have tier 2 and tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182. https://doi.org/10.1080/1045988X.2014.895573.

    Article  Google Scholar 

  45. Laursen, B., & Hoff, E. (2006). Person-centered and variable-centered approaches to longitudinal data. Merrill-Palmer Quarterly, 52(3), 377–389.

    Article  Google Scholar 

  46. Lubke, G., & Neale, M. C. (2008). Distinguishing between latent classes and continuous factors with categorical outcomes: Class invariance of parameters of factor mixture models. Multivariate Behavioral Research, 43(4), 592–620. https://doi.org/10.1080/00273170802490673.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Lyon, A. R., Maras, M. A., Pate, C. M., Igusa, T., & Stoep, A. V. (2016). Modeling the impact of school-based universal depression screening on additional service capacity needs: A system dynamics approach. Administration and Policy in Mental Health and Mental Health Services Research, 43, 168–188. https://doi.org/10.1007/s10488-015-0628-y.

    Article  PubMed  Google Scholar 

  48. Makol, B. A., Youngstrom, E. A., Racz, S. J., Qasmieh, N., Glenn, L. E., & De Los Reyes, A. (2020). Integrating multiple informants’ reports: How conceptual and measurement models may address long-standing problems in clinical decision-making. Clinical Psychological Science.

  49. Masyn, K. E. (2013). 25 latent class analysis and finite mixture modeling. The Oxford handbook of quantitative methods, 551

  50. Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling, 16, 191–225.

    Article  Google Scholar 

  51. Meyer, J. P., & Morin, A. J. S. (2016). A person-centered approach to commitment research: Theory, research, and methodology. Journal of Organizational Behavior, 37, 584–612. https://doi.org/10.1002/job.2085.

    Article  Google Scholar 

  52. Pastor, D. A., Barron, K. E., Miller, B. J., & Davis, S. L. (2007). A latent profile analysis of college students’ achievement goal orientation. Contemporary Educational Psychology, 32, 8–47.

    Article  Google Scholar 

  53. Pendergast, L., von der Embse, N. P., Kilgus, S. P., & Eklund, K. (2017). Measurement equivalence in school psychology research: A primer and illustrated example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82.

    Article  Google Scholar 

  54. Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., Hedden, S. L., Crosby, A. E., Visser, S. N., Schieve, L. A., Parks, S. E., Hall, J. E., Brody, D., Simile, C. M., Thompson, W. W., Baio, J., Avenevoli, S., Kogan, M. D., & Huang, L. N. (2013). Mental health surveillance among children--United States, 2005-2011. Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report, 62(2), 1–35.

    Google Scholar 

  55. Romer, N., von der Embse, N., Eklund, K., Kilgus, S., Perales, K., Splett, J. W., Sudlo, S., Wheeler, D. (2020). Best Practices in Social, Emotional, and Behavioral Screening: An Implementation Guide. Version 2.0. From www.smhcollaborative.org/universalscreening

  56. Shelton, A. J., Wang, C. D., & Zhu, W. (2017). Perceived social support and mental health: Cultural orientations as moderators. Journal of College Counseling, 20(3), 194–207.

    Article  Google Scholar 

  57. Skiba, R. J., Michael, R. S., Nardo, A. C., & Peterson, R. L. (2002). The color of discipline: Sources of racial and gender disproportionality in school punishment. The Urban Review, 34(4), 317–342.

    Article  Google Scholar 

  58. Sklad, M., Diekstra, R., Ritter, M., Ben, J., & Gravesteijn, C. (2012). Effectiveness of school-based universal social, emotional, behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment? Psychology in the Schools, 49(9), 892–909. https://doi.org/10.1002/pits.21641.

    Article  Google Scholar 

  59. Substance Abuse and Mental Health Services Administration. (2014). National Survey on Drug Use and Health: Summary of Methodological Studies, 1971–2014 [Internet]. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2014 Nov. Available from: https://www.ncbi.nlm.nih.gov/books/NBK519735/

  60. Taylor, C., Allen, A., Kilgus, S. P., von der Embse, N. P., & Garbacz, S. A. (2018). Development and validation of a parent version of the social, academic, and emotional behavior risk screener (SAEBRS) in an elementary sample. Behavioral Disorders., 44, 205–214. https://doi.org/10.1177/0198742918798561.

    Article  Google Scholar 

  61. Tein, J., Coxe, S., & Cham, H. (2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling, 20(4), 640–657. https://doi.org/10.1080/10705511.2013.824781.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Vermunt, J. K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469. https://doi.org/10.1093/pan/mpq025.

    Article  Google Scholar 

  63. von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017). Development and factor structure of the Social, Academic, and Emotional Behavior Risk Screener Student Rating Scale (SAEBRS-SRS). Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.

    Article  Google Scholar 

  64. von der Embse, N. P., Iaccarino, S., Mankin, A., Kilgus, S., & Magen, E. (2017a). Development and validation of the social, academic, and emotional behavior risk screener-student rating scale. Assessment for Effective Intervention, 42(3), 186–192. https://doi.org/10.1177/1534508416679410.

    Article  Google Scholar 

  65. von der Embse, N. P., Kilgus, S. P., Iaccarino, S., & Levi-Nielsen, S. (2017b). Screening for student mental health risk: Diagnostic accuracy, measurement invariance, and predictive validity of the social, academic, and emotional behavior risk screener-student rating scale (SAEBRS-SRS). School Mental Health, 9(3), 273–283. https://doi.org/10.1007/s12310-017-9214-7.

    Article  Google Scholar 

  66. von der Embse, N. P., Kilgus, S. P., Eklund, K., Ake, E., & Levi-Neilsen, S. (2018). Training teachers to facilitate early identification of mental and behavioral health risks. School Psychology Review, 47(4), 372–384. https://doi.org/10.17105/SPR-2017-0094.V47-4.

    Article  Google Scholar 

  67. von der Embse, N., Kim, E. S., Kilgus, S., Dedrick, R., & Sanchez, A. (2019). Multi-informant universal screening: Evaluation of rater, item, and construct variance using a trifactor model. Journal of School Psychology, 77, 52–66. https://doi.org/10.1016/j.jsp.2019.09.005.

    Article  PubMed  Google Scholar 

  68. von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. (2016). Evaluating the applied use of a mental health screener: Structural validity of the social, academic, and emotional behavior risk screener (SAEBRS). Psychological Assessment, 28(10), 1265–1275. https://doi.org/10.1037/pas0000253.

    Article  PubMed  Google Scholar 

  69. Youngstrom, E. A., Halverson, T. F., Youngstrom, J. K., Lindhiem, O., & Findling, R. L. (2018). Evidence-based assessment from simple clinical judgments to statistical learning: Evaluating a range of options using pediatric bipolar disorder as a diagnostic challenge. Clinical Psychological Science, 6(2), 243–265. https://doi.org/10.1177/2167702617741845.

    Article  PubMed  Google Scholar 

  70. Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038–1050. https://doi.org/10.1037//0022-006X.68.6.1038.

    Article  PubMed  Google Scholar 

Download references

Availability of Data and Material

Not applicable.

Funding

No funding was used to support the present investigation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Nathaniel von der Embse.

Ethics declarations

Conflicts of Interest/Competing Interests

The authors report no conflicts of interest or competing interests.

Experiment Participants

The authors report no experiment participants as these data are from a de-identified existing dataset

Code Availability

Not applicable.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

von der Embse, N., Kim, E., Jenkins, A. et al. Profiles of Rater Dis/Agreement within Universal Screening in Predicting Distal Outcomes. J Psychopathol Behav Assess (2021). https://doi.org/10.1007/s10862-021-09869-0

Download citation

Keywords

  • Universal screening
  • School mental health
  • Latent profile analysis
  • Prevention