Skip to main content

Advertisement

Log in

Incremental Validity of Teacher and Parent Symptom and Impairment Ratings when Screening for Mental Health Difficulties

  • Published:
Journal of Abnormal Child Psychology Aims and scope Submit manuscript

Abstract

Although universal screening for mental health difficulties is increasingly recognized as a way to identify children who are at risk and provide early intervention, little research exists to inform decisions about screening, such as the choice of informants and the type of information collected. The present study examined the incremental validity of teacher- and parent-rated (primarily mothers) symptoms and impairment in a non-referred sample of early elementary school children (n = 320, 49 % boys, ages 6 to 9) in terms of predicting impairment as rated by a different teacher 1 year later. Teacher-rated symptoms and impairment and parent-rated impairment were each unique predictors of later impairment; however, parent-rated symptoms did not contribute to the prediction of later impairment above and beyond these other indicators. The results indicate that, when screening for mental health difficulties in the school system, impairment ratings collected across settings add useful information, but it may not be necessary to use parent symptom ratings when teacher symptom ratings are available.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. This analysis was run as a linear regression as negative binomial regression is not available for imputed data in Mplus.

References

  • Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. doi:10.1037/0033-2909.101.2.213.

    Article  PubMed  Google Scholar 

  • Aitken, M., Martinussen, R., Wolfe, R. G., & Tannock, R. (2015). Factor structure of the Strengths and Difficulties Questionnaire in a Canadian elementary school sample. Assessment for Effective Intervention, 40(3), 155–165. doi:10.1177/1534508414560347.

    Article  Google Scholar 

  • Albers, C. A., Glover, T. A., & Kratochwill, T. R. (2007). Introduction to the special issue: How can universal screening enhance educational and mental health outcomes? Journal of School Psychology, 45(2), 113–116. doi:10.1016/j.jsp.2006.12.002.

    Article  Google Scholar 

  • Allison, P. D. (2012). Regression for count data. In Logistic regression using SAS: theory and applications (pp. 265–290). Cary, NC: SAS Institute, Inc..

    Google Scholar 

  • Andrade, B. F., & Tannock, R. (2013). The direct effects of inattention and hyperactivity/impulsivity on peer problems and mediating roles of prosocial and conduct problem behaviors in a community sample of children. Journal of Attention Disorders, 17(8), 670–680. doi:10.1177/1087054712437580.

    Article  PubMed  Google Scholar 

  • Angold, A., Costello, E. J., Farmer, E. M. Z., Burns, B. J., & Erkanli, A. (1999). Impaired but undiagnosed. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2), 129–137. doi:10.1097/00004583-199902000-00011.

    Article  PubMed  Google Scholar 

  • Baker, J. A. (2008). Assessing school risk and protective factors. In B. Doll & J. A. Cummings (Eds.), Transforming school mental health services: population-based approaches to promoting the competency and wellness of children (pp. 43–68). Thousand Oaks, CA: Corwin Press.

    Google Scholar 

  • Bringewatt, E. H., & Gershoff, E. T. (2010). Falling through the cracks: gaps and barriers in the mental health system for America’s disadvantaged children. Children and Youth Services Review, 32(10), 1291–1299. doi:10.1016/j.childyouth.2010.04.021.

    Article  Google Scholar 

  • Costello, E. J., Angold, A., & Keeler, G. P. (1999). Adolescent outcomes of childhood disorders: The consequences of severity and impairment. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2), 121–128. doi:10.1097/00004583-199902000-00010.

    Article  PubMed  Google Scholar 

  • Dadds, M. R., Holland, D. E., Laurens, K. R., Mullins, M., Barrett, P. M., & Spence, S. H. (1999). Early intervention and prevention of anxiety disorders in children: results at 2-year follow-up. Journal of Consulting and Clinical Psychology, 67(1), 145–150.

    Article  PubMed  Google Scholar 

  • De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37(5), 637–642. doi:10.1007/s10802-009-9307-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. (2013). Principles underlying the use of multiple informants’ reports. Annual Review of Clinical Psychology, 9, 123–149. doi:10.1146/annurev-clinpsy-050212-185617.

    Article  PubMed  Google Scholar 

  • De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Deborah, A. G., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin . doi:10.1037/a0038498.Advance online publication.

    PubMed  PubMed Central  Google Scholar 

  • Dever, B. V., Dowdy, E., Raines, T. C., & Carnazzo, K. (2015). Stability and change of behavioral and emotional screening scores. Psychology in the Schools, 52(6), 618–629. doi:10.1002/pits.21825.

    Article  Google Scholar 

  • Dirks, M. A., Boyle, M. H., & Georgiades, K. (2011). Psychological symptoms in youth and later socioeconomic functioning: do associations vary by informant? Journal of Clinical Child and Adolescent Psychology, 40(1), 10–22. doi:10.1080/15374416.2011.533403.

    Article  PubMed  Google Scholar 

  • Dirks, M. A., De Los Reyes, A., Briggs-Gowan, M., Cella, D., & Wakschlag, L. S. (2012). Annual research review: embracing not erasing contextual variability in children’s behavior - Theory and utility in the selection and use of methods and informants in developmental psychopathology. Journal of Child Psychology and Psychiatry, 53(5), 558–574. doi:10.1111/j.1469-7610.2012.02537.x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dodge, K. A., & Pettit, G. S. (2003). A biopsychosocial model of the development of chronic conduct problems in adolescence. Developmental Psychology, 39(2), 349–371. doi:10.1037/0012-1649.39.2.349.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dodge, K. A., Bierman, K. L., Coie, J. D., Greenberg, M. T., Lochman, J. E., McMahon, R. J., & Pinderhughes, E. E. (2015). Impact of early intervention on psychopathology, crime, and well-being at age 25. American Journal of Psychiatry, 172(1), 59–70. doi:10.1176/appi.ajp.2014.13060786.

    Article  PubMed  Google Scholar 

  • Dowdy, E., & Kim, E. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.

    Google Scholar 

  • Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010a). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll & W. Pfohl (Eds.), Handbook of youth prevention science (pp. 70–95). New York: Routledge.

    Google Scholar 

  • Dowdy, E., Ritchey, K., & Kamphaus, R. W. (2010b). School-based screening: a population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. doi:10.1007/s12310-010-9036-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Drabick, D. A., Gadow, K. D., & Sprafkin, J. (2006). Co-occurrence of conduct disorder and depression in a clinic-based sample of boys with ADHD. Journal of Child Psychology and Psychiatry, 47(8), 766–774. doi:10.1111/j.1469-7610.2006.01625.x.

    Article  PubMed  Google Scholar 

  • Durlak, J. A., & Wells, A. M. (1998). Evaluation of indicated preventive intervention (secondary prevention) mental health programs for children and adolescents. American Journal of Community Psychology, 26(5), 775–802. doi:10.1023/A:1022162015815.

    Article  PubMed  Google Scholar 

  • Edelbrock, C., Costello, A. J., Dulcan, M. K., Kalas, R., & Conover, N. C. (1985). Age differences in the reliability of the psychiatric interview of the child. Child Development, 56(1), 265–275.

    Article  PubMed  Google Scholar 

  • Epkins, C. C. (1993). A preliminary comparison of teacher ratings and child self-report of depression, anxiety, and aggression in inpatient and elementary school samples. Journal of Abnormal Child Psychology, 21(6), 649–661. doi:10.1007/BF00916448.

    Article  PubMed  Google Scholar 

  • Feeney-Kettler, K. A., Kratochwill, T. R., Kaiser, A. P., Hemmeter, M. L., & Kettler, R. J. (2010). Screening young children’s risk for mental health problems: a review of four measures. Assessment for Effective Intervention, 35(4), 218–230. doi:10.1177/1534508410380557.

    Article  Google Scholar 

  • Gadow, K. D., Kaat, A. J., & Lecavalier, L. (2013). Relation of symptom-induced impairment with other illness parameters in clinic-referred youth. Journal of Child Psychology and Psychiatry, 54(11), 1198–1207. doi:10.1111/jcpp.12077.

    Article  PubMed  Google Scholar 

  • Gardner, W., Mulvey, E. P., & Shaw, E. C. (1995). Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychological Bulletin, 118(3), 392–404. doi:10.1037/0033-2909.118.3.392.

    Article  PubMed  Google Scholar 

  • Girio-Herrera, E., Dvorsky, M. R., & Owens, J. S. (2015). Mental health screening in kindergarten youth: a multistudy examination of the concurrent and diagnostic validity of the Impairment Rating Scale. Psychological Assessment, 27(1), 215–227. doi:10.1037/a0037787.

    Article  PubMed  Google Scholar 

  • Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. doi:10.1016/j.jsp.2006.05.005.

    Article  Google Scholar 

  • Goodman, R. (1997). The Strengths and Difficulties Questionnaire: a research note. Journal of Child Psychology and Psychiatry, 38(11), 581–586. doi:10.1097/00004583-200111000-00015.

    Article  PubMed  Google Scholar 

  • Goodman, R. (1999). The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40(5), 791–799. doi:10.1111/1469-7610.00494.

    Article  PubMed  Google Scholar 

  • Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40(511), 1337–1345.

    Article  PubMed  Google Scholar 

  • Goodman, R., Ford, T., Simmons, H., Gatward, R., & Meltzer, H. (2000). Using the Strengths and Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. British Journal of Psychiatry, 177, 534–539. doi:10.1192/bjp.177.6.534.

    Article  PubMed  Google Scholar 

  • Gordon, M., Antshel, K., Faraone, S., Barkley, R., Lewandowski, L., Hudziak, J. J., & Cunningham, C. (2006). Symptoms versus impairment: the case for respecting DSM-IV’s Criterion D. Journal of Attention Disorders, 9(3), 465–475. doi:10.1521/adhd.2005.13.4.1.

    Article  PubMed  Google Scholar 

  • Hay, D. F., Pawlby, S., Sharp, D., Schmücker, G., Mills, A., Allen, H., & Kumar, R. (1999). Parents’ judgements about young children's problems: why mothers and fathers might disagree yet still predict later outcomes. Journal of Child Psychology and Psychiatry, 40(8), 1249–1258. doi:10.1111/1469-7610.00541.

    Article  PubMed  Google Scholar 

  • Herrman, H. (2014). Early intervention as a priority for world psychiatry. Early Intervention in Psychiatry, 8(4), 305–306. doi:10.1111/eip.12198.

    Article  PubMed  Google Scholar 

  • Hill, L. G., Coie, J. D., Lochman, J. E., & Greenberg, M. T. (2004). Effectiveness of early screening for externalizing problems: issues of screening accuracy and utility. Journal of Consulting and Clinical Psychology, 72(5), 809–820. doi:10.1037/0022-006X.72.5.809.

    Article  PubMed  PubMed Central  Google Scholar 

  • Huculak, S., & McLennan, J. D. (2014). Using teacher ratings to assess the association between mental health symptoms and impairment in children. School Mental Health, 6, 15–26. doi:10.1007/s12310-013-9107-3.

    Article  Google Scholar 

  • Hunsley, J. (2003). Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment, 15(4), 443–445. doi:10.1037/1040-3590.15.4.443.

    Article  PubMed  Google Scholar 

  • Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29–51. doi:10.1146/annurev.clinpsy.3.022806.091419.

    Article  PubMed  Google Scholar 

  • Jenkins, L. N., Demaray, M. K., Wren, N. S., Secord, S. M., Lyell, K. M., Magers, A. M., & Tennant, J. (2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18(4), 241–254. doi:10.1007/s40688-014-0026-6.

    Article  Google Scholar 

  • Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: the validity of multi-informant reports. Developmental Medicine and Child Neurology, 56(5), 453–459. doi:10.1111/dmcn.12360.

    Article  PubMed  Google Scholar 

  • Johnston, C., & Murray, C. (2003). Incremental validity in the psychological assessment of children and adolescents. Psychological Assessment, 15(4), 496–507. doi:10.1037/1040-3590.15.4.496.

    Article  PubMed  Google Scholar 

  • Jones, D., Dodge, K. A., Foster, E. M., & Nix, R. (2002). Early identification of children at risk for costly mental health service use. Prevention Science, 3(4), 247–256.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kremelberg, D. (2011). Logistic, ordered, multinomial, negative binomial, and poisson regression. In Practical statistics: a quick and easy guide to IBM® SPSS® statistics, STATA, and other statistical software (pp. 235–287). Thousand Oaks, CA: SAGE Publications. doi:10.4135/9781483385655.n6.

    Google Scholar 

  • Lane, K. L., Kalberg, J. R., Menzies, H., Bruhn, A., Eisner, S., & Crnobori, M. (2011). Using systematic screening data to assess risk and identify students for targeted supports: illustrations across the K-12 continuum. Remedial and Special Education, 32(1), 39–54. doi:10.1177/0741932510361263.

    Article  Google Scholar 

  • Lane, K. L., Menzies, H. M., Oakes, W. P., & Kalberg, J. R. (2012). Systematic screenings of behavior to support instruction. New York: Guilford.

    Google Scholar 

  • Levitt, J. M., Saka, N., Romanelli, L. H., & Hoagwood, K. (2007). Early identification of mental health problems in schools: the status of instrumentation. Journal of School Psychology, 45(2), 163–191. doi:10.1016/j.jsp.2006.11.005.

    Article  Google Scholar 

  • Lochman, J. E., & The Conduct Problems Prevention Research Group (1995). Screening of child behavior problems for prevention programs at school entry. Journal of Consulting and Clinical Psychology, 63(4), 549–559. doi:10.1037//0022-006X.63.4.549.

    Article  PubMed  Google Scholar 

  • MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037//1082-989X.7.1.19.

    Article  PubMed  Google Scholar 

  • Mental Health Commission of Canada. (2012). Changing directions changing lives: the mental health strategy for Canada. Retrieved from http://www.strategy.mentalhealthcommission.ca.

  • Normand, S., Flora, D. B., Toplak, M. E., & Tannock, R. (2012). Evidence for a general ADHD factor from a longitudinal general school population study. Journal of Abnormal Child Psychology, 40(4), 555–567. doi:10.1007/s10802-011-9584-5.

    Article  PubMed  Google Scholar 

  • Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Sanford, M., & Lipman, E. L. (1996). Integrating assessment data from multiple informants. Journal of the American Academy of Child and Adolescent Psychiatry, 35(8), 1078–1085. doi:10.1097/00004583-199608000-00019.

    Article  PubMed  Google Scholar 

  • Pickles, A., Rowe, R., Simonoff, E., Foley, D., Rutter, M., & Silberg, J. (2001). Child psychiatric symptoms and psychosocial impairment: relationship and prognostic significance. British Journal of Psychiatry, 179, 230–235. doi:10.1192/bjp.179.3.230.

    Article  PubMed  Google Scholar 

  • Power, T. J., Andrews, T. J., Eiraldi, R. B., Doherty, B. J., Ikeda, M. J., DuPaul, G. J., & Landau, S. (1998). Evaluating attention deficit hyperactivity disorder using multiple informants: the incremental utility of combining teacher with parent reports. Psychological Assessment, 10(3), 250–260. doi:10.1037/1040-3590.10.3.250.

    Article  Google Scholar 

  • Rapee, R. M., Bőgels, S. M., van der Sluis, C. M., Craske, M. G., & Ollendick, T. (2012). Annual research review: conceptualising functional impairment in children and adolescents. Journal of Child Psychology and Psychiatry, 53(5), 454–468. doi:10.1111/j.1469-7610.2011.02479.x.

    Article  PubMed  Google Scholar 

  • Rijlaarsdam, J., Stevens, G. W., van der Ende, J., Hofman, A., Jaddoe, V. W., Verhulst, F. C., & Tiemeier, H. (2015). Prevalence of DSM-IV disorders in a population-based sample of 5- to 8-year-old children: the impact of impairment criteria. European Child & Adolescent Psychiatry. doi:10.1007/s00787-015-0684-6.

    Google Scholar 

  • Sellers, R., Maughan, B., Pickles, A., Thapar, A., & Collishaw, S. (2015). Trends in parent- and teacher-rated emotional, conduct and ADHD problems and their impact in prepubertal children in Great Britain: 1999–2008. Journal of Child Psychology and Psychiatry, 56(1), 49–57. doi:10.1111/jcpp.12273.

    Article  PubMed  Google Scholar 

  • Sharp, C., Croudace, T. J., & Goodyer, I. M. (2005). The Strengths and Difficulties Questionnaire: predictive validity of parent and teacher ratings for help-seeking behaviour over one year. Educational and Child Psychology, 22(3), 28–44.

    Google Scholar 

  • Shemmassian, S. K., & Lee, S. S. (2012). Comparing four methods of integrating parent and teacher symptom ratings of attention-deficit/hyperactivity disorder (ADHD). Journal of Psychopathology and Behavioral Assessment, 34(1), 1–10. doi:10.1007/s10862-011-9262-5.

    Article  Google Scholar 

  • Sowerby, P., & Tripp, G. (2009). Evidence-based assessment of attention-deficit/hyperactivity disorder (ADHD). In J. L. Matson, F. Andrasik, & M. L. Matson (Eds.), Assessing Childhood Psychopathology and Developmental Disabilities (pp. 209–239). New York: Springer. doi:10.1007/978-0-387-09528-8.

    Chapter  Google Scholar 

  • Stone, L. L., Otten, R., Engels, R. C., Vermulst, A., & Janssens, J. M. (2010). Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-Year-olds: a review. Clinical Child and Family Psychology Review, 13, 254–274. doi:10.1007/s10567-010-0071-2.

    Article  PubMed  PubMed Central  Google Scholar 

  • Stringaris, A., & Goodman, R. (2013). The value of measuring impact alongside symptoms in children and adolescents: a longitudinal assessment in a community sample. Journal of Abnormal Child Psychology, 41(7), 1109–1120. doi:10.1007/s10802-013-9744-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • van Dulmen, M. H., & Egeland, B. (2011). Analyzing multiple informant data on child and adolescent behavior problems: predictive validity and comparison of aggregation procedures. International Journal of Behavioral Development, 35, 84–92. doi:10.1177/0165025410392112.

    Article  Google Scholar 

  • Vandeventer, M. C. (2008). Universal emotional and behavioral screening for children and adolescents: are multiple gates and informants worth it? University of Georgia.

  • Verhulst, F. C., Koot, H. M., & van der Ende, J. (1994). Differential predictive value of parents’ and teachers' reports of children's problem behaviors: a longitudinal study. Journal of Abnormal Child Psychology, 22(5), 531–546.

    Article  PubMed  Google Scholar 

  • White, J., Connelly, G., Thompson, L., & Wilson, P. (2013). Assessing wellbeing at school entry using the strengths and difficulties questionnaire: professional perspectives. Educational Research, 55(1), 87–98. doi:10.1080/00131881.2013.767027.

    Article  Google Scholar 

  • Wille, N., Bettge, S., Wittchen, H.-U., & Ravens-Sieberer, U. (2008). How impaired are children and adolescents by mental health problems? Results of the BELLA study. European Child & Adolescent Psychiatry, 17 Suppl 1(S1), 42–51. doi:10.1007/s00787-008-1005-0

  • Woerner, W., Fleitlich-Bilyk, B., Martinussen, R., Fletcher, J., Cucchiaro, G., Dalgalarrondo, P., & Tannock, R. (2004). The strengths and difficulties questionnaire overseas: evaluations and applications of the SDQ beyond Europe. European Child & Adolescent Psychiatry . doi:10.1007/s00787-004-2008-0.Supplement, 13(2004)

    Google Scholar 

  • World Health Organization. (2013). Mental health action plan 2013–2020. Retrieved from http://apps.who.int/iris/bitstream/10665/89966/1/9789241506021_eng.pdf.

Download references

Acknowledgments

This research was financially supported by a grant from the Social Sciences and Humanities Research Council of Canada (SSHRC #410-2008-1052) and the Canada Research Chair Program (RT), and by a Social Sciences and Humanities Research Council of Canada Joseph-Armand Bombardier Canada Graduate Doctoral Scholarship and an Ontario Graduate Scholarship (MA).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Madison Aitken.

Ethics declarations

Conflict of Interest

Dr. Tannock was a member of the DSM-5 Work Group for Attention-Deficit/Hyperactivity Disorder (ADHD), developed the Teacher Telephone Interview for ADHD, and has received honoraria and travel costs from Shire, Janssen, Opopharma Vertiebs VG for unrestricted scientific talks. Ms. Aitken and Dr. Martinussen have no potential conflicts of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aitken, M., Martinussen, R. & Tannock, R. Incremental Validity of Teacher and Parent Symptom and Impairment Ratings when Screening for Mental Health Difficulties. J Abnorm Child Psychol 45, 827–837 (2017). https://doi.org/10.1007/s10802-016-0188-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10802-016-0188-y

Keywords

Navigation