Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Initial Development of Multi-item Direct Behavior Rating Measures of Academic Enablers

Abstract

Although there is a growing body of evidence to support the use of direct behavior rating (DBR) as a formative behavioral assessment method in school-based problem-solving models, this work has centered largely on the assessment of attending and problem behaviors and on the use of single-item DBR scales. The primary purpose of the current study was to report on the development and initial validation of teacher-completed multi-item DBR scales (DBR-MIS) designed to expand upon prior DBR research to assess three constructs representing behaviors that are widely considered to support student academic achievement (academic engagement, interpersonal skills, and study skills), only one of which (academic engagement) has been the target of DBR assessment methods. Development of the scales involved a content validity study utilizing feedback from a panel of consumers (teachers, parents, and school administrators) and a panel of researchers with expertise in school-based behavioral assessment. Results of an exploratory factor analysis of ratings of students completed by teachers in Grades K through 3 (N = 307) supported a one-factor solution for each of the aforementioned constructs. Favorable internal consistency was found for each scale.

This is a preview of subscription content, log in to check access.

References

  1. Anthony, C. J., & DiPerna, J. C. (2017). Identifying sets of maximally efficient items from the Academic Competence Evaluation Scales—Teacher Form. School Psychology Quarterly,32, 552–559. https://doi.org/10.1037/spq0000205.

  2. Anthony, C. J., & DiPerna, J. C. (2018). Piloting a short form of the academic competence evaluation scales. School Mental Health,10, 1–8. https://doi.org/10.1007/s12310-018-9254-7.

  3. Atkins, M., Hoagwood, K., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health in schools. Administration and Policy in Mental Health,37, 40–47. https://doi.org/10.1007/s10488-010-0299-7.

  4. Barkley, R. A. (Ed.). (2015). Attention-deficit hyperactivity disorder: A handbook for diagnosis and treatment (4th ed.). New York, NY: The Guilford Press.

  5. Briesch, A. M., Chafouleas, S. M., & Riley-Tillman, T. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review,39, 408–421.

  6. Briesch, A. M., Chafouleas, S. M., & Riley-Tillman, T. C. (2016). Direct Behavior Rating (DBR): Linking assessment, communication, and intervention. New York, NY: Guilford Press.

  7. Briesch, A. M., Ferguson, T. D., Volpe, R. J., & Briesch, J. M. (2013). Examining teachers’ perceptions of social-emotional and behavioral referral concerns. Remedial and Special Education,34, 249–256.

  8. Briesch, A. M., & Volpe, R. J. (2007). Important considerations in the selection of progress monitoring measures for classroom behaviors. School Psychology Forum,1, 59–74.

  9. Briesch, A. M., Volpe, R. J., & Ferguson, T. D. (2014). The influence of student characteristics on the dependability of observation data. School Psychology Quarterly, 29, 171–181.

  10. Chafouleas, S. M. (2011). Direct Behavior Rating: A review of the issues and research in its development. Education and Treatment of Children,34(4), 575–591.

  11. Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., Christ, T. C., Black, A. C., & Kilgus, S. P. (2010a). An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology,48, 219–246. https://doi.org/10.1016/j.jsp.2010.02.001.

  12. Chafouleas, S. M., Christ, T. J., Riley-Tillman, T. C., Briesch, A. M., & Chanese, J. A. (2007a). Generalizability and dependability of direct behavior ratings to assess social behavior of preschoolers. School Psychology Review,36, 63–79.

  13. Chafouleas, S. M., Riley-Tillman, T. C., & Christ, T. J. (2009). Direct behavior rating (DBR) an emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention,34, 195–200.

  14. Chafouleas, S., Riley-Tillman, T. C., & Sugai, G. M. (2007b). School-based behavioral assessment: Informing intervention and instruction. New York, NY: Guilford Press.

  15. Chafouleas, S. M., Sanetti, L. M. H., Kilgus, S. P., & Maggin, D. M. (2012). Evaluating sensitivity to behavioral change across consultation cases using Direct Behavior Rating Single-Item Scales (DBR-SIS). Exceptional Children, 78, 491–505.

  16. Chafouleas, S. M., Volpe, R. J., Gresham, F. M., & Cook, C. R. (2010b). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review,39, 343–349.

  17. Christ, T. J., Riley-Tillman, T. C., & Chafouleas, S. M. (2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention,34, 201–213.

  18. Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

  19. DiPerna, J. C., & Elliott, S. N. (1999). The development and validation of the Academic Competence Evaluation Scales. Journal of Psychoeducational Assessment,17, 207–225. https://doi.org/10.1177/073428299901700302.

  20. DiPerna, J. C., & Elliott, S. N. (2000). Academic competence evaluation scales. San Antonio, TX: The Psychological Corporation.

  21. DiPerna, J. C., & Elliott, S. N. (2002). Promoting academic enablers to improve student achievement: An introduction to the miniseries. School Psychology Review,31, 293–297.

  22. DiPerna, J. C., Volpe, R. J., & Elliott, S. N. (2002). A model of academic enablers and elementary reading/language arts achievement. School Psychology Review,31, 298–312.

  23. DiPerna, J. C., Volpe, R. J., & Elliott, S. N. (2005). A model of academic enablers and mathematics achievement in the elementary grades. Journal of School Psychology,43, 379–392. https://doi.org/10.1016/j.jsp.2005.09.002.

  24. Every Student Succeeds Act (ESSA) of 2015, Pub. L. 114-95, § 114.

  25. Fabiano, G. A., Pyle, K., Kelty, M. B., & Parham, B. R. (2017). Progress monitoring using direct behavior rating single item scales in a multiple baseline design study of the daily report card intervention. Assessment for Effective Intervention, 43, 21–33.

  26. Fabiano, G. A., Vujnovic, R., Pelham, W. E., Waschbusch, D. A., Massetti, G. M., Yu, J., et al. (2010). Enhancing the effectiveness of special education programming for children with ADHD using a daily report card. School Psychology Review,39, 219–239.

  27. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods,4, 272–299. https://doi.org/10.1037/1082-989X.4.3.272.

  28. Ferguson, T. D., Briesch, A. M., Volpe, R. J., & Daniels, B. (2012). The influence of observation length on the dependability of data. School Psychology Quarterly,27, 187–197. https://doi.org/10.1037/spq0000005.

  29. Gettinger, M., & Seibert, J. K. (2002). Contributions of study skills to academic competence. School Psychology Review,31, 350–365.

  30. Greenwood, C. R. (1996). The case for performance-based instructional models. School Psychology Quarterly,11, 283–296. https://doi.org/10.1037/h0088935.

  31. Gresham, F. M., Cook, C. R., Collins, T., Dart, E., Rasetshwane, K., Truelson, E., et al. (2010). Developing a change-sensitive brief behavior rating scale as a progress monitoring tool for social behavior: An example using the Social Skills Rating System—Teacher Form. School Psychology Review,39, 364–379.

  32. Harvey, V. S., & Chickie-Wolfe, L. A. (2007). Fostering independent learning: Practical strategies to promote student success. New York, NY: Guilford Press.

  33. Henricsson, L., & Rydell, A. (2006). Children with behaviour problems: The influence of social competence and social relations on problem stability, school achievement and peer acceptance across the first six years of school. Infant and Child Development,15, 347–366. https://doi.org/10.1002/icd.448.

  34. Hintze, J. M., & Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review,33, 258–270.

  35. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika,30, 179–185.

  36. Individuals with Disabilities Education Act (IDEA) of 2004, Pub. L. 108-446. 118 U.S.C. § 2647.

  37. Jenkins, L. N., & Demaray, M. K. (2015). An investigation of relations among academic enablers and reading outcomes. Psychology in the Schools,52, 379–389. https://doi.org/10.1002/pits.21830.

  38. Malecki, C. K., & Elliott, S. N. (2002). Children’s social behaviors as predictors of academic achievement: A longitudinal analysis. School Psychology Quarterly,17, 1–23. https://doi.org/10.1521/scpq.17.1.1.19902.

  39. Massetti, G. M., Lahey, B. B., Pelham, W. E., Loney, J., Ehrhardt, A., Lee, S. S., et al. (2008). Academic achievement over 8 years among children who met modified criteria for attention-deficit/hyperactivity disorder at 4–6 years of age. Journal of Abnormal Child Psychology,36, 399–410. https://doi.org/10.1007/s10802-007-9186-4.

  40. McDonald, R. P. (2013). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.

  41. Meade, A. W., & Bauer, D. J. (2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling,14, 611–635. https://doi.org/10.1080/10705510701575461.

  42. National Center on Intensive Intervention. (2018). Behavioral progress monitoring tools. Retrieved June 26, 2018, from https://charts.intensiveintervention.org/chart/behavioral-progress-monitoring-tools.

  43. National Center on Response to Intervention. (2010). Essential components of RTI-A closer look at response to intervention. Washington, DC: U.S. Department of Education, Office of Special Education Programs.

  44. Osborne, J. W., & Costello, A. B. (2009). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pan-Pacific Management Review,12, 131–146.

  45. Owens, J. S., Holdaway, A. S., Zoromski, A. K., Evans, S. W., Himawan, L. K., Girio-Herrera, E., et al. (2012). Incremental benefits of a daily report card intervention over time for youth with disruptive behavior. Behavior Therapy,43, 848–861. https://doi.org/10.1016/j.beth.2012.02.002.

  46. Pelham, W. E., Jr., Fabiano, G. A., & Massetti, G. M. (2005). Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology,34, 449–476. https://doi.org/10.1207/s15374424jccp3403_5.

  47. Rabiner, D., & Coie, J. D. (2000). Early attention problems and children’s reading achievement: A longitudinal investigation. Journal of the American Academy of Child and Adolescent Psychiatry,39, 859–867. https://doi.org/10.1097/00004583-200007000-00014.

  48. Reyes, M. R., Brackett, M. A., Rivers, S. E., White, M., & Salovey, P. (2012). Classroom emotional climate, student engagement, and academic achievement. Journal of Educational Psychology,104, 700–712. https://doi.org/10.1037/a0027268.

  49. Reynolds, C. R., & Kamphaus, R. W. (2015). BASC-3 flex monitor. Minneapolis, MN: NCS Pearson.

  50. Reynolds, A. J., & Walberg, H. J. (1992). A process model of mathematics achievement and attitude. Journal for Research in Mathematics Education,23, 306–328. https://doi.org/10.2307/749308.

  51. Riley-Tillman, T. C., Chafouleas, S. M., Sassu, K. A., Chanese, J. A. M., & Glazer, A. D. (2008). Examining the agreement of Direct Behavior Ratings and Systematic Direct Observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions,10, 136–143. https://doi.org/10.1177/1098300707312542.

  52. Shapiro, E. S., & Heick, P. F. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools,41, 551–561. https://doi.org/10.1002/pits.10176.

  53. Spectrum K12 School Solutions (2011). Response to intervention adoption survey 2011. Retrieved June 26, 2018, from http://www.globalscholar.com/2011RTI.

  54. Stevens, J. (1992). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Erlbaum.

  55. Sugai, G., & Horner, R. H. (2009). Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality,17, 223–237. https://doi.org/10.1080/09362830903235375.

  56. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). New York: Pearson Education Inc.

  57. Tilly, D. (2008). The evolution of school psychology to science based practice. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology, V (pp. 17–36). Bethesda, MD: National Association of School Psychologists.

  58. U.S. Department of Education. (2018). National Center for Education Statistics, Common Core of Data (CCD), “Public Elementary/Secondary School Universe Survey,” 2017-2018. Washington, DC: National Center for Education Statistics.

  59. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika,41, 321–327.

  60. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment (pp. 41–71). Boston, MA: Springer.

  61. Volpe, R. J., & Briesch, A. M. (2012). Generalizability and dependability of single-item and multiple-item direct behavior rating scales for engagement and disruptive behavior. School Psychology Review,41, 246–261.

  62. Volpe, R. J., & Briesch, A. M. (2015). Multi-item Direct Behavior Ratings: Dependability of two levels of assessment specificity. School Psychology Quarterly,30, 431–442. https://doi.org/10.1037/spq0000115.

  63. Volpe, R. J., DiPerna, J. C., Hintze, J. M., & Shapiro, E. S. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review,34, 454–474.

  64. Volpe, R. J., DuPaul, G. J., DiPerna, J. C., & Jitendra, A. K. (2006). Attention deficit hyperactivity disorder and scholastic achievement: A model of mediation via academic enablers. School Psychology Review,35, 47–61.

  65. Volpe, R. J., & Fabiano, G. A. (2013). Daily behavior report cards: An evidence-based system of assessment and intervention. New York, NY: The Guilford Press.

  66. Volpe, R. J., & Gadow, K. D. (2010). Creating abbreviated rating scales to monitor classroom inattention-overactivity, aggression, and peer conflict: Reliability, validity, and treatment sensitivity. School Psychology Review,39, 350–363.

  67. Volpe, R. J., Gadow, K. D., Blom-Hoffman, J., & Feinberg, A. B. (2009a). Factor-analytic and individualized approaches to constructing brief measures of ADHD behaviors. Journal of Emotional and Behavioral Disorders,17, 118–128. https://doi.org/10.1177/1063426608323370.

  68. Volpe, R. J., McConaughy, S. H., & Hintze, J. M. (2009b). Generalizability of classroom behavior problem and on-task scores from the Direct Observation Form. School Psychology Review,38, 382–401.

  69. Wentzel, K. R. (1993). Does being good make the grade? Social behavior and academic competence in middle school. Journal of Educational Psychology, 85, 357–364.

  70. Wolf, M. M. (1978). Social validity: The case for subjective measurement of how behavior analysis is finding its heart. Journal of Applied Behavior Analysis,11, 203–214. https://doi.org/10.1901/jaba.1978.11-203.

  71. Wood, B. K., Hojnoski, R. L., Laracy, S. D., & Olson, C. L. (2015). Comparison of observational methods and their relation to ratings of engagement in young children. Topics in Early Childhood Special Education,35, 211–222. https://doi.org/10.1177/0271121414565911.

  72. Zirkel, P. A. (2011). State laws and guidelines for RTI: Additional implementation features. Communiqué,39(7), 30–32.

Download references

Funding

Preparation of this article was supported by funding provided by the Institute of Education Sciences, US Department of Education (R324A150071). Opinions expressed herein do not necessarily reflect the position of the US Department of Education, and such endorsements should not be inferred.

Author information

Correspondence to Robert J. Volpe.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical Approval

All procedures performed were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was deemed exempt from obtaining signed written consent from participants in accordance with HHS regulations at 45 CFR 46.117(c).

Human and Animal Rights

This article does not contain any studies with animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Volpe, R.J., Chaffee, R.K., Yeung, T.S. et al. Initial Development of Multi-item Direct Behavior Rating Measures of Academic Enablers. School Mental Health 12, 77–87 (2020). https://doi.org/10.1007/s12310-019-09338-w

Download citation

Keywords

  • Behavioral assessment
  • Direct behavior rating
  • Academic enablers
  • Progress monitoring
  • Factor analysis
  • Scale development
  • Formative assessment
  • Classroom behavior
  • Student behavior