Skip to main content
SpringerLink
Log in
Book cover

Evidence-Based Practices and Treatments for Children with Autism pp 41–51Cite as

On the Reliability and Accuracy of the Evaluative Method for Identifying Evidence-Based Practices in Autism

  • Chapter
  • First Online:

Abstract

The editors of this book recently described the development and application of an “evaluative method” for assessing evidence-based practices (EBP) in Autism (Reichow et al. 2008). The major results of this investigation, which were presented at the International Meeting for Autism Research (Reichow et al. 2007) indicated that the method produced highly reliable and valid results, whether deriving from an assessment of primary or secondary quality indicators from published peer-reviewed, group research reports or from published and peer-reviewed, single subject experimental design (SSED) reports. The levels of inter-examiner agreement, ranged between 85%, with a Kappa or chance-corrected level of 0.69 (Cohen 1960), and 96%, with a Kappa value of 0.93. By applying the criteria of Cicchetti (2001) and Cicchetti et al. (1995), the levels of reliability ranged between good (85%, with a Kappa value of 0.69) and excellent (96%, with a Kappa value equal to 0.93).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Abbreviations

EBP:

Evidence-based practice

K:

Kappa

PC:

Proportion of chance agreement

PNA:

Predicted negative accuracy

PO:

Proportion of observed agreement

POneg:

Proportion of observed negative agreement

POpos:

Proportion of observed positive agreement

PPA:

Predicted positive accuracy

QI−:

Quality indicator absent

QI+:

Quality indicator present

Se:

Sensitivity

Sp:

Specificity

SSED:

Single subject experimental design

References

  • Cicchetti, D. V. (1988). When diagnostic agreement is high, but reliability is low: Some paradoxes occurring in joint independent neuropsychology assessments. Journal of Clinical and Experimental Neuropsychology, 10, 605–622.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  • Cicchetti, D. V. (2001). The precision of reliabi lity and validity estimates re-visited: Distinguishing between clinical and statistical significance of sample size requirements. Journal of Clinical and Experimental Neuropsychology, 23, 695–700.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551–568.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V., & Sparrow, S. S. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–137.

    PubMed  Google Scholar 

  • Cicchetti, D. V., Volkmar, F., Klin, A., & Showalter, D. (1995). Diagnosing autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychology, 1, 26–37.

    Article  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 23, 37–46.

    Article  Google Scholar 

  • Doehring, P., Reichow, B., & Volkmar, F. R. (2007). Is it evidenced-based? How to evaluate claims of effectiveness for autism. Paper presented at the International Association for Positive Behavior Support Conference, March, Boston, MA.

    Google Scholar 

  • Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low Kappa: I the problem of two paradoxes. Journal of Clinical Epidemiology, 43, 543–549.

    Article  PubMed  Google Scholar 

  • Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5). 323–327.

    Google Scholar 

  • Fleiss, J. L. (1975). Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31, 651–659.

    Article  PubMed  Google Scholar 

  • Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley.

    Google Scholar 

  • Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions (3rd ed.). New York: Wiley.

    Book  Google Scholar 

  • Klin, A., Lang, J., Cicchetti, D. V., & Volkmar, F. (2000). Inter-rater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: Results of the DSM-IV autism field trial. Journal of Autism and Developmental Disorders, 30, 163–167.

    Article  PubMed  Google Scholar 

  • Kraemer, H. C. (1982). Estimating false alarms and missed events from interobserver agreement: Comment on Kaye. Psychological Bulletin, 92, 749–754.

    Article  Google Scholar 

  • Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-based interventions in school psychology: Conceptual foundations of the procedural and coding manual of Division 16 and the Society for the Study of School Psychology task force. School Psychology Quarterly, 17, 341–389.

    Article  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measure of observer agreement for categorical data. Biometrics, 3, 159–174.

    Article  Google Scholar 

  • Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998). Empirically supported psychosocial interventions for children: An overview. Journal of Clinical Child Psychology, 27, 138–145.

    Article  PubMed  Google Scholar 

  • Lord, C., Bristol-Power, M., Filipek, P. A., Gallagher, J. J., Harris, S. L., et al. (2001). Educating children with autism. Washington, DC: National Academy.

    Google Scholar 

  • Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71, 137–148.

    Google Scholar 

  • Reichow, B., Barton, E. E., Volkmar, F. R., & Cicchetti, D. V. (2007). The status of research on interventions for young children with autism spectrum disorders. Poster presented at the International Meeting for Autism Research, May, Seattle, WA.

    Google Scholar 

  • Reichow, B., Volkmar, F. R., & Cicchetti, D. V. (2008). Development of an evaluative method for determining the strength of research evidence in autism. Journal of Autism and Developmental Disorders, 38, 1311–1319.

    Article  PubMed  Google Scholar 

  • Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3, 32–35.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Yale Child Study Center, Yale School of Medicine, New Haven, CT, USA

    Domenic V. Cicchetti

Authors
  1. Domenic V. Cicchetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domenic V. Cicchetti .

Editor information

Editors and Affiliations

  1. Yale Child Study Center, South Frontage Rd. 230, New Haven, 06520, Connecticut, USA

    Brian Reichow

  2. Center for Autism Research, The Children's Hospital of Philadelphia, Market Street 3535, Philadelphia, 19146, Pennsylvania, USA

    Peter Doehring

  3. Yale Child Study Center, South Frontage Rd. 230, New Haven, 06520, Connecticut, USA

    Domenic V. Cicchetti

  4. Yale Child Study Center, South Frontage Rd. 230, New Haven, 06520, Connecticut, USA

    Fred R. Volkmar

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Cicchetti, D.V. (2011). On the Reliability and Accuracy of the Evaluative Method for Identifying Evidence-Based Practices in Autism. In: Reichow, B., Doehring, P., Cicchetti, D., Volkmar, F. (eds) Evidence-Based Practices and Treatments for Children with Autism. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-6975-0_3

Download citation

Publish with us

Policies and ethics

Search

Navigation