Skip to main content
Log in

Identifying Autism with a Brief and Low-Cost Screening Instrument—OERA: Construct Validity, Invariance Testing, and Agreement Between Judges

  • Original Paper
  • Published:
Journal of Autism and Developmental Disorders Aims and scope Submit manuscript

Abstract

Simple and low-cost observational-tools to detect symptoms of Autism Spectrum Disorder (ASD) are still necessary. The OERA is a new assessment tool to screen children eliciting observable behaviors with no substantial knowledge on ASD required. The sample was 99 children aged 3–10: 76 with ASD and 23 without ASD (11/23 had intellectual disability). The 13 remained items exhibited high interrater agreement and high reliability loaded onto a single latent trait. Such model showed excellent fit indices evaluated via confirmatory factor analysis and no item showed differential function in terms of age/sex/IQ. A cutoff of five points or higher resulted in the highest sensitivity (92.75) and specificity (90.91) percentages. OERA is a brief, stable, low-cost standardized observational-screening to identify ASD children.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Abrahams, B. S., & Geschwind, D. H. (2008). Advances in autism genetics: On the threshold of a new neurobiology. Nature Reviews. Genetics, 9(5), 341–355. https://doi.org/10.1038/nrg2346.

    Article  PubMed  PubMed Central  Google Scholar 

  • Agency for Healthcare Research and Quality, A. E. H. C. P. (2014). Therapies for children with autism spectrum disorder: Behavioral interventions update. Rockville, MD.

  • Alckmin-Carvalho, F., Teixeira, M. C. T. V., Brunoni, D., Strauss, V. G., & Paula, C. S. (2014). Identification of early signs of autism according to a structured observational protocol: A follow-up study. Psico, 45(4), 502. https://doi.org/10.15448/1980-8623.2014.4.15873.

    Article  Google Scholar 

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders, 5th edn. Arlington, VA: American Psychiatric Publishing.

    Book  Google Scholar 

  • Ashwood, K. L., Buitelaar, J., Murphy, D., Spooren, W., & Charman, T. (2015). European clinical network: Autism spectrum disorder assessments and patient characterisation. European Child & Adolescent Psychiatry, 24(8), 985–995. https://doi.org/10.1007/s00787-014-0648-2.

    Article  Google Scholar 

  • Bagaiolo, L., Mari, JdeJ., Bordini, D., Ribeiro, T. C., Martone, M. C. C., Brunoni, D., … Paula, C. S. (2017). Procedures and compliance of a videomodeling ABA intervention for Brazilian parents of children with Autism Spectrum Disorders. Autism: The International Journal of Research and Practice, 21(5), 603–610.

    Article  Google Scholar 

  • Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd edn.). CRC Press: Boca Raton, FL.

    Google Scholar 

  • Becker, M. M., Wagner, M. B., Bosa, C., Schmidt, C., Longo, D., Papaleo, C., & Riesgo, R. S. (2012). Translation and validation of autism diagnostic interview-revised (ADI-R) for autism diagnosis in Brazil. Arquivos de Neuro-Psiquiatria, 70(3), 185–190. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/22392110.

  • Bishop, S. L., Huerta, M., Gotham, K., Havdahl, A. K., Pickles, A., Duncan, A., … Lord, C. (2017). The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children. Autism Research, 10(1), 78–88. https://doi.org/10.1002/aur.1645.

    Article  PubMed  Google Scholar 

  • Bollen, K. (1998). Structural equations with latent variables. New York: Wiley.

    Google Scholar 

  • Bölte, S., Westerwald, E., Holtmann, M., Freitag, C., & Poustka, F. (2011). Autistic traits and autism spectrum disorders: The clinical validity of two measures presuming a continuum of social communication skills. Journal of Autism and Developmental Disorders, 41(1), 66–72. https://doi.org/10.1007/s10803-010-1024-9.

    Article  PubMed  Google Scholar 

  • Brown, T. (2015). Confirmatory factor analysis for applied research. New York: Guilford Publications.

    Google Scholar 

  • Charman, T. G. K., & Gotham, K. (2013). Measurment Issues: Screening and diagnostic instruments for autism spectrum disorders—lessons from research and practicve. Child and Adolescent Mental Health, 18(1), 52–63. https://doi.org/10.1111/j.1475-3588.2012.00664.x.

    Article  PubMed  Google Scholar 

  • Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494. https://doi.org/10.1177/0049124108314720.

    Article  Google Scholar 

  • Cohen, J. (2013). Statistical power analysis for the behavioral sciences. New York: Academic Press.

    Google Scholar 

  • Cruz, M. (2005). WISC III: Escala de Inteligência Wechsler para crianças: Manual. Avaliação Psicológica, 4(2), 199–201.

    Google Scholar 

  • Dawson, G., Rogers, S., Munson, J., Smith, M., Winter, J., Greenson, J., … Varley, J. (2010). Randomized, controlled trial of an intervention for toddlers with autism: The Early Start Denver Model. Pediatrics, 125(1), e17–e23. https://doi.org/10.1542/peds.2009-0958.

    Article  PubMed  Google Scholar 

  • de Bildt, A., Mulder, E. J., Hoekstra, P. J., van Lang, N. D. J., Minderaa, R. B., & Hartman, C. A. (2009). Validity of the Children’s Social Behavior Questionnaire (CSBQ) in children with intellectual disability: Comparing the CSBQ with ADI-R, ADOS, and clinical DSM-IV-TR classification. Journal of Autism and Developmental Disorders, 39(10), 1464–1470. https://doi.org/10.1007/s10803-009-0764-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • de Macedo, E. C., Mecca, T. P., Valentini, F., Laros, J. A., de Lima, R. M. F., & Schwartzman, J. S. (2013). Utilizando o teste não verbal de inteligência SON-R 2 1/2—7 [a] para avaliar crianças com Transtornos do Espectro do Autismo. Revista Educação Especial, 26(47), 603–618. https://doi.org/10.5902/1984686X9779.

    Article  Google Scholar 

  • Du Toit, M. (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Lincolnwood: Scientific Software International.

    Google Scholar 

  • Duda, M., Kosmicki, J. A., & Wall, D. P. (2014). Testing the accuracy of an observation-based classifier for rapid detection of autism risk. Translational Psychiatry, 4(8), e424. https://doi.org/10.1038/tp.2014.65.

    Article  PubMed  PubMed Central  Google Scholar 

  • Falkmer, T., Anderson, K., Falkmer, M., & Horlin, C. (2013). Diagnostic procedures in autism spectrum disorders: A systematic literature review. European Child & Adolescent Psychiatry, 22(6), 329–340. https://doi.org/10.1007/s00787-013-0375-0.

    Article  Google Scholar 

  • Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2189948.

  • Feldman, H. M., Dale, P. S., Campbell, T. F., Colborn, D. K., Kurs-Lasky, M., Rockette, H. E., & Paradise, J. L. (2005). Concurrent and predictive validity of parent reports of child language at ages 2 and 3 years. Child Development, 76(4), 856–868. https://doi.org/10.1111/j.1467-8624.2005.00882.x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Figueiredo, V. (2002). WISC-III: Escala de Inteligência Weschler para crianças—adaptação brasileira da 3° edição. São Paulo: Casa do Psicólogo.

    Google Scholar 

  • Gabrielsen, T. P., Farley, M., Speer, L., Villalobos, M., Baker, C. N., & Miller, J. (2015). Identifying autism in a brief observation. Pediatrics, 135(2), e330–e338. https://doi.org/10.1542/peds.2014-1428.

    Article  PubMed  Google Scholar 

  • Garcia, A. H. C., Viveiros, M. M., Schwartzman, J. S., & Brunoni, D. (2016). Transtornos do Espectro do Autismo: Avaliação e Comorbidades em Alunos de Barueri., São Paulo. Psicologia—Teoria E Prática, 18(1), 166–177. https://doi.org/10.15348/1980-6906/psicologia.v18n1p166-177.

    Article  Google Scholar 

  • Garfin, D. G., McCallon, D., & Cox, R. (1988). Validity and reliability of the Childhood Autism Rating Scale with autistic adolescents. Journal of Autism and Developmental Disorders, 18(3), 367–378. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/3170455.

  • Gollenberg, A. L., Lynch, C. D., Jackson, L. W., McGuinness, B. M., & Msall, M. E. (2010). Concurrent validity of the parent-completed Ages and Stages Questionnaires, 2nd Ed. with the Bayley Scales of Infant Development II in a low-risk sample. Child: Care, Health and Development, 36(4), 485–490. https://doi.org/10.1111/j.1365-2214.2009.01041.x.

    Google Scholar 

  • Grodberg, D., Weinger, P. M., Halpern, D., Parides, M., Kolevzon, A., & Buxbaum, J. D. (2014). The autism mental status exam: Sensitivity and specificity using DSM-5 criteria for autism spectrum disorder in verbally fluent adults. Journal of Autism and Developmental Disorders, 44(3), 609–614. https://doi.org/10.1007/s10803-013-1917-5.

    Article  PubMed  PubMed Central  Google Scholar 

  • Grodberg, D., Weinger, P. M., Kolevzon, A., Soorya, L., & Buxbaum, J. D. (2012). Brief report: The autism mental status examination: Development of a brief autism-focused exam. Journal of Autism and Developmental Disorders, 42(3), 455–459. https://doi.org/10.1007/s10803-011-1255-4.

    Article  PubMed  Google Scholar 

  • Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843. https://doi.org/10.1148/radiology.148.3.6878708.

    Article  PubMed  Google Scholar 

  • Harstad, E. B., Fogler, J., Sideridis, G., Weas, S., Mauras, C., & Barbaresi, W. J. (2015). Comparing diagnostic outcomes of autism spectrum disorder using DSM-IV-TR and DSM-5 criteria. Journal of Autism and Developmental Disorders, 45(5), 1437–1450. https://doi.org/10.1007/s10803-014-2306-4.

    Article  PubMed  Google Scholar 

  • Hu, L., & Bentler, P. (1999). No cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. https://doi.org/10.1080/10705519909540118.

    Article  Google Scholar 

  • Kim, M. A., & Lord, C. (2012). Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. Journal of Child Psychology and Psychiatry, 53(2), 143–151. https://doi.org/10.1111/j.1469-7610.2011.02458.x.

    Article  PubMed  Google Scholar 

  • Krug, D. A., Arick, J., & Almond, P. (1980). Behavior checklist for identifying severely handicapped individuals with high levels of autistic behavior. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 21(3), 221–229. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7430288.

  • Lai, M.-C., Lombardo, M. V., Auyeung, B., Chakrabarti, B., & Baron-Cohen, S. (2015). Sex/gender differences and autism: Setting the scene for future research. Journal of the American Academy of Child & Adolescent Psychiatry, 54(1), 11–24. https://doi.org/10.1016/j.jaac.2014.10.003.

    Article  Google Scholar 

  • Landa, R. J. (2008). Diagnosis of autism spectrum disorders in the first 3 years of life. Nature Clinical Practice Neurology, 4(3), 138–147. https://doi.org/10.1038/ncpneuro0731.

    Article  PubMed  Google Scholar 

  • Laros, J. A., Almeida, G. O. M., Valentini, F., & Lima, R. M. F. (2015). Dimensionalidade e evidências de validade convergente do SON-R 6-40. Temas Em Psicologia, 23(4), 929–945. https://doi.org/10.9788/TP2015.4-10.

    Article  Google Scholar 

  • Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989). Autism diagnostic interview: A standardized investigator-based instrument. Journal of Autism and Developmental Disorders, 19(3), 363–387. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2793783.

  • Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., … Rutter, M. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11055457.

  • Luyster, R., Gotham, K., Guthrie, W., Coffing, M., Petrak, R., Pierce, K., … Lord, C. (2009). The autism diagnostic observation schedule—Toddler module: A new module of a standardized diagnostic measure for autism spectrum disorders. Journal of Autism and Developmental Disorders, 39(9), 1305–1320. https://doi.org/10.1007/s10803-009-0746-z.

    Article  PubMed  PubMed Central  Google Scholar 

  • Marsh, H., Hau, K., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s 1999 findings. Structural Equation Modeling, 11, 320–341.

    Article  Google Scholar 

  • Marteleto, M. R. F., & Pedromônico, M. R. M. (2005). Validity of autism behavior checklist (ABC): Preliminary study. Revista Brasileira de Psiquiatria, 27(4), 295–301. https://doi.org/10.1590/S1516-44462005000400008.

    Article  PubMed  Google Scholar 

  • McEwen, F. S., Stewart, C. S., Colvert, E., Woodhouse, E., Curran, S., Gillan, N., … Bolton, P. (2016). Diagnosing autism spectrum disorder in community settings using the Development and Well-Being Assessment: Validation in a UK population-based twin sample. Journal of Child Psychology and Psychiatry, 57(2), 161–170. https://doi.org/10.1111/jcpp.12447.

    Article  PubMed  Google Scholar 

  • Muthen, B., & Asparouhov, T. (2004). IRT studies of many groups: The alignment method. Frontiers in Psychology, 11, 320–341.

    Google Scholar 

  • Muthén, B., Toit, D. S. H., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Psychometrika, 75, 1–45.

    Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (2002). How to use a monte carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 599–620. https://doi.org/10.1207/S15328007SEM0904_8.

    Article  Google Scholar 

  • Obuchowski, N., McClish, D. (1997). Sample size determination for diagnostic accuracy studies involving binormal roc curve indices. Statistics in Medicine, 16(13), 1529–1542.

    Article  PubMed  Google Scholar 

  • Ozonoff, S., Goodlin-Jones, B. L., & Solomon, M. (2005). Evidence-based assessment of autism spectrum disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 34(3), 523–540. https://doi.org/10.1207/s15374424jccp3403_8.

    Article  Google Scholar 

  • Pereira, A., Riesgo, R. S., & Wagner, M. B. (2008). Childhood autism: Translation and validation of the Childhood Autism Rating Scale for use in Brazil. Jornal de Pediatria, 84(6), 487–494. https://doi.org/10.2223/JPED.1828.

    Article  PubMed  Google Scholar 

  • Raykov, T. (1997). Scale reliability, Cronbach’s coefficient alpha, and violations of essential tau-equivalence for fixed congeneric components. Multivariate Behavioral Research, 32, 329–354.

    Article  PubMed  Google Scholar 

  • Raykov, T. (2001). Bias of coefficient afor fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25(1), 69–76. https://doi.org/10.1177/01466216010251005.

    Article  Google Scholar 

  • Samms-Vaughan, M. E. (2014). The status of early identification and early intervention in autism spectrum disorders in lower- and middle-income countries. International Journal of Speech-Language Pathology, 16(1), 30–35. https://doi.org/10.3109/17549507.2013.866271.

    Article  PubMed  Google Scholar 

  • Schopler, E., Reichler, R. J., DeVellis, R. F., & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10(1), 91–103. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/6927682.

  • Shattuck, P. T., Durkin, M., Maenner, M., Newschaffer, C., Mandell, D. S., Wiggins, L., … Cuniff, C. (2009). Timing of identification among children with an autism spectrum disorder: Findings from a population-based surveillance study. Journal of the American Academy of Child and Adolescent Psychiatry, 48(5), 474–483. https://doi.org/10.1097/CHI.0b013e31819b3848.

    Article  PubMed  PubMed Central  Google Scholar 

  • Soto, S., Linas, K., Jacobstein, D., Biel, M., Migdal, T., & Anthony, B. J. (2015). A review of cultural adaptations of screening tools for autism spectrum disorders. Autism, 19(6), 646–661. https://doi.org/10.1177/1362361314541012.

    Article  PubMed  Google Scholar 

  • The Psychological Corporation. (1992). Weschler Individual Achievement Test. San Antonio, CA: The Psychological Corporation.

    Google Scholar 

  • Van Bourgondien, M. E., Marcus, L. M., & Schopler, E. (1992). Comparison of DSM-III-R and childhood autism rating scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22(4), 493–506. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1483973.

  • Van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthen, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 23(4), 770. https://doi.org/10.3389/fpsyg.2013.00770.

    Google Scholar 

  • Virués-Ortega, J. (2010). Applied behavior analytic intervention for autism in early childhood: Meta-analysis, meta-regression and dose-response meta-analysis of multiple outcomes. Clinical Psychology Review, 30(4), 387–399. https://doi.org/10.1016/j.cpr.2010.01.008.

    Article  PubMed  Google Scholar 

  • Wall, D. P., Kosmicki, J., DeLuca, T. F., Harstad, E., & Fusaro, V. A. (2012). Use of machine learning to shorten observation-based screening and diagnosis of autism. Translational Psychiatry, 2(4), e100. https://doi.org/10.1038/tp.2012.10.

    Article  PubMed  PubMed Central  Google Scholar 

  • Yu, C.-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, Los Angeles.

  • Zwaigenbaum, L., Bauman, M. L., Choueiri, R., Fein, D., Kasari, C., Pierce, K., … Wetherby, A. (2015a). Early identification and interventions for autism spectrum disorder: Executive summary. Pediatrics, 136(Supplement), S1–S9. https://doi.org/10.1542/peds.2014-3667B.

    Article  PubMed  Google Scholar 

  • Zwaigenbaum, L., Bauman, M. L., Choueiri, R., Kasari, C., Carter, A., Granpeesheh, D., … Natowicz, M. R. (2015b). Early intervention for children with autism spectrum disorder under 3 years of age: Recommendations for practice and research. Pediatrics, 136(Supplement), S60–S81. https://doi.org/10.1542/peds.2014-3667E.

    Article  PubMed  Google Scholar 

  • Zwaigenbaum, L., Bryson, S., Lord, C., Rogers, S., Carter, A., Carver, L., … Yirmiya, N. (2009). Clinical assessment and management of toddlers with suspected autism spectrum disorder: Insights from studies of high-risk infants. Pediatrics, 123(5), 1383–1391. https://doi.org/10.1542/peds.2008-1606.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

This study was funded by the State of São Paulo Funding Agency-FAPESP under a special agreement with Maria Cecília Souto Vidigal Foundation [Grant No. 2012/51584-0], and by the National Research Council—CNPQ [Grant No. 401468/2010-0]. CSP and JJM are researchers from the National Research Council (CNPQ).

Author information

Authors and Affiliations

Authors

Contributions

CSP conceived of the study, participated in its design, coordination, data collection, interpretation of data, and drafted the manuscript; GRC was one of the expert clinicians responsible for the ASD diagnosis, helped in the statistical analysis, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis; participated in the study design, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis, and helped to coordinate the data collection; ACM helped to coordinate the data collection, and participated in the interpretation of data; CAB was responsible for the ADI-R evaluations, and helped to draft the manuscript; JJM participated in the coordination of the study, and helped to draft the manuscript; HCM performed statistical analysis, and helped to draft the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Cristiane S. Paula.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of the two institutional research committees and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Any additional data related to the manuscript data will be provided by the first author upon request.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Paula, C.S., Cunha, G.R., Bordini, D. et al. Identifying Autism with a Brief and Low-Cost Screening Instrument—OERA: Construct Validity, Invariance Testing, and Agreement Between Judges. J Autism Dev Disord 48, 1780–1791 (2018). https://doi.org/10.1007/s10803-017-3440-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-017-3440-6

Keywords

Navigation