Automated Behavioral Modeling and Pattern Analysis of Children with Autism in a Joint Attention Training Application: A Preliminary Study

  • Tiffany Y. TangEmail author
  • Pinata Winoto
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11590)


Although recent research works have highlighted and demonstrated the applicability of the activity and behavioral pattern analysis mechanisms in offering early windows of opportunities in the assessment and intervention for individuals with autism spectrum disorder (ASD), the computational cost and sophistication of such behavioral modeling systems might prevent these automatic and semi-automatic systems from deploying, which might in turn restrict its actual use. As such, in this paper, we proposed an easily deployable automatic system to train joint attention (JA) skills, characterizing and evaluating JA and reciprocity patterns (i.e. the frequency and degree of reciprocity, initiating and responding to JA bids). Our proposed approach is different from most of earlier attempts in that we do not capitalize the sophisticated feature-space construction methodology; instead, the simple designs and in-game automatic data collection offers hassle-free benefits for such individuals as special education teachers and parents to use in both classrooms and at homes.


Joint attention skills Children Pattern Autism Behavioral modeling Puzzle Training application 



The authors gratefully acknowledge financial support from Zhejiang Provincial Natural Science Foundation of China (LGJ19F020001). Our thanks to Aonan Guan for implementing the system; Haoyu Yu for her design of the pictures used in the application. Thanks also go to Jie Chen, for participating in the preliminary test.


  1. 1.
    Winoto, P., Tang, T.Y.: A multi-user tabletop application to train children with autism social attention coordination skills without forcing eye-gaze following. In: Proceedings of the 16th ACM Interaction Design and Children Conference (ACM IDC 2017), pp. 527–532. ACM Press (2017)Google Scholar
  2. 2.
    Brooks, R., Meltzoff, A.N.: The development of gaze following and its relation to language. Dev. Sci. 8, 535–543 (2005)CrossRefGoogle Scholar
  3. 3.
    Mundy, P., Block, J., Delgado, C., Pomares, Y., Van Hecke, A.V., Parlade, M.: Individual differences and the development of joint attention in infancy. Child Dev. 78, 938–954 (2007)CrossRefGoogle Scholar
  4. 4.
    Kasari, C., Paparella, T., Freeman, S., Jahromi, L.B.: Language outcome in autism: randomized comparison of joint attention and play interventions. J. Consult. Clin. Psychol. 76(1), 125–137 (2008)CrossRefGoogle Scholar
  5. 5.
    Nelson, P.B., Adamson, L.B., Bakeman, R.: Toddlers’ joint engagement experience facilitates preschoolers’ acquisition of theory of mind. Dev. Sci. 11, 847–852 (2008)CrossRefGoogle Scholar
  6. 6.
    Van Hecke, A.V., et al.: Infant joint attention, temperament, and social competence in preschool children. Child Dev. 78, 53–69 (2007)CrossRefGoogle Scholar
  7. 7.
    Raver, C.: Relations between social contingency in mother–child interaction and 2-year-olds’ social competence. Dev. Psychol. 32, 850–859 (1996)CrossRefGoogle Scholar
  8. 8.
    Bruinsma, Y., Koegel, R.L., Koegel, L.K.: Joint attention and children with autism: a review of the literature. Ment. Retard. Dev. Disabil. Res. Rev. 10, 169–175 (2004)CrossRefGoogle Scholar
  9. 9.
    Carpenter, M., Pennington, B.F., Rogers, S.J.: Interrelations among social-cognitive skills in young children with autism. J. Autism Dev. Disord. 32, 91–106 (2002)CrossRefGoogle Scholar
  10. 10.
    Stone, W., Ousley, O.Y., Yoder, P.J., Hogan, K.L., Hepburn, S.L.: Nonverbal communication in two- and three-year-old children with autism. J. Autism Dev. Disord. 6, 677–695 (1997)CrossRefGoogle Scholar
  11. 11.
    White, P.J., et al.: Best practices for teaching joint attention: a systematic review of the intervention literature. Res. Autism Spectr. Disord. 5(4), 1283–1295 (2011)CrossRefGoogle Scholar
  12. 12.
    Giusti, L., Zancanaro, M., Gal, E., Weiss, P.L.T.: Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pp. 3295–3304 (2011)Google Scholar
  13. 13.
    Gal, E., Lamash, L., Bauminger-Zviely, N., Zancanaro, M., Weiss, P.L.T.: Using multitouch collaboration technology to enhance social interaction of children with high-functioning autism. Phys. Occup. Ther. Pediatr. 36(1), 46–58 (2016)CrossRefGoogle Scholar
  14. 14.
    Winoto, P., Tang, T.Y., Guan, A.: “I will help you pass the puzzle piece to your partner if this is what you want me to”: the design of collaborative puzzle games to train Chinese children with autism spectrum disorder joint attention skills. In: Proceedings of the 15th ACM Interaction Design and Children Conference (ACM IDC 2016), pp. 601–606. ACM Press (2016)Google Scholar
  15. 15.
    Whalen, C., Schreibman, L.: Joint attention training for children with autism using behavior modification procedures. Phys. Occup. Ther. Pediatr. 44(3), 456–468 (2003)Google Scholar
  16. 16.
    Green, J., Charman, T., McConachie, H., PACT Consortium, et al.: Parent-mediated communication-focused treatment in children with autism (PACT): a randomized controlled trial. Lancet, 375, 2152–2160 (2010)Google Scholar
  17. 17.
    Kasari, C., et al.: Randomized controlled trial of parental responsiveness intervention for toddlers at high risk for autism. Infant Behav. Dev. 37(4), 711–721 (2014)CrossRefGoogle Scholar
  18. 18.
    Schertz, H.H., Odom, S.L., Baggett, K.M., Sideris, J.H.: Effects of joint attention mediated learning for toddlers with autism spectrum disorders: an initial randomized controlled study. Early Child. Res. Q. 28(2), 249–258 (2013)CrossRefGoogle Scholar
  19. 19.
    Battocchi, A., et al.: Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS 2009), pp. 197–204 (2009)Google Scholar
  20. 20.
    Goh, W.B., Shou, W., Tan, J., Lum, G.T.: Interaction design patterns for multi-touch tabletop collaborative games. In: Proceedings of CHI 2012 Extended Abstracts on Human Factors in Computing Systems (CHI 2012), pp. 141–150 (2012)Google Scholar
  21. 21.
    Piper, A.M., O’Brien, E., Morris, M.R., Winograd, T.: SIDES: a cooperative tabletop computer game for social skills development. In: Proceedings of the 20th Conference on Computer Supported Cooperative Work (ACM CSCW 2006), pp. 1–10 (2006)Google Scholar
  22. 22.
    Silva, G.F.M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. J. Comput. Support. Coop. Work 24, 149–175 (2015)CrossRefGoogle Scholar
  23. 23.
    Ke, Y., Sukthankar, R., Hebert, M.: Volumetric features for video event detection. Int. J. Comput. Vis. 88(3), 339–362 (2010)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2008), pp. 1–8 (2008)Google Scholar
  25. 25.
    Messing, R., Pal, C., Kautz, H.: Activity recognition using the velocity histories of tracked keypoints. In: Proceedings of IEEE 12th International Conference on Computer Vision (ICCV 2009), pp. 104–111 (2009)Google Scholar
  26. 26.
    Tran, D., Sorokin, A.: Human activity recognition with metric learning. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 548–561. Springer, Heidelberg (2008). Scholar
  27. 27.
    Choi, W., Shahid, K., Savarese, S.: Learning context for collective activity recognition. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), pp. 3273–3280 (2012)Google Scholar
  28. 28.
    Chong, E., et al.: Detecting gaze towards eyes in natural social interactions and its use in child assessment. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 3, Article No. 43 (2017)Google Scholar
  29. 29.
    Anzulewicz, A., Sobota, K., Delafield-Butt, J.T.: Toward the autism motor signature: gesture patterns during smart tablet gameplay identify children with autism. Sci. Rep. 6 (2016). Article number 31107Google Scholar
  30. 30.
    Lan, T., Wang, Y., Yang, W., Mori, G.: Beyond actions: discriminative models for contextual group activities. In: Proceedings of NIPS, pp. 1216–1224 (2010)Google Scholar
  31. 31.
    Prabhakar, K., Rehg, James M.: Categorizing turn-taking interactions. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 383–396. Springer, Heidelberg (2012). Scholar
  32. 32.
    Rehg, J.M., et al.: Decoding children’s social behavior. In: Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), pp. 3414–3421 (2013)Google Scholar
  33. 33.
    Rehg, J.M., Rozga, A., Abowd, G.D., Goodwin, M.S.: Behavioral imaging and autism. IEEE Pervasive Comput. 13(2), 84–87 (2014)CrossRefGoogle Scholar
  34. 34.
    Winoto, P., Chen, C.G., Tang, T.Y.: The development of a kinect-based online socio-meter for users with social and communication skill impairments: a computational sensing approach. In: Proceedings of IEEE International Conference on Knowledge Engineering and Applications (ICKEA 2016), pp. 139–143 (2016)Google Scholar
  35. 35.
    Kong, H.K., Lee, J., Ding, J., Karahalios, K.: EnGaze: designing behavior visualizations with and for behavioral scientists. In Proceedings of 2016 ACM Conference on Designing Interactive Systems (ACM DIS 2016), pp. 1185–1196 (2016)Google Scholar
  36. 36.
    Higuchi, K., et al.: Visualizing gaze direction to support video coding of social attention for children with autism spectrum disorder. In: Proceedings of 23rd International Conference on Intelligent User Interfaces (ACM IUI 2018), pp. 571–582 (2018)Google Scholar
  37. 37.
    Kientz, J.A., Boring, S., Abowd, G.D., Hayes, G.R.: Abaris: evaluating automated capture applied to structured autism interventions. In: Beigl, M., Intille, S., Rekimoto, J., Tokuda, H. (eds.) UbiComp 2005. LNCS, vol. 3660, pp. 323–339. Springer, Heidelberg (2005). Scholar
  38. 38.
    Zwaigenbaum, L., et al.: Early identification and interventions for autism spectrum disorder: executive summary. Pediatrics 136(Suppl), S1–S9 (2015)CrossRefGoogle Scholar
  39. 39.
    Cason, J., Richmond, T.: Telehealth opportunities in occupational therapy. In: Kumar, S., Cohn, E. (eds.) Telerehabilitation. Health Informatics, pp. 139–162. Springer, London (2013). Scholar
  40. 40.
    Peretti, A., Amenta, F., Tayebati, S.K., Nittari, G., Mahdi, S.S.: Telerehabilitation: review of the state-of-the-art and areas of application. JMIR Rehabil. Assist. Technol. 4(2), e7 (2017)CrossRefGoogle Scholar
  41. 41.
    Redcay, E., Kleiner, M., Saxe, R.: Look at this: the neural correlates of initiating and responding to bids for joint attention. Front. Hum. Neurosci. 6, 169 (2012)CrossRefGoogle Scholar
  42. 42.
    Tang, T.Y., Winoto, P.: Providing adaptive and personalized visual support based on behavioural tracking of children with autism for assessing reciprocity and coordination skills in a joint attention training application. In: Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion (ACM IUI 2018), Article No. 40. ACM Press (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Media Lab, Department of Computer ScienceWenzhou-Kean UniversityWenzhouChina

Personalised recommendations