Psychophysiological Feedback for Adaptive Human–Robot Interaction (HRI)

Part of the Human–Computer Interaction Series book series (HCIS)


Recent advances in robotics and sensing have given rise to a diverse set of robots and their applications. In recent years robots have increasingly applied in the service industry, search and rescue operations and therapeutic applications. The introduction of robots to interact with humans resulted in a dedicated field called human–robot interaction (HRI). Social HRI is of particular importance as it is the main focus of this chapter. This chapter presents an affect-inspired approach for social HRI. Physiological processing together with machine learning was employed to model affective states for an adaptive social HRI and its application in social interaction in the context of autism therapy was investigated.


Autism Spectrum Disorder Affective State Emotion Recognition Galvanic Skin Response Pulse Transit Time 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported in part by a Marino Autism Research Institute (MARI) grant, an Autism Speaks Foundation Pilot grant, the National Science Foundation Grant [award number 0967170], and the National Institute of Health Grant [award number 1R01MH091102-01A1]. We would like to thank all colleagues that helped in this research and give special thanks to all subjects and their families.


  1. Agrawal P, Liu C, Sarkar N (2008) Interaction between human and robot an affect-inspired approach. Interact Stud 9(2):230–257CrossRefGoogle Scholar
  2. Bethel CL, Burke JL, Murphy RR, Salomon K (2007) Psychophysiological experimental design for use in human-robot interaction studies. In: International Symposium on Collaborative Technologies and Systems, 2007. CTS 2007. IEEE, pp 99–105Google Scholar
  3. Bethel CL, Salomon K, Murphy RR (2009) Preliminary results: humans find emotive non-anthropomorphic robots more calming. In: 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2009. IEEE, pp 291–292Google Scholar
  4. Bien ZZ, Lee HE (2007) Effective learning system techniques for human–robot interaction in service environment. Knowl-Based Syst 20(5):439–456CrossRefGoogle Scholar
  5. Cacioppo JT, Tassinary LG, Berntson G (2007) Handbook of psychophysiology. Cambridge University Press, CambridgeGoogle Scholar
  6. Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37CrossRefGoogle Scholar
  7. Conn K, Liu C, Sarkar N, Stone W, Warren Z (2010) Towards affect-sensitive assistive intervention technologies for children with autism. In Jimmy OR (ed) Affective computing: focus on emotion expression, synthesis and recognition. ARS/I-Tech Education and Publishing, Austria, pp 365–390Google Scholar
  8. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction. Sig Process Mag IEEE 18(1):32–80CrossRefGoogle Scholar
  9. Croft DKEA (2003) Estimating intent for human-robot interaction. In: IEEE International Conference on Advanced Robotics, 2003. pp 810–815Google Scholar
  10. D’Mello SK, Graesser A (2010) Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model User-Adap Inter 20(2):147–187CrossRefGoogle Scholar
  11. Dawson ME, Schell AM, Filion DL (2007) The Electrodermal System. In: Cacioppo JT, Tassinary LG, Berntson GG (eds) Handbook of psychophysiology. Cambridge University Press, New York, p 159Google Scholar
  12. Diehl JJ, Schmitt LM, Villano M, Crowell CR (2012) The clinical use of robots for individuals with autism spectrum disorders: A critical review. Res Autism Spectrum Disord 6(1):249–262CrossRefGoogle Scholar
  13. Dunn L, Williams KT, Wang JJ, Booklets N (1997) Peabody picture vocabulary test, (PPVT-III): Form IIA. American Guidance Service Inc, Circle PinesGoogle Scholar
  14. Fairclough SH (2009) Fundamentals of physiological computing. Interact Comput 21(1):133–145CrossRefGoogle Scholar
  15. Feil-Seifer D, Mataric M (2011) Automated detection and classification of positive vs. negative robot interactions with children with autism using distance-based features. In: Proceedings of the 6th international conference on Human-robot interaction, 2011. ACM, pp 323–330Google Scholar
  16. Feil-Seifer D, Mataric MJ (2005) A multi-modal approach to selective interaction in assistive domains. In: IEEE International Workshop on Robot and Human Interactive Communication, 2005. ROMAN 2005. IEEE, pp 416–421Google Scholar
  17. Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3):143–166zbMATHCrossRefGoogle Scholar
  18. Goodrich MA, Schultz AC (2007) Human-robot interaction: a survey. Found Trends Hum Comput Interact 1(3):203–275zbMATHCrossRefGoogle Scholar
  19. Hussain M, AlZoubi O, Calvo R, D’Mello S (2011) Affect detection from multichannel physiology during learning sessions with AutoTutor. In: Artificial intelligence in education. Springer, Berlin, pp 131–138Google Scholar
  20. Hussain M, Monkaresi H, Calvo R (2012) Categorical vs. dimensional representations in multimodal affect detection during learning. In: Intelligent tutoring systems. Springer, Berlin, pp 78–83Google Scholar
  21. Iani C, Gopher D, Lavie P (2004) Effects of task difficulty and invested mental effort on peripheral vasoconstriction. Psychophysiology 41(5):789–798CrossRefGoogle Scholar
  22. Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: IEEE 7th International Colloquium on Signal Processing and its Applications (CSPA), 2011. IEEE, pp 410–415Google Scholar
  23. Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083CrossRefGoogle Scholar
  24. Koenig A, Novak D, Omlin X, Pulfer M, Perreault E, Zimmerli L, Mihelj M, Riener R (2011) Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training. IEEE Trans Neural Syst Rehabil Eng 19(4):453–464CrossRefGoogle Scholar
  25. Kubicek W, Karnegis J, Patterson R, Witsoe D, Mattson R (1966) Development and evaluation of an impedance cardiac output system. Aerosp Med 37(12):1208Google Scholar
  26. Kulic D, Croft E (2005) Anxiety detection during human-robot interaction. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005.(IROS 2005). IEEE, pp 616–621Google Scholar
  27. Kulić D, Croft E (2007a) Pre-collision safety strategies for human-robot interaction. Auton Robots 22(2):149–164Google Scholar
  28. Kulic D, Croft EA (2007b) Affective state estimation for human–robot interaction. IEEE Trans Robot 23(5):991–1000Google Scholar
  29. Kwakkel G, Kollen BJ, Krebs HI (2008) Effects of robot-assisted therapy on upper limb recovery after stroke: a systematic review. Neurorehabil Neural Repair 22(2):111–121CrossRefGoogle Scholar
  30. Lacey JI, Lacey BC (1958) Verification and extension of the principle of autonomic response-stereotypy. The Am J Psychol 71(1):50–73MathSciNetCrossRefGoogle Scholar
  31. Lahiri U, Welch KC, Sarkar M (2012) Psychophysiological response in virtual reality based human-computer interaction in adolescents with ASD. In: Imaging and signal processing in health care and technology/772: human-computer interaction/773: communication, internet and information technology. ACTA Press, USAGoogle Scholar
  32. Leon E, Clarke G, Callaghan V, Sepulveda F (2007) A user-independent real-time emotion recognition system for software agents in domestic environments. Eng Appl Artif Intell 20(3):337–345CrossRefGoogle Scholar
  33. Liu C, Conn K, Sarkar N, Stone W (2007) Affect recognition in robot assisted rehabilitation of children with autism spectrum disorder. In: IEEE International Conference on Robotics and Automation, 2007. IEEE, pp 1755–1760Google Scholar
  34. Liu C, Conn K, Sarkar N, Stone W (2008a) Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE Trans Robot 24(4):883–896CrossRefGoogle Scholar
  35. Liu C, Conn K, Sarkar N, Stone W (2008b) Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. Int J Hum Comput Stud 66(9):662–677CrossRefGoogle Scholar
  36. Liu C, Rani P, Sarkar N (2005) An empirical study of machine learning techniques for affect recognition in human-robot interaction. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, (IROS 2005). IEEE, pp 2662–2667Google Scholar
  37. Liu C, Rani P, Sarkar N (2006) Affective state recognition and adaptation in human-robot interaction: A design approach. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems 2006. IEEE, pp 3099–3106Google Scholar
  38. Murphy RR (2004) Human-robot interaction in rescue robotics. IEEE Trans Syst Man Cybern Part C Appl Rev 34(2):138–153CrossRefGoogle Scholar
  39. Nourbakhsh IR, Sycara K, Koes M, Yong M, Lewis M, Burion S (2005) Human-robot teaming for search and rescue. IEEE Pervasive Comput 4(1):72–79CrossRefGoogle Scholar
  40. Pantic M, Pentland A, Nijholt A, Huang T (2007) Human computing and machine understanding of human behavior: A survey. Artifical intelligence for human computing. Springer, Berlin, pp 47–71Google Scholar
  41. Papillo J, Shapiro D (1990) The cardiovascular system. In: Principles of psychophysiology: physical, social and inferential elements. Cambridge University Press, CambridgeGoogle Scholar
  42. Pecchinenda A (1996) The affective significance of skin conductance activity during a difficult problem-solving task. Cogn Emot 10(5):481–504CrossRefGoogle Scholar
  43. Picard R (1997) Affective computing. MIT Press, CambridgeGoogle Scholar
  44. Picard RW (1999) Affective computing for HCI. In: Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Ergonomics and User Interfaces, 1999. pp 829–833Google Scholar
  45. Rani P, Liu C, Sarkar N, Vanman E (2006) An empirical study of machine learning techniques for affect recognition in human–robot interaction. Pattern Anal Appl 9(1):58–69CrossRefGoogle Scholar
  46. Rani P, Sarkar N (2005) Making robots emotion-sensitive-preliminary experiments and results. In: IEEE International Workshop on Robot and Human Interactive Communication, 2005. ROMAN 2005. IEEE, pp 1–6Google Scholar
  47. Rani P, Sarkar N, Smith CA, Kirby LD (2004) Anxiety detecting robotic system-towards implicit human-robot collaboration. Robotica 22(1):85–95CrossRefGoogle Scholar
  48. Rani P, Sims J, Brackin R, Sarkar N (2002) Online stress detection using psychophysiological signals for implicit human-robot cooperation. Robotica 20(06):673–685CrossRefGoogle Scholar
  49. Sarkar N (2002) Psychophysiological control architecture for human-robot coordination-concepts and initial experiments. In: Proceedings of the ICRA’02 IEEE International Conference on Robotics and Automation, 2002. IEEE, pp 3719–3724Google Scholar
  50. Scassellati B, Admoni H, Mataric M (2012) Robots for use in autism research. Annu Rev Biomed Eng 14:275–294CrossRefGoogle Scholar
  51. Severinson-Eklundh K, Green A, Hüttenrauch H (2003) Social and collaborative aspects of interaction with a service robot. Robot Auton Syst 42(3):223–234zbMATHCrossRefGoogle Scholar
  52. Tao J, Tan T (2005) Affective computing: A review. Affective computing and intelligent interaction. Springer, Heidelberg, pp 981–995Google Scholar
  53. Tapus A, Mataric M (2008) Socially assistive robots: The link between personality, empathy, physiological signals, and task performance. In: AAAI Spring, 2008. pp 3–4Google Scholar
  54. Thrun S (2004) Toward a framework for human-robot interaction. Hum Comput Interact 19(1–2):9–24CrossRefGoogle Scholar
  55. Vansteelandt K, Van Mechelen I, Nezlek JB (2005) The co-occurrence of emotions in daily life: A multilevel approach. J Res Pers 39(3):325–335CrossRefGoogle Scholar
  56. Viventi J, Kim D-H, Moss JD, Kim Y-S, Blanco JA, Annetta N, Hicks A, Xiao J, Huang Y, Callans DJ (2010) A conformal, bio-interfaced class of silicon electronics for mapping cardiac electrophysiology. Sci Transl Med 2(24):24ra22Google Scholar
  57. Wagner J, Kim J, André E (2005) From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: IEEE International Conference on Multimedia and Expo, 2005 ICME 2005. IEEE, pp 940–943Google Scholar
  58. Welch K, Lahiri U, Liu C, Weller R, Sarkar N, Warren Z (2009) An affect-sensitive social interaction paradigm utilizing virtual reality environments for autism intervention. Human-Computer Interaction Ambient, Ubiquitous and Intelligent Interaction, pp 703–712Google Scholar
  59. Wiering MA (2005) QV (lambda)-learning: A new on-policy reinforcement learning algorithm. Proceedings of the 7th European Workshop on Reinforcement LearningGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  1. 1.Vanderbilt UniversityNashvilleUSA

Personalised recommendations