Abstract
To recognize boredom in users interacting with machines is valuable to improve user experiences in human-machine long term interactions, especially for intelligent tutoring systems, health-care systems, and social assistants. This paper proposes a two-staged framework and feature design for boredom recognition in multiparty human-robot interactions. At the first stage the proposed framework detects boredom-indicating user behaviors based on skeletal data obtained by motion capture, and then it recognizes boredom in combination with detection results and two types of multiparty information, i.e., gaze direction to other participants and incoming-and-outgoing of participants. We experimentally confirmed the effectiveness of both the proposed framework and the multiparty information. In comparison with a simple baseline method, the proposed framework gained 35% points in the F1 score.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This is intended to make an annotator’s judgement on an interval independent of the neighboring intervals as much as possible in a reasonable cost of annotation. In forward annotation, the influence of the judgement of an interval on the judgement of the next interval could be larger. The playback of each interval was of course forwarded.
- 2.
Although it is known that there are several types of boredom [6], we employ a naive notion of boredom for the sake of simplicity.
- 3.
The meanings of strong/weak are different between the boredom state and the other boredom behavior categories (BB-*). For the boredom state, ‘strong’, ‘weak’, and ‘none’ respectively mean the samples with boredom state labels ‘BORED’, ‘MAYBE’, and ‘NOT BORED’. For the other behavior categories, ‘strong’ means the samples that contain relevant behaviors more than 2.5 s. ‘weak’ means those less than 2.5 s, and ‘none’ means those with no relevant behaviors.
- 4.
‘#targets’ means the numbers of samples in column ‘strong’ shown in Table 2. ‘#residuals’ means the sums of the numbers of the ‘weak’ and ‘none’ samples.
References
Bixler, R., D’Mello, S.: Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. In: 2013 International Conference on Intelligent User Interfaces, pp. 225–234 (2013)
Brand, M., Oliver, N., Pentland, A.: Coupled hidden markov models for complex action recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 994–999 (1997)
Breazeal, C., Aryananda, L.: Recognition of affective communicative intent in robot-directed speech. Auton. Robots 12, 83–104 (2002)
Castellano, G., Pereira, A., Leite, I., Paiva, A., McOwan, P.W.: Detecting user engagement with a robot companion using task and social interaction-based features. In: 2009 International Conference on Multimodal Interfaces, pp. 119–126 (2009)
D’Mello, S., Chipman, P., Graesser, A.: Posture as a predictor of learner’s affective engagement. In: 29th Annual Conference of the Cognitive Science Society, pp. 905–910 (2007)
Goetz, T., Frenzel, A.C., Hall, N.C., Nett, U.E., Pekrun, R., Lipnevich, A.A.: Types of boredom: an experience sampling approach. Motiv. Emot. 38(3), 401–419 (2014)
Gratch, J., Wang, N., Okhmatovskaia, A., Lamothe, F., Morales, M., Werf, R.J., Morency, L.-P.: Can virtual humans be more engaging than real ones? In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 286–297. Springer, Heidelberg (2007)
Hudlicka, E.: To feel or not to feel: the role of affect in human-computer interaction. Int. J. Hum. Comput. Stud. 59(1), 1–32 (2003)
Jacko, J.A.: Human Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. CRC Press, Boca Raton (2012)
Jacobs, A.M., Fransen, B., McCurry, J.M., Heckel, F.W., Wagner, A.R., Trafton, J.G.: A preliminary system for recognizing boredom. In: 4th ACM/IEEE International Conference on Human-Robot Interaction, pp. 299–300 (2009)
Kennedy, L.S., Ellis, D.P.: Pitch-based emphasis detection for characterization of meeting recordings. In: 2003 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 243–248 (2003)
Liu, Y., Chee, Y.S.: Intelligent pedagogical agents with multiparty interaction support. In: IEEE/WIC/ACM International Conference on Intelligent Agent Technology, pp. 134–140 (2004)
Mehrabian, A.: Silent messages. Wadsworth, Belmont (1971)
Mota, S., Picard, R.W.: Automated posture analysis for detecting learner’s interest level. In: Computer Vision and Pattern Recognition Workshop, pp. 49–49 (2003)
Pantic, M., Caridakis, G., André, E., Kim, J., Karpouzis, K., Kollias, S.: Multimodal emotion recognition from low-level cues. In: Petta, P., Pelachaud, C., Cowie, R. (eds.) Emotion-Oriented Systems, pp. 115–132. Springer, Heidelberg (2011)
Qvarfordt, P., Beymer, D., Zhai, S.: RealTourist – a study of augmenting human-human and human-computer dialogue with eye-gaze overlay. In: Costabile, M.F., Paternò, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 767–780. Springer, Heidelberg (2005). doi:10.1007/11555261_61
Sugiyama, T., Funakoshi, K., Nakano, M., Komatani, K.: Estimating response obligation in multi-party human-robot dialogues. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots, pp. 166–172 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Shibasaki, Y., Funakoshi, K., Shinoda, K. (2017). Boredom Recognition Based on Users’ Spontaneous Behaviors in Multiparty Human-Robot Interactions. In: Amsaleg, L., Guðmundsson, G., Gurrin, C., Jónsson, B., Satoh, S. (eds) MultiMedia Modeling. MMM 2017. Lecture Notes in Computer Science(), vol 10132. Springer, Cham. https://doi.org/10.1007/978-3-319-51811-4_55
Download citation
DOI: https://doi.org/10.1007/978-3-319-51811-4_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-51810-7
Online ISBN: 978-3-319-51811-4
eBook Packages: Computer ScienceComputer Science (R0)