Abstract
Psychophysiology is the study of the relationship between psychology and its physiological manifestations. That relationship is of particular importance for both game design and ultimately gameplaying. Players’ psychophysiology offers a gateway towards a better understanding of playing behavior and experience. That knowledge can, in turn, be beneficial for the player as it allows designers to make better games for them; either explicitly by altering the game during play or implicitly during the game design process. This chapter argues for the importance of physiology for the investigation of player affect in games, reviews the current state of the art in sensor technology and outlines the key phases for the application of psychophysiology in games.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
AlZoubi O, Calvo R, Stevens R (2009) Classification of EEG for affect recognition: an adaptive approach. In: AI 2009: advances in artificial intelligence. Springer, pp 52–61
Ambinder M (2011) Biofeedback in gameplay: how Valve measures physiology to enhance gaming experience. In: Game developers conference, San Francisco
Andreassi JL (2000) Psychophysiology: human behavior and physiological response. Psychology Press
Arroyo I, Cooper DG, Burleson W, Woolf BP, Muldner K, Christopherson R (2009) Emotion sensors go to school. In: Proceedings of conference on artificial intelligence in education (AIED). IOS Press, pp 17–24
Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose–application in an e-learning environment. Multimed Tools Appl 41(3):469–493
Asteriadis S, Karpouzis K, Shaker N, Yannakakis GN (2012) Does your profile say it all? Using demographics to predict expressive head movement during gameplay. In: UMAP workshops, citeseer
Banse R, Scherer KR (1996) Acoustic profiles in vocal emotion expression. J Personal Soc Psychol 70(3):614
Bänziger T, Tran V, Scherer KR (2005) The Geneva emotion wheel: a tool for the verbal report of emotional reactions. Poster presented at ISRE
Benedek M, Kaernbach C (2010) A continuous measure of phasic electrodermal activity. J Neurosci Methods 190(1):80–91
Bersak D, McDarby G, Augenblick N, McDarby P, McDonnell D, McDonald B, Karkun R (2001) Intelligent biofeedback using an immersive competitive environment. Paper at the designing ubiquitous computing games workshop at UbiComp
Bianchi-Berthouze N, Lisetti CL (2002) Modeling multimodal expression of users affective subjective experience. User Model User-Adapt Interact 12(1):49–84
Boucsein W (2012) Electrodermal activity. Springer, New York
Brosschot JF, Van Dijk E, Thayer JF (2007) Daily worry is related to low heart rate variability during waking and the subsequent nocturnal sleep period. Int J Psychophysiol 63(1):39–47
Busso C, Deng Z, Yildirim S, Bulut M, Lee CM, Kazemzadeh A, Lee S, Neumann U, Narayanan S (2004) Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proceedings of international conference on multimodal interfaces (ICMI). ACM, pp 205–211
Cacioppo JT, Berntson GG, Larsen JT, Poehlmann KM, Ito TA et al (2000) The psychophysiology of emotion. In: Lewis M, Haviland-Jones JM (eds) Handbook of emotions, vol 2. Guilford Press, New York, pp 173–191
Cacioppo JT, Tassinary LG, Berntson G (2007) Handbook of psychophysiology. Cambridge University Press, Cambridge/New York
Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37
Calvo RA, D’Mello SK (2011) New perspectives on affect and learning technologies, vol 3. Springer, New York
Calvo R, Brown I, Scheding S (2009) Effect of experimental factors on the recognition of affective mental states through physiological measures. In: AI 2009: advances in artificial intelligence. Springer, pp 62–70
Conati C, Maclaren H (2009) Modeling user affect from causes and effects. In: User modeling, adaptation, and personalization, Trento, pp 4–15
Cowie R, Cornelius RR (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1):5–32
Cowie R, Sawey M (2011) Gtrace-general trace program from queens, belfast
Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schröder M (2000) ’FEELTRACE’: an instrument for recording perceived emotion in real time. In: ISCA tutorial and research workshop (ITRW) on speech and emotion
Critchley HD, Mathias CJ, Dolan RJ (2002) Fear conditioning in humans: the influence of awareness and autonomic arousal on functional neuroanatomy. Neuron 33(4):653–663
De Melo C, Paiva A (2007) Expression of emotions in virtual humans using lights, shadows, composition and filters. In: Affective computing and intelligent interaction. Springer, Berlin/New York, pp 546–557
Dennerlein J, Becker T, Johnson P, Reynolds C, Picard RW (2003) Frustrating computer users increases exposure to physical factors. In: Proceedings of the international ergonomics association (IEA), Seoul
Devillers L, Vidrascu L (2006) Real-life emotions detection with lexical and paralinguistic cues on human-human call center dialogs. In: Proceedings of conference of the international speech communication association (Interspeech), Pittsburgh, pp 801–804
D’Mello S, Graesser A (2009) Automatic detection of learner’s affect from gross body language. Appl Artif Intell 23(2):123–150
Drachen A, Nacke L, Yannakakis GN, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the SIGGRAPH symposium on video games. ACM-SIGGRAPH Publishers, New York
Eladhari M, Nieuwdorp R, Fridenfalk M (2006) The soundtrack of your mind: mind music-adaptive audio for game characters. In: Proceedings of the 2006 ACM SIGCHI international conference on advances in computer entertainment technology. ACM, p 54
El-Nasr MS, Vasilakos A, Rao C, Zupko J (2009) Dynamic intelligent lighting for directing visual attention in interactive 3-D scenes. IEEE Trans Comput Intell AI Games 1(2):145–153
Farrugia VE, Martínez HP, Yannakakis GN (2015) The preference learning toolbox. arXiv preprint arXiv:1506.01709
Fürnkranz J, Hüllermeier E (2005) Preference learning. Künstliche Intelligenz 19(1):60–61
Giannatos S, Nelson MJ, Cheong Y-G, Yannakakis GN (2011) Suggesting new plot elements for an interactive story. In: Intelligent narrative technologies
Goldberger JJ, Challapalli S, Tung R, Parker MA, Kadish AH (2001) Relationship of heart rate variability to parasympathetic effect. Circulation 103(15):1977–1983
Grafsgaard J, Boyer K, Lester J (2011) Predicting facial indicators of confusion with hidden Markov models. In: Proceedings of international conference on affective computing and intelligent interaction (ACII). Springer, Memphis, pp 97–106
Haykin S, Widrow B (2003) Least-mean-square adaptive filters, vol 31. Wiley, Hoboken
Hazlett RL (2006) Measuring emotional valence during interactive experiences: boys at video game play. In: Proceedings of SIGCHI conference on human factors in computing systems (CHI). ACM, New York, pp 1023–1026
Holmgård C, Yannakakis GN, Karstoft K-I, Andersen HS (2013) Stress detection for PTSD via the startlemart game. In: 2013 humaine association conference on affective computing and intelligent interaction (ACII). IEEE, Piscataway, pp 523–528
Holmgård C, Yannakakis GN, Martínez HP, Karstoft K-I (2015) To rank or to classify? Annotating stress for reliable PTSD profiling. In: 2015 international conference on affective computing and intelligent interaction (ACII), Xi’an
Holmgård C, Yannakakis GN, Martínez HP, Karstoft K-I, Andersen HS (2015) Multimodal PTSD characterization via the startlemart game. J Multimodal User Interfaces 9(1):3–15
Hussain M, AlZoubi O, Calvo R, D’Mello S (2011) Affect detection from multichannel physiology during learning sessions with autotutor. In: Proceedings of international conference in artificial intelligence in education (AIED). Springer, Heidelberg, pp 131–138
Johnstone T, Scherer KR (2000) Vocal communication of emotion. In: Lewis M, Haviland-Jones JM (eds) Handbook of emotions, vol 2. Guilford Press, New York, pp 220–235
Jönsson P (2007) Respiratory sinus arrhythmia as a function of state anxiety in healthy individuals. Int J Psychophysiol 63(1):48–54
Juslin PN, Scherer KR (2005) Vocal expression of affect. Oxford University Press, Oxford
Kaliouby R, Picard R, Baron-Cohen S (2006) Affective computing and autism. Ann N Y Acad Sci 1093(1):228–248
Kannetis T, Potamianos A (2009) Towards adapting fantasy, curiosity and challenge in multimodal dialogue systems for preschoolers. In: Proceedings of international conference on multimodal interfaces (ICMI). ACM, New York, pp 39–46
Kannetis T, Potamianos A, Yannakakis GN (2009) Fantasy, curiosity and challenge as adaptation indicators in multimodal dialogue systems for preschoolers. In: Proceedings of the 2nd workshop on child, computer and interaction. ACM, New York, p 1
Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum-Comput Stud 65(8):724–736
Kivikangas JM, Ekman I, Chanel G, Järvelä S, Salminen M, Cowley B, Henttonen P, Ravaja N (2010) Review on psychophysiological methods in game research. In: Procedings of Nordic digital games research association conference (Nordic DiGRA)
Leite I, Mascarenhas S, Pereira A, Martinho C, Prada R, Paiva A (2010) “Why can’t we be friends?” An empathic game companion for long-term interaction. In: Intelligent virtual agents. Springer, Berlin, pp 315–321
Lisetti CL, Nasoz F (2004) Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J Appl Signal Process 2004:1672–1687
Lisetti C, Nasoz F, LeRouge C, Ozyer O, Alvarez K (2003) Developing multimodal intelligent affective interfaces for tele-home health care. Int J Hum-Comput Stud 59(1):245–255
Liu C, Conn K, Sarkar N, Stone W (2008) Physiology-based affect recognition for computer-assisted intervention of children with autism spectrum disorder. Int J Hum-Comput Stud 66(9):662–677
Lopes P, Liapis A, Yannakakis GN (2015) Sonancia: sonification of procedurally generated game levels. In: Proceedings of the 1st computational creativity and games workshop
Lopes P, Liapis A, Yannakakis GN (2015) Targeting horror via level and soundscape generation
Malandrakis N, Potamianos A, Evangelopoulos G, Zlatintsi A (2011) A supervised approach to movie emotion tracking. In: 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Piscataway, pp 2376–2379
Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int J Hum-Comput Stud 65(4):329–347
Mandryk RL, Inkpen KM, Calvert TW (2006) Using psychophysiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25(2):141–158
Martínez HP, Yannakakis GN (2010) Genetic search feature selection for affective modeling: a case study on reported preferences. In: Proceedings of the 3rd international workshop on affective interaction in natural environments. ACM, pp 15–20
Martínez HP, Yannakakis GN (2011) Mining multimodal sequential patterns: a case study on affect detection. In: Proceedings of the 13th international conference on multimodal interfaces. ACM, pp 3–10
Martínez HP, Bengio Y, Yannakakis GN (2013) Learning deep physiological models of affect. IEEE Comput Intell Mag 9(1):20–33
Martinez H, Yannakakis G, Hallam J (2014) Don’t classify ratings of affect; rank them!. IEEE Trans Affect Comput 5(3):314–326
Mateas M, Stern A (2003) Façade: an experiment in building a fully-realized interactive drama. In: Game developers conference, vol 2
McQuiggan S, Lee S, Lester J (2007) Early prediction of student frustration. In: Proceedings of international conference on affective computing and intelligent interaction. Springer, pp 698–709
Mcquiggan SW, Mott BW, Lester JC (2008) Modeling self-efficacy in intelligent tutoring systems: an inductive approach. User Model User-Adapt Interact 18(1):81–123
Messinger DS, Cassel TD, Acosta SI, Ambadar Z, Cohn JF (2008) Infant smiling dynamics and perceived positive emotion. J Nonverbal Behav 32(3):133–155
Nacke L, Lindley CA (2008) Flow and immersion in first-person shooters: measuring the player’s gameplay experience. In: Proceedings of conference on future play: research, play, share. ACM, pp 81–88
Nagel F, Kopiez R, Grewe O, Altenmüller E (2007) Emujoy: software for continuous measurement of perceived emotions in music. Behav Res Methods 39(2):283–290
Nijholt A (2009) BCI for games: a ‘state of the art’ survey. In: Entertainment computing-ICEC 2008. Springer, pp 225–228
Pedersen C, Togelius J, Yannakakis GN (2010) Modeling player experience for content creation. IEEE Trans Comput Intell AI Games 2(1):54–67
Picard RW (2009) Future affective technology for autism and emotion communication. Philos Trans R Soc B: Biol Sci 364(1535):3575–3584
Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191
Picard RW, Papert S, Bender W, Blumberg B, Breazeal C, Cavallo D, Machover T, Resnick M, Roy D, Strohecker C (2004) Affective learning – a manifesto. BT Technol J 22(4):253–269
Poels K, de Kort Y, Ijsselsteijn W (2007) It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. In: Proceedings of the 2007 conference on future play. ACM, pp 83–89
Poh M-Z, McDuff DJ, Picard RW (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 18(10):10762–10774
Qu L, Wang N, Johnson W (2005) Using learner focus of attention to detect learner motivation factors. In: Proceedings of international conference on user modeling (UM). Springer, pp 149–149
Rani P, Sarkar N, Liu C (2005) Maintaining optimal challenge in computer games through real-time physiological feedback. In: Proceedings of the 11th international conference on human computer interaction, pp 184–192
Ravaja N, Saari T, Laarni J, Kallinen K, Salminen M, Holopainen J, Jarvinen A (2005) The psychophysiology of video gaming: phasic emotional responses to game events. In: Proceedings of digital games research association conference (DiGRA)
Ravaja N, Saari T, Salminen M, Laarni J, Kallinen K (2006) Phasic emotional reactions to video game events: a psychophysiological investigation. Media Psychol 8(4):343–367
Rebolledo-Mendez G, Dunwell I, Martínez-Mirón E, Vargas-Cerdán M, De Freitas S, Liarokapis F, García-Gaona A (2009) Assessing neurosky’s usability to detect attention levels in an assessment exercise. In: Human-computer interaction. New trends, pp 149–158
Riedl M, Bulitko V (2012) Interactive narrative: a novel application of artificial intelligence for computer games. AAAI, Citeseer
Robison J, McQuiggan S, Lester J (2009) Evaluating the consequences of affective feedback in intelligent tutoring systems. In: Proceedings of international conference on affective computing and intelligent interaction (ACII). IEEE, pp 1–6
Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39 (6):1161
Schwarz N (2000) Emotion, cognition, and decision making. Cogn Emot 14(4):433–440
Shaker N, Yannakakis GN, Togelius J (2010) Towards automatic personalized content generation for platform games. In: Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment (AIIDE). AAAI Press
Shaker N, Asteriadis S, Yannakakis GN, Karpouzis K (2013) Fusing visual and behavioral cues for modeling user experience in games. IEEE Trans Cybern 43(6):1519–1531
Sharma N, Gedeon T (2012) Objective measures, sensors and computational techniques for stress recognition and classification: a survey. Comput Methods Programs Biomed 108(3):1287–1301
Sokolov EN (1963) Higher nervous functions: the orienting reflex. Annu Rev Physiol 25(1):545–580
Tijs T, Brokken D, Ijsselsteijn W (2008) Dynamic game balancing by recognizing affect. In: Proceedings of international conference on fun and games. Springer, pp 88–93
Togelius J, Schmidhuber J (2008) An experiment in automatic game design. In: IEEE symposium on computational intelligence and games, CIG’08. IEEE, pp 111–118
Togelius J, De Nardi R, Lucas SM (2007) Towards automatic personalised content creation for racing games. In: IEEE symposium on computational intelligence and games, CIG 2007. IEEE, pp 252–259
Togelius J, Preuss M, Beume N, Wessing S, Hagelback J, Yannakakis GN (2010) Multiobjective exploration of the starcraft map space. In: 2010 IEEE symposium on computational intelligence and games (CIG). IEEE, pp 265–272
Togelius J, Yannakakis GN, Stanley KO, Browne C (2011) Search-based procedural content generation: a taxonomy and survey. IEEE Trans Comput Intell AI Games 3(3):172–186
Tognetti S, Garbarino M, Bonarini A, Matteucci M (2010) Modeling enjoyment preference from physiological responses in a car racing game. In: Proceedings of IEEE conference on computational intelligence and games (CIG). IEEE, pp 321–328
van den Hoogen WM, IJsselsteijn WA, de Kort YAW (2008) Exploring behavioral expressions of player experience in digital games. In: Proceedings of the workshop on facial and bodily expression for control and adaptation of games (ECAG), pp 11–19
Vogt T, André E (2005) Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of IEEE international conference on multimedia and expo (ICME). IEEE, pp 474–477
Yang Y-H, Chen HH (2011) Ranking-based emotion recognition for music organization and retrieval. IEEE Trans Audio Speech Lang Process 19(4):762–774
Yannakakis GN (2009) Preference learning for affective modeling. In: 3rd international conference on affective computing and intelligent interaction and workshops, ACII 2009, Amsterdam, Sept 2009. IEEE, pp 1–6
Yannakakis GN, Hallam J (2007) Towards optimizing entertainment in computer games. Appl Artif Intell 21(10):933–971
Yannakakis GN, Hallam J (2008) Entertainment modeling through physiology in physical play. Int J Hum-Comput Stud 66(10):741–755
Yannakakis GN, Hallam J (2009) Real-time game adaptation for optimizing player satisfaction. IEEE Trans Comput Intell AI Games 1(2):121–133
Yannakakis G, Hallam J (2011) Rating vs. preference: a comparative study of self-reporting. In: Proceedings of international conference on affective computing and intelligent interaction (ACII). Springer, pp 437–446
Yannakakis GN, Martínez HP (2015) Grounding truth via ordinal annotation. In: 2015 international conference on affective computing and intelligent interaction (ACII)
Yannakakis GN, Martínez HP (2015) Ratings are overrated! Front ICT 2:13
Yannakakis GN, Paiva A (2013) Emotion in games. In: Handbook on affective computing, p 20
Yannakakis GN, Togelius J (2011) Experience-driven procedural content generation. IEEE Trans Affect Comput 2(3):147–161
Yannakakis GN, Hallam J, Lund HH (2008) Entertainment capture through heart rate activity in physical interactive playgrounds. User Model User-Adapt Interact 18(1):207–243
Yannakakis GN, Martínez HP, Jhala A (2010) Towards affective camera control in games. User Model User-Adapt Interact 20(4):313–340
Yildirim S, Narayanan S, Potamianos A (2011) Detecting emotional state of a child in a conversational computer game. Comput Speech Lang 25(1):29–44
Zeng Z, Pantic M, Roisman G, Huang TS et al (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58
Acknowledgements
The work is supported, in part, by the EU-funded FP7 ICT iLearnRW project (project no: 318803).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Yannakakis, G.N., Martinez, H.P., Garbarino, M. (2016). Psychophysiology in Games. In: Karpouzis, K., Yannakakis, G. (eds) Emotion in Games. Socio-Affective Computing, vol 4. Springer, Cham. https://doi.org/10.1007/978-3-319-41316-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-41316-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-41314-3
Online ISBN: 978-3-319-41316-7
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)