Skip to main content
SpringerLink
Log in

From Sinewaves to Physiologically-Adaptive Soundscapes: The Evolving Relationship Between Sound and Emotion in Video Games

  • Chapter
  • First Online:
Emotion in Games

Part of the book series: Socio-Affective Computing ((SAC,volume 4))

Abstract

This chapter examines the dynamic and evolving relationship between sound and emotion within the context of video games. How sound in games has been utilised to both infer and evoke emotion is discussed, commencing with an historical review that traces back to video games’ humble beginnings. As we move towards the present day this chapter looks at how biofeedback technology, that can facilitate the control and procedural generation of game sound content by way of player-emotion, is transforming the lateral affective interplay between player and video game into something more circular.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A form of tuning that spaces out each note within the octave equally along the frequency spectrum.

  2. 2.

    A designer responsible for recreating natural sounds in a studio.

  3. 3.

    A singular, short and intense note or chord typically utilised to add dramatic impact.

  4. 4.

    A short recurring musical phrase that has a designated association with a locale, event or character.

References

  1. Alves V, Roque L (2009) A proposal of soundscape design guidelines for user experience enrichment. Audio Mostly 2009. September 2nd-3rd, Glasgow

    Google Scholar 

  2. Amon K, Campbell A (2008) Can children with ADHD learn relaxation and breathing techniques through biofeedback video games? Aust J Educ Dev Psychol 8:72–84

    Google Scholar 

  3. AMD (2015) TrueAudio. http://www.amd.com/en-us/innovations/software-technologies/trueaudio. Retrieved 08 July 2015

  4. Atari POKEY data Sheet (2015) http://krap.pl/mirrorz/atari/homepage.ntlworld.com/kryten_droidAtari/800XL/atari_hw/pokey.htm. Retrieved 01 July 2015

  5. Boris D (1998) Odyssey 2 Technical Specs http://atarihq.com/danb/files/o2doc.pdf. Retrieved 25 June 2015

  6. Bradley MM, Lang P (2000) Affective reactions to acoustic stimuli. Psychophysiology 37:204–215

    Article  CAS  PubMed  Google Scholar 

  7. Breinbjerg M (2005) The aesthetic experience of sound – staging of auditory spaces in 3D computer games. Aesthetics of Play. Bergen, Norway, October 14th–15th

    Google Scholar 

  8. Buchanan K (2013) You’ll never guess how the dinosaur sounds in Jurassic Park were made. http://www.vulture.com/2013/04/how-the-dino-sounds-in-jurassic-park-were-made.html. Retrieved 07 July 2015

  9. Cacioppo JT, Tassinary LG (1990) Principles of psychophysiology: physical, social, and inferential elements. Cambridge University Press, New York

    Google Scholar 

  10. Cantor D (1971) A computer program that accepts common musical notation. Comput Hum 6(2):103–109

    Article  Google Scholar 

  11. Chanel G, Kierkels JJM, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67(8):607–662

    Article  Google Scholar 

  12. Chang K, Kim G, Kim T (2007, August) Video game console audio: evolution and future trends. In Computer graphics, imaging and visualisation, 2007. CGIV’07

    Google Scholar 

  13. Cho J, Yi E, Cho G (2001) Physiological responses evoked by fabric sounds and related mechanical and acoustical properties. Text Res J 71(12):1068–1073

    Article  CAS  Google Scholar 

  14. Cole T (2015) The tragedy of betrayal: how the design of Ico and Shadow of the Colossus elicits emotion

    Google Scholar 

  15. Collins K (2008) Game sound: an introduction to the history, theory, and practice of video game music and sound design. MIT Press

    Google Scholar 

  16. Collins K (2011) Making gamers cry: mirror neurons and embodied interaction with game sound. In Proceedings of the 6th audio mostly conference: a conference on interaction with sound, 39–46. ACM

    Google Scholar 

  17. Connor S (2010) Suspense-building music mimicks sounds of animals in distress. Independent Online. http://www.independent.co.uk/news/science/why-calls-of-the-wild-are-the-secret-of-a-good-horror-film-1982965.html. Retrieved 07 July 2015

  18. Critchley HD, Mathias CJ, Dolan RJ (2002) Fear conditioning in humans: the influence of awareness and autonomic arousal on functional neuroanatomy. Neuron 33:653–663

    Article  CAS  PubMed  Google Scholar 

  19. Dekker A, Champion E (2007) Please biofeed the zombies: enhancing the gameplay and display of a horror game using biofeedback. Proceedings of DiGRA, 550–558

    Google Scholar 

  20. Dixon MJ, MacLaren V, Jarick M, Fugelsang JA, Harrigan KA (2013) The frustrating effects of just missing the jackpot: slot machine near-misses trigger large skin conductance responses, but no post-reinforcement pauses. J Gambl Stud 29(4):661–674

    Article  PubMed  Google Scholar 

  21. Drachen A, Nacke LE, Yannakakis G, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. 5th ACM SIGGRAPH Symposium on Video Games. 49–54. ACM

    Google Scholar 

  22. Duggan M (2008) Torque for teens. Cengage Learning

    Google Scholar 

  23. Ekman I (2008) Psychologically motivated techniques for emotional sound in computer games. Audio Mostly 2008 (Pitea, Sweden)

    Google Scholar 

  24. Fischer G, Grudin J, Lemke A et al (1992) Supporting indirect collaborative design with integrated knowledge-based design environments. Hum Comput Interact 7(3):281–314

    Article  Google Scholar 

  25. Gilroy SW, Porteous J, Charles F, Cavazza MO (2012) Exploring passive user interaction for adaptive narratives. Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces. Lisbon, Portugal, 14th–17th February 2012, ACM, New York, pp 119–128

    Google Scholar 

  26. Gonzalez-Sanchez J, Chavez-Echeagaray ME, Atkinson R, Burleson W (2011) ABE: an agent based software architecture for a multimodal emotion recognition framework. In: Proceedings of the ninth working IEEE/IFIP conference on software architecture. pp 187–193

    Google Scholar 

  27. Graetz JM (1981) The origin of spacewar. Creat Comput 7(8):56–67

    Google Scholar 

  28. Grimshaw M, Garner T (2015) Sonic virtuality: sound as emergent perception. Oxford University Press, New York

    Book  Google Scholar 

  29. Grimshaw M, Lindley CA, Nacke L (2008) Sound and immersion in the first-person shooter: mixed measurement of the player's sonic experience. Audio Mostly 2008, Piteå, Sweden

    Google Scholar 

  30. Hamalainen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounesmaa OV (1993) Magnetoencephalography: theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev Mod Physiol 65:413–497

    Article  CAS  Google Scholar 

  31. Hamilton MF, Blackstock DT (1998) Nonlinear acoustics, vol 427. Academic, San Diego

    Google Scholar 

  32. Headlee K, Koziupa T, Siwiak D (2010) Sonic virtual reality game: How does your body sound? In: International conference on new interfaces for musical expression, Sydney, Australia, pp 15–18

    Google Scholar 

  33. Hermann T, Hunt A, Neuhoff (2011) The sonification handbook. Logos Verlag, Berlin

    Google Scholar 

  34. Huron DB (2006) Sweet anticipation: music and the psychology of expectation. MIT Press

    Google Scholar 

  35. Ismail F, Biedert R, Dengel A, Buscher G (2011) Emotional text tagging. http://gbuscher.com/publications/IsmailBiedert11_EmotionalTextTagging.pdf

  36. Jennett C, Cox AL, Cairns P et al (2008) Measuring and defining the experience of immersion in games. Int J Hum Comput Stud 66(9):641–661

    Article  Google Scholar 

  37. Jørgensen K (2006) On the functional aspects of computer game audio. Audio-Mostly 2006, Pitea, Sweden

    Google Scholar 

  38. Koelsch S, Kilches S, Steinbeis N, Schelinski S (2008) Effects of unexpected chords and of performer’s expression on brain responses and electrodermal activity. PLoS ONE 3(7)

    Google Scholar 

  39. Kuikkaniemi K, Laitinen T, Turpeinen M, Saari T, Kosunen I, Ravaja N (2010) The influence of implicit and explicit biofeedback in first-person shooter games. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, pp 859–868

    Google Scholar 

  40. Lang PJ, Bradley MM, Cuthbert BN, Patrick CJ (1993) Emotion and psychopathology: a startle probe analysis. Prog Exp Pers Psychopathol Res 16:163–199

    CAS  PubMed  Google Scholar 

  41. Linzmayer O (1983) The voice of odyssey. Creat Comput Video Arcade Games 1(1):60

    Google Scholar 

  42. Lopes P, Liapis A, Yannakakis GN (2015) Sonancia: sonification of procedurally Generated Game Levels. http://www.ccgworkshop.org/wp-content/uploads/2015/06/CCGW2015_submission_4.pdf. Retrieved 09 July 2015

  43. McAllister N (2011) Biometrics – the future of video games? Edge Online, http://www.edge-online.com/features/biometrics-future-videogames. Retrieved 11 Feb 2014

  44. Mirza-Babaei P, Long S, Foley E, McAllister G (2011) Understanding the contribution of biometrics to games user research, DiGRA, pp 329–347

    Google Scholar 

  45. Moncrieff S, Venkatesh S, Dorai C (2001) Affect computing in film through sound energy dynamics International Multimedia Conference. Ninth ACM Int Conf Multimed 9:525–527

    Article  Google Scholar 

  46. Murugappan M, Ramachandran N, Sazali Y (2009) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3(4):390–396

    Article  Google Scholar 

  47. Nacke L, Ambinder M, Canossa A, Mandryk R, Stach T (2009) Game metrics and biometrics: the future of player experience research. Future Play, 2009

    Google Scholar 

  48. Nacke LE, Kalyn M, Lough C, Mandryk RL (2011) Biofeedback game design: using direct and indirect physiological control to enhance game interaction. In: SIGCHI conference on human factors in computing systems. pp 103–112

    Google Scholar 

  49. Oculus VR (2016) www.oculus.com. Accessed 29 Sep 2016

  50. Parker JR, Heerema J (2008) Audio interaction in computer mediated games. Int J Comput Games Technol 2008:1

    Article  Google Scholar 

  51. Perron B (2004) Sign of a threat: the effects of warning systems in survival horror games. Cosign 2004 Proceedings. Art Academy, University of Split, pp 132–141

    Google Scholar 

  52. Plans E, Morelli D, Plans D (2015) AudioNode: prototypical affective modelling in experience-driven procedural music generation. http://www.ccgworkshop.org/wp-content/uploads/2015/06/CCGW2015_submission_7.pdf. Retrieved 09 July 2015

  53. Poh MZ, Loddenkemper T, Swenson NC, Goyal S, Madsen JR, Picard RW (2010) Continuous monitoring of electrodermal activity during epileptic seizures using a wearable sensor. In: Conf Proc IEEE Eng Med Biol Soc. pp 4415–4418

    Google Scholar 

  54. Ravaja N (2004) Contributions of psychophysiology to media research: review and recommendations. Media Psychol 6:193–235

    Article  Google Scholar 

  55. Sato W, Fujimura T, Suzuki N (2008) Enhanced facial EMG activity in response to dynamic facial expressions. Int J Psychophysiol 70(1):70–74

    Article  PubMed  Google Scholar 

  56. Shilling R, Zyda M, Wardynski EC (2002) Introducing emotion into military simulation and video game design America’s Army: operations and VIRTE. GAME-ON

    Google Scholar 

  57. SPC700 APU Manual (1997) http://snesmusic.org/files/spc700_apu_manual.txt. Retrieved 03 July 2015

  58. Steckler L (1975) TV Games Al ‘I-Iome

    Google Scholar 

  59. Stony Brook University (2011) Tennis for two: history, http://www.stonybrook.edu/libspecial/videogames/tennis.html. Retrieved 02 July 2015

  60. Sweet M (2014) Writing interactive music for video games: a composer’s guide. Pearson Education

    Google Scholar 

  61. Tan ES (2000) Emotion, art, and the humanities. In: Lewis M, Haviland-Jones JM (eds) Handbook of emotions, 2nd edn. Guilford Press, New York, pp 116–134

    Google Scholar 

  62. Väisänen J, Väisänen O, Malmivuo J, Hyttinen J (2008) New method for analysing sensitivity distributions of electroencephalography measurements. Med Biol Eng Comput 46:101–108

    Article  PubMed  Google Scholar 

  63. Video Game Consile Library (2014) http://www.videogameconsolelibrary.com/pg70-bally.htm#page=specs. Retrieved 20 June 2015

  64. Visch VT, Tan ES, Molenaar D (2010) The emotional and cognitive effect of immersion in film viewing. Cognit Emot 24(8):1439–1445

    Article  Google Scholar 

  65. Winter D (2013) Atari PONG: the first steps, http://www.pong-story.com/atpong1.htm. Retrieved 01 July 2015

  66. Woodward JD, Orlans NM, Higgins PT (2003) Biometrics: [identity assurance in the information age]. McGraw-Hill/Osborne, New York

    Google Scholar 

  67. Xu M, Chia L, Jin J (2005) Affective content analysis in comedy and horror videos by audio emotional event detection. In: IEEE International Conference on Multimedia and Expo

    Google Scholar 

  68. Yokota T, Fujimori B (1962) Impedence change of the skin during the galvanic skin reflex. Jpn J Physiol 12:200–209

    Article  CAS  PubMed  Google Scholar 

  69. Yoo MJ, Beak JW, Lee IK (2011) Creating musical expression using kinect. In: Proceedings of New Interfaces for Musical Expression. Oslo, Norway

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Creative and Cultural Industries, University of Portsmouth, Portsmouth, UK

    Tom A. Garner

Authors
  1. Tom A. Garner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tom A. Garner .

Editor information

Editors and Affiliations

  1. Institute of Communication and Computer Systems, National Technical University of Athens, Zographou, Greece

    Kostas Karpouzis

  2. Institute of Digital Games, University of Malta, Msida, Malta

    Georgios N. Yannakakis

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Garner, T.A. (2016). From Sinewaves to Physiologically-Adaptive Soundscapes: The Evolving Relationship Between Sound and Emotion in Video Games. In: Karpouzis, K., Yannakakis, G. (eds) Emotion in Games. Socio-Affective Computing, vol 4. Springer, Cham. https://doi.org/10.1007/978-3-319-41316-7_12

Download citation

Publish with us

Policies and ethics

Search

Navigation