Motion Controllers, Sound, and Music in Video Games: State of the Art and Research Perspectives

Chapter
Part of the International Series on Computer Entertainment and Media Technology book series (ISCEMT)

Abstract

This chapter is dedicated to the use of motion-sensing controllers in video games, with particular attention to gameplay mechanics that link body movement, sound, an music. First, we present a review of the state of the art. This includes an overview of recent motion controllers describing the different motion-sensing technologies employed and the implications such technologies have on game design and gameplay. This is followed by a set of examples of relationships between motion control, sound, music, and gameplay mechanics in recent video games. Secondly, we present a survey on recent research in expressive movement analysis and motion-based interaction, introducing the concepts of motion descriptor and parameter mapping. We then report on two serious games designed for children affected by autism using full-body motion capture and emotion recognition, and studies of player engagement in motion-controlled video games. In light of the interdisciplinary research presented, we finally discuss perspectives for motion-based game design.

References

  1. Alaoui, S.F., Caramiaux, B., Serrano, M., Bevilacqua, F.: Movement qualities as interaction modality. In: Proceedings of the Designing Interactive Systems Conference, pp. 761–769. ACM (2012)Google Scholar
  2. Bernardo, F., Arner, N., Batchelor, P.: O Soli Mio : exploring millimeter wave radar for musical interaction. In: Erkut, C. (ed.) NIME’17 International Conference on New Interfaces for Musical Expression, pp. 283–286, Copenhagen (2017). http://homes.create.aau.dk/dano/nime17/papers/0054/paper0054.pdf
  3. Bianchi-Berthouze, N.: Understanding the role of body movement in player engagement. Hum. Comp. Interact. 28(1), 40–75 (2013). http://www.tandfonline.com/doi/abs/10.1080/07370024.2012.688468
  4. Brown, D., Renney, N., Stark, A., Nash, C., Mitchell, T.: Leimu: gloveless music interaction using a wrist mounted leap motion. Proc. Int. Conf. New Interf. Mus. Expr. 16, 300–304 (2016). http://www.nime.org/proceedings/2016/nime2016{\_}paper0059.pdf
  5. Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2, 1–43 (1998)CrossRefGoogle Scholar
  6. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) Gesture-based Communication in Human-Computer Interaction, pp. 20–39, vol. 2915, Springer, Berlin (2004). http://link.springer.com/10.1007/978-3-540-24598-8{\_}3
  7. Camurri, A., Coletta, P., Demurtas, M., Peri, M., Ricci, A., Sagoleo, R., Simonetti, M., Varni, G., Volpe, G.: A platform for real-time multimodal processing. In: Proceedings International Conference Sound and Music Computing, pp. 11–13 (2007). http://www.smc-conference.org/smc07/SMC07Proceedings/SMC07Paper59.pdf
  8. Caramiaux, B., Françoise, J., Schnell, N., Bevilacqua, F.: Mapping through listening. Comput. Music J. 38(3), 34–48 (2014)Google Scholar
  9. CBS Interactive: Dance central for Xbox 360 reviews - Metacritic (2017a). http://www.metacritic.com/game/xbox-360/dance-central
  10. CBS Interactive: Just dance for Wii reviews - Metacritic (2017b). http://www.metacritic.com/game/wii/just-dance
  11. Champy, A.: Elements of motion: 3D sensors in intuitive game design. Analog. Dialogue. 41(2), 11–14 (2007). http://www.analog.com/en/analog-dialogue/articles/3d-sensors-in-intuitive-game-design.html
  12. Collins, K.: Game Sound : An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press (2008). https://mitpress.mit.edu/books/game-sound
  13. Cook, P.: Principles for designing computer music controllers. In: Proceedings of the International Conference on New Interfaces for Musical Expression 2001 (NIME 2001), pp. 3–6 (2001). http://www.nime.org/proceedings/2001/nime2001{\_}003.pdf. http://dl.acm.org/citation.cfm?id=1085152.1085154
  14. Die Gute Fabrik: Presskit — Johann Sebastian Joust (2014). http://www.jsjoust.com/presskit/
  15. George, R.: Behind the Scenes of Zelda: Skyward Sword (2011). http://uk.ign.com/articles/2011/11/11/behind-the-scenes-of-zelda-skyward-sword
  16. Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., Scherer, K.: Toward a minimal representation of affective gestures. IEEE Trans. Affect. Comput. 2(2), 106–118 (2011). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5740837. http://ieeexplore.ieee.org/document/5740837/
  17. Godøy, R.I., Leman, M. (eds.): Musical Gestures: Sound, Movement, and Meaning. Routledge (2010)Google Scholar
  18. Google ATAP: Project soli (2017). https://atap.google.com/soli/
  19. Halston, T.: New PS4 camera vs old PS4 camera comparison what’s the difference? (2016). https://www.psu.com/news/31081/New-PS4-camera-vs-old-PS4-camera-comparison---whats-the-difference
  20. Holz, D.: Leap motion sets a course for VR (2014). http://blog.leapmotion.com/leap-motion-sets-a-course-for-vr/
  21. Hunt, A.D., Wanderley, M.M., Paradis, M.: The importance of parameter mapping in electronic instrument design. In: Casey, C., Schneider, K., Hammond, E. (eds.) Proceedings of the 2002 Conference on New Instruments for Musical Expression (NIME-02), pp. 88–93, Dublin (2002). http://www.nime.org/proceedings/2002/nime2002{\_}088.pdf
  22. Jensenius, A.R.: To gesture or Not? An Analysis of Terminology in NIME Proceedings 2001–2013. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 217–220 (2014). http://www.nime.org/proceedings/2014/nime2014{\_}351.pdf
  23. Jensenius, A.R., Lyons, M.J.: A NIME Reader, Current Research in Systematic Musicology, vol. 3. Springer, Cham (2017). http://link.springer.com/10.1007/978-3-319-47214-0
  24. Kuchera, B.: Nintendo Switch’s World of Goo shows off system’s Wii-style pointer controls (2017). https://www.polygon.com/2017/3/16/14947750/nintendo-switch-world-of-goo-controls-wii
  25. Larboulette, C., Gibet, S.: A review of computable expressive descriptors of human motion. In: Proceedings of the 2nd International Workshop on Movement and Computing - MOCO ‘15. pp. 21–28. ACM Press, New York, NY (2015). http://dl.acm.org/citation.cfm?id=2790994.2790998. http://dl.acm.org/citation.cfm?doid=2790994.2790998
  26. Lazzaro, N.: Why We Play Games: Four Keys to More Emotion Without Story. Technical Report, XEODesign Inc., Oakland, CA (2004)Google Scholar
  27. Piana, S., Stagliano, A., Camurri, A., Odone, F.: A set of full-body movement features for’ emotion recognition to help children affected by autism spectrum condition. In: IDGEI International Workshop, Crete (2013)Google Scholar
  28. Piana, S., Stagliano, A., Odone, F., Camurri, A.: Adaptive body gesture representation for’ automatic emotion recognition. ACM Trans. Interact. Intell. Syst. 6(1), 1–31 (2016). http://dl.acm.org/citation.cfm?id=2896319.2818740
  29. Rovan, J.B., Wanderley, M.M., Dubnov, S., Depalle, P.: Instrumental gestural mapping strategies as expressivity determinants in computer music performance. In: Proceedings of the AIMI International Workshop, pp. 68–73. Kansei-The Technology of Emotion Workshop (1997)Google Scholar
  30. Serafin, S., Erkut, C., Kojs, J., Nilsson, N.C., Nordahl, R.: Virtual reality musical instruments: state of the art, design principles, and future directions. Comput. Music. J. 40(3), 22–40 (2016). http://www.mitpressjournals.org/doi/10.1162/COMJ{\_}a{\_}00372
  31. Sixense Entertainment: Razer hydra (2011). http://sixense.com/razerhydra-3
  32. STMicroelectronics: Semiconductor solutions from STMicroelectronics selected by Nintendo for Nintendo switch (2017). http://www.st.com/content/st{\_}com/en/about/media-center/press-item.html/t3934.html
  33. Takahashi, D.: Nintendo Switch has high-tech Joy-Con controllers with motion detectioncamera (2017). https://venturebeat.com/2017/01/12/nintendo-switch-has-high-tech-joy-con-controllers-with-motion-detection/
  34. Trojaniello, D., Cereatti, A., Pelosin, E., Avanzino, L., Mirelman, A., Hausdorff, J.M., Della Croce, U.: Estimation of step-by-step spatio-temporal parameters of normal and impaired gait using shank-mounted magneto-inertial sensors: application to elderly, hemiparetic, parkinsonian and choreic gait. J. Neuroeng. Rehabil. 11(1), 152 (2014). http://jneuroengrehab.biomedcentral.com/articles/10.1186/1743-0003-11-152. http://www.jneuroengrehab.com/content/11/1/152
  35. Van Nort, D., Wanderley, M.M., Depalle, P.: Mapping control structures for sound synthesis: functional and topological perspectives. Comput. Music J. 38(3), 6–22 (2014)Google Scholar
  36. Visi, F.: Methods and technologies for the analysis and interactive use of body movements in instrumental music performance (2017). https://pearl.plymouth.ac.uk/handle/10026.1/8805?show=full
  37. Visi, F., Schramm, R., Miranda, E.: Gesture in performance with traditional musical instruments and electronics. In: Proceedings of the 2014 International Workshop on Movement and Computing - MOCO ‘14. pp. 100–105. ACM Press, New York, NY (2014). http://dl.acm.org/citation.cfm?id=2618013. http://dl.acm.org/citation.cfm?doid=2617995.2618013
  38. Visi, F., Caramiaux, B., Mcloughlin, M., Miranda, E.: A knowledge-based, data-driven method for action-sound mapping. In: NIME’17 International Conference on New Interfaces for Musical Expression (2017a)Google Scholar
  39. Visi, F., Coorevits, E., Schramm, R., Miranda, E.R.: Musical instruments, body movement, space, and motion data: music as an emergent multimodal choreography. Hum. Technol. 13(1), 58–81 (2017b). http://humantechnology.jyu.fi/archive/vol-13/issue-1/musical-instruments-body-movement-space-and-motion-data/@@display-file/fullPaper/Visi{\_}Coorevits{\_}Schramm{\_}Reck-Miranda.pdf
  40. Visi, F., Georgiou, T., Holland, S., Pinzone, O., Donaldson, G., Tetley, J.: Assessing the accuracy of an algorithm for the estimation of spatial gait parameters using inertial measurement units: application to healthy subject and hemiparetic stroke survivor. In: 4th International Conference on Movement Computing (2017c)Google Scholar
  41. Wasenmuller, O., Stricker, D.: Comparison of kinect v1 and v2 depth images in terms of¨ accuracy and precision. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10117 LNCS, pp. 34–45. Springer, Cham (2017). http://link.springer.com/10.1007/978-3-319-54427-4{\_}3
  42. Wattanasoontorn, V., Boada, I., Garcia, R., Sbert, M.: Serious games for health. Entertain. Comput. 4(4), 231–247 (2013). http://linkinghub.elsevier.com/retrieve/pii/S1875952113000153
  43. Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the leap motion controller. Sensors. 13(5), 6380–6393 (2013). http://www.ncbi.nlm.nih.gov/pubmed/23673678. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3690061. http://www.mdpi.com/1424-8220/13/5/6380/
  44. Wessel, D., Wright, M.: Problems and prospects for intimate musical control of computers. In: Poupyrev, I., Lyons, M.J., Fels, S.S., Blaine, T. (eds.) Proceedings of the 2001 International Conference on New Interfaces for Musical Expression (NIME-01), pp. 11–14, Seattle, WA (2001). http://www.nime.org/proceedings/2001/nime2001{\_}011.pdf
  45. Wöllner, C.: Body, Sound and Space in Music and Beyond: Multimodal Explorations. Routledge (2017)Google Scholar
  46. Yang, L.: Knuckles cap sense overview (2017a). http://steamcommunity.com/sharedfiles/filedetails/?id=943495896
  47. Zamborlin, B., Bevilacqua, F., Gillies, M., D’inverno, M.: Fluid gesture interaction design. ACM Trans. Interact. Intell. Syst. 3(4), 1–30 (2014). http://dl.acm.org/citation.cfm?id=2543921. http://research.gold.ac.uk/9619/1/TiiS2013.pdf. http://dl.acm.org/citation.cfm?doid=2567808.2543921
  48. Zappi, V., McPherson, A.: Dimensionality and appropriation in digital musical instrument design. In: Caramiaux, B., Tahiroglu, K., Fiebrink, R., Tanaka, A. (eds.) Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 455–460. Goldsmiths, University of London, London (2014). http://www.nime.org/proceedings/2014/nime2014{\_}409.pdf

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Universität HamburgHamburgGermany

Personalised recommendations