Motion Controllers, Sound, and Music in Video Games: State of the Art and Research Perspectives

Part of the International Series on Computer Entertainment and Media Technology book series (ISCEMT)


This chapter is dedicated to the use of motion-sensing controllers in video games, with particular attention to gameplay mechanics that link body movement, sound, an music. First, we present a review of the state of the art. This includes an overview of recent motion controllers describing the different motion-sensing technologies employed and the implications such technologies have on game design and gameplay. This is followed by a set of examples of relationships between motion control, sound, music, and gameplay mechanics in recent video games. Secondly, we present a survey on recent research in expressive movement analysis and motion-based interaction, introducing the concepts of motion descriptor and parameter mapping. We then report on two serious games designed for children affected by autism using full-body motion capture and emotion recognition, and studies of player engagement in motion-controlled video games. In light of the interdisciplinary research presented, we finally discuss perspectives for motion-based game design.


  1. Alaoui, S.F., Caramiaux, B., Serrano, M., Bevilacqua, F.: Movement qualities as interaction modality. In: Proceedings of the Designing Interactive Systems Conference, pp. 761–769. ACM (2012)Google Scholar
  2. Bernardo, F., Arner, N., Batchelor, P.: O Soli Mio : exploring millimeter wave radar for musical interaction. In: Erkut, C. (ed.) NIME’17 International Conference on New Interfaces for Musical Expression, pp. 283–286, Copenhagen (2017).
  3. Bianchi-Berthouze, N.: Understanding the role of body movement in player engagement. Hum. Comp. Interact. 28(1), 40–75 (2013).
  4. Brown, D., Renney, N., Stark, A., Nash, C., Mitchell, T.: Leimu: gloveless music interaction using a wrist mounted leap motion. Proc. Int. Conf. New Interf. Mus. Expr. 16, 300–304 (2016).{\_}paper0059.pdf
  5. Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2, 1–43 (1998)CrossRefGoogle Scholar
  6. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) Gesture-based Communication in Human-Computer Interaction, pp. 20–39, vol. 2915, Springer, Berlin (2004).{\_}3
  7. Camurri, A., Coletta, P., Demurtas, M., Peri, M., Ricci, A., Sagoleo, R., Simonetti, M., Varni, G., Volpe, G.: A platform for real-time multimodal processing. In: Proceedings International Conference Sound and Music Computing, pp. 11–13 (2007).
  8. Caramiaux, B., Françoise, J., Schnell, N., Bevilacqua, F.: Mapping through listening. Comput. Music J. 38(3), 34–48 (2014)Google Scholar
  9. CBS Interactive: Dance central for Xbox 360 reviews - Metacritic (2017a).
  10. CBS Interactive: Just dance for Wii reviews - Metacritic (2017b).
  11. Champy, A.: Elements of motion: 3D sensors in intuitive game design. Analog. Dialogue. 41(2), 11–14 (2007).
  12. Collins, K.: Game Sound : An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press (2008).
  13. Cook, P.: Principles for designing computer music controllers. In: Proceedings of the International Conference on New Interfaces for Musical Expression 2001 (NIME 2001), pp. 3–6 (2001).{\_}003.pdf.
  14. Die Gute Fabrik: Presskit — Johann Sebastian Joust (2014).
  15. George, R.: Behind the Scenes of Zelda: Skyward Sword (2011).
  16. Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., Scherer, K.: Toward a minimal representation of affective gestures. IEEE Trans. Affect. Comput. 2(2), 106–118 (2011).
  17. Godøy, R.I., Leman, M. (eds.): Musical Gestures: Sound, Movement, and Meaning. Routledge (2010)Google Scholar
  18. Google ATAP: Project soli (2017).
  19. Halston, T.: New PS4 camera vs old PS4 camera comparison what’s the difference? (2016).
  20. Holz, D.: Leap motion sets a course for VR (2014).
  21. Hunt, A.D., Wanderley, M.M., Paradis, M.: The importance of parameter mapping in electronic instrument design. In: Casey, C., Schneider, K., Hammond, E. (eds.) Proceedings of the 2002 Conference on New Instruments for Musical Expression (NIME-02), pp. 88–93, Dublin (2002).{\_}088.pdf
  22. Jensenius, A.R.: To gesture or Not? An Analysis of Terminology in NIME Proceedings 2001–2013. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 217–220 (2014).{\_}351.pdf
  23. Jensenius, A.R., Lyons, M.J.: A NIME Reader, Current Research in Systematic Musicology, vol. 3. Springer, Cham (2017).
  24. Kuchera, B.: Nintendo Switch’s World of Goo shows off system’s Wii-style pointer controls (2017).
  25. Larboulette, C., Gibet, S.: A review of computable expressive descriptors of human motion. In: Proceedings of the 2nd International Workshop on Movement and Computing - MOCO ‘15. pp. 21–28. ACM Press, New York, NY (2015).
  26. Lazzaro, N.: Why We Play Games: Four Keys to More Emotion Without Story. Technical Report, XEODesign Inc., Oakland, CA (2004)Google Scholar
  27. Piana, S., Stagliano, A., Camurri, A., Odone, F.: A set of full-body movement features for’ emotion recognition to help children affected by autism spectrum condition. In: IDGEI International Workshop, Crete (2013)Google Scholar
  28. Piana, S., Stagliano, A., Odone, F., Camurri, A.: Adaptive body gesture representation for’ automatic emotion recognition. ACM Trans. Interact. Intell. Syst. 6(1), 1–31 (2016).
  29. Rovan, J.B., Wanderley, M.M., Dubnov, S., Depalle, P.: Instrumental gestural mapping strategies as expressivity determinants in computer music performance. In: Proceedings of the AIMI International Workshop, pp. 68–73. Kansei-The Technology of Emotion Workshop (1997)Google Scholar
  30. Serafin, S., Erkut, C., Kojs, J., Nilsson, N.C., Nordahl, R.: Virtual reality musical instruments: state of the art, design principles, and future directions. Comput. Music. J. 40(3), 22–40 (2016).{\_}a{\_}00372
  31. Sixense Entertainment: Razer hydra (2011).
  32. STMicroelectronics: Semiconductor solutions from STMicroelectronics selected by Nintendo for Nintendo switch (2017).{\_}com/en/about/media-center/press-item.html/t3934.html
  33. Takahashi, D.: Nintendo Switch has high-tech Joy-Con controllers with motion detectioncamera (2017).
  34. Trojaniello, D., Cereatti, A., Pelosin, E., Avanzino, L., Mirelman, A., Hausdorff, J.M., Della Croce, U.: Estimation of step-by-step spatio-temporal parameters of normal and impaired gait using shank-mounted magneto-inertial sensors: application to elderly, hemiparetic, parkinsonian and choreic gait. J. Neuroeng. Rehabil. 11(1), 152 (2014).
  35. Van Nort, D., Wanderley, M.M., Depalle, P.: Mapping control structures for sound synthesis: functional and topological perspectives. Comput. Music J. 38(3), 6–22 (2014)Google Scholar
  36. Visi, F.: Methods and technologies for the analysis and interactive use of body movements in instrumental music performance (2017).
  37. Visi, F., Schramm, R., Miranda, E.: Gesture in performance with traditional musical instruments and electronics. In: Proceedings of the 2014 International Workshop on Movement and Computing - MOCO ‘14. pp. 100–105. ACM Press, New York, NY (2014).
  38. Visi, F., Caramiaux, B., Mcloughlin, M., Miranda, E.: A knowledge-based, data-driven method for action-sound mapping. In: NIME’17 International Conference on New Interfaces for Musical Expression (2017a)Google Scholar
  39. Visi, F., Coorevits, E., Schramm, R., Miranda, E.R.: Musical instruments, body movement, space, and motion data: music as an emergent multimodal choreography. Hum. Technol. 13(1), 58–81 (2017b).{\_}Coorevits{\_}Schramm{\_}Reck-Miranda.pdf
  40. Visi, F., Georgiou, T., Holland, S., Pinzone, O., Donaldson, G., Tetley, J.: Assessing the accuracy of an algorithm for the estimation of spatial gait parameters using inertial measurement units: application to healthy subject and hemiparetic stroke survivor. In: 4th International Conference on Movement Computing (2017c)Google Scholar
  41. Wasenmuller, O., Stricker, D.: Comparison of kinect v1 and v2 depth images in terms of¨ accuracy and precision. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10117 LNCS, pp. 34–45. Springer, Cham (2017).{\_}3
  42. Wattanasoontorn, V., Boada, I., Garcia, R., Sbert, M.: Serious games for health. Entertain. Comput. 4(4), 231–247 (2013).
  43. Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the leap motion controller. Sensors. 13(5), 6380–6393 (2013).
  44. Wessel, D., Wright, M.: Problems and prospects for intimate musical control of computers. In: Poupyrev, I., Lyons, M.J., Fels, S.S., Blaine, T. (eds.) Proceedings of the 2001 International Conference on New Interfaces for Musical Expression (NIME-01), pp. 11–14, Seattle, WA (2001).{\_}011.pdf
  45. Wöllner, C.: Body, Sound and Space in Music and Beyond: Multimodal Explorations. Routledge (2017)Google Scholar
  46. Yang, L.: Knuckles cap sense overview (2017a).
  47. Zamborlin, B., Bevilacqua, F., Gillies, M., D’inverno, M.: Fluid gesture interaction design. ACM Trans. Interact. Intell. Syst. 3(4), 1–30 (2014).
  48. Zappi, V., McPherson, A.: Dimensionality and appropriation in digital musical instrument design. In: Caramiaux, B., Tahiroglu, K., Fiebrink, R., Tanaka, A. (eds.) Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 455–460. Goldsmiths, University of London, London (2014).{\_}409.pdf

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Universität HamburgHamburgGermany

Personalised recommendations