We propose an environment for assisting in the transfer of motor skills by supporting nonverbal communication with wearable haptic technology. The proposed setting comprises a human expert and several learners, all endowed with wearable devices. Learners benefit from the automated feedback obtained via wearable technology regarding a desired move relevant to a specific motor skill. Similarly, the expert benefits from automated means to provide instructions to the learner as well as from monitoring data provided by wearable devices. We focus on tactile communication as the basis for this interaction, thus supporting users who are not able to rely on other senses that are relevant to most motor skill-related tasks, such as sight and hearing. We discuss leader-follower dance as a motor skill context in order to instantiate and validate critical components of our skill transfer environment, using accelerometers and gyroscopes as the basis for sensing movements. In this particular scenario, our approach focuses on the haptic communication elements needed to perform a leader-follower dance, assessing movements by the learners in response to the intent of the expert. In order to validate our approach, we have prototyped and conducted experiments with the three critical components of our environment that enables tactile communication between experts and learners. Our development is informed by user research conducted with expert dance teachers, through which we gathered expert dance moves that resulted in a reference database that makes up one of the components that are critical to demonstrate the potential of our proposed environment. We also implemented a tactile encoder that is used to send movement instructions to learners through the sense of touch, as well as an autonomous classifier that uses the reference database as input. Overall, our results provide a roadmap for building an environment for skill transfer using nonverbal communication based on haptic wearable devices.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Grosshauser T, Hermann T (2009) Augmented haptics–an interactive feedback system for musicians. In International Conference on Haptic and Audio Interaction Design. Springer, Heidelberg, pp 100–108
Fogtmann MH, Fritsch J, Kortbek KJ (2008) Kinesthetic interaction: revealing the bodily potential in interaction design. In Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat (pp. 89-96). ACM
Ludvigsen M, Fogtmann MH, Grønbæk K (2010) TacTowers: an interactive training equipment for elite athletes. In Proceedings of the 8th ACM Conference on Designing Interactive Systems. ACM, pp 412-415
Grønbæk K, Iversen OS, Kortbek KJ, Nielsen KR, Aagaard L (2007) Interactive floor support for kinesthetic interaction in children learning environments. In: IFIP Conference on Human-Computer Interaction. Springer, Heidelberg, pp 361–375
Yokokohji Yasuyoshi, Hollis Ralph L., Kanade Takeo, Henmi Kazuyuki, and Yoshikawa Tsuneo (1996) Toward machine mediated training of motor skills. Skill transfer from human to human via virtual environment. Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN’96 TSUKUBA. IEEE
Chellali A, Dumas C, Milleville-Pennel I (2010) WYFIWIF: a haptic communication paradigm for collaborative motor skills learning. In Web Virtual Reality and Three-Dimensional Worlds 2010. IADIS, pp 301-308
Asadipour A (2020) A technology-aided multi-modal training approach to assist abdominal palpation training and its assessment in medical education. Int J Hum Comput Stud:102394
Lutz J, Memmert D, Raabe D, Dornberger R, Donath L (2020) Wearables for integrative performance and tactic analyses: opportunities, challenges, and future directions. Int J Environ Res Public Health 17(1):59
Georgiou T, Holland S, van der Linden J (2016) Wearable haptic devices for post-stroke gait rehabilitation. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, pp 1114-1119
Tsetserukou D (2010) HaptiHug: a novel haptic display for communication of hug over a distance. In: International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, Heidelberg, pp 340–347
Van Hooren B, Goudsmit J, Restrepo J, Vos S (2020) Real-time feedback by wearables in running: current approaches, challenges and suggestions for improvements. J Sports Sci 38(2):214–230
Jensen MM, Mueller FF (2014) Running with technology: where are we heading?. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design. ACM, pp 527-530
Yang Chenguang, Liang Peidong, Ajoudani Arash, Li Zhijun, Bicchi Antonio (2016) Development of a robotic teaching interface for human to human skill transfer. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE
Pedemonte N, Laliberté T, Gosselin C (2016) Bidirectional haptic communication: application to the teaching and improvement of handwriting capabilities. Machines 4(1):6
Furukawa M, Kajimoto H, Tachi S. (2012) KUSUGURI: a shared tactile interface for bidirectional tickling. In Proceedings of the 3rd Augmented Human International Conference. ACM, p 9
Damian I, André E (2016) Exploring the potential of realtime haptic feedback during social interactions. In Proceedings of the TEI’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, pp 410-416
Eaves DL, Breslin G, Van Schaik P, Robinson E, Spears IR (2011) The short-term effects of real-time virtual reality feedback on motor learning in dance. Presence Teleop Virt 20(1):62–77
Raina A, Lakshmi TG, Murthy S (2017) CoMBaT: wearable technology based training system for novice badminton players. In 2017 IEEE 17th international Conference on Advanced Learning Technologies (ICALT). IEEE, pp 153-157
Narazani Marla, Seaborn Katie, Hiyama Atsushi, Inami Masahiko (2018) StepSync: wearable skill transfer system for real-time foot-based interaction” Proceedings of the 23rd Annual Meeting of the Virtual Reality Society of Japan
Aggravi, Marco, Gionata Salvietti, Domenico Prattichizzo (2016) Haptic assistive bracelets for blind skier guidance.” Proceedings of the 7th Augmented Human International Conference 2016. ACM
Havlucu H, Eskenazi T, Akgün B, Onbaşlı MC, Coşkun A, Özcan O (2018) Flow state feedback through sports wearables: a case study on tennis. In Proceedings of the 2018 on Designing Interactive Systems Conference 2018. ACM, pp 1025-1039
Gardner, Howard (2005) Intelligence in seven steps. New Horizons For Learning, Creating the Future. Report retrieved December21 (2003)
Camarillo-Abad HM, Sánchez JA, Starostenko O (2019) Organizing knowledge on nonverbal communication mediated through haptic technology. In: Iberoamerican Workshop on Human-Computer Interaction. Springer, Cham, pp 269–283
Camarillo-Abad HM, Sanchez JA, Starostenko O, Sandoval MG (2019) A basic tactile language to support leader-follower dancing. J Intell Fuzzy Syst 36(5):5011–5022
Valsted Frederik Mørch, Nielsen Christopher VH, Jensen Jacob Qvist, Sonne Tobias, Jensen Mads Møller (2017) Strive: exploring assistive haptic feedback on the run. Proceedings of the 29th Australian Conference on Computer-Human Interaction. ACM
Matsuyama H, Hiroi K, Kaji K, Yonezawa T, Kawaguchi N (2019) Ballroom dance step type recognition by random forest using video and wearable sensor. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. pp 774-780
Guerreiro J, Sato D, Ahmetovic D, Ohn-Bar E, Kitani KM, Asakawa C (2020) Virtual navigation for blind people: transferring route knowledge to the real-world. Int J Hum Comput Stud 135:102369
Trajkova M (2019) Towards AI-enhanced ballet learning. In Proceedings of the 6th International Conference on Movement and Computing. pp 1-5
Levine D, Cheng A, Olaleye D, Leonardo K, Shifrin M, Ishii H (2019). AUFLIP-an auditory feedback system towards implicit learning of advanced motor skills. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. pp 1-6
Brewster, Stephen, Lorna M. Brown (2004) Tactons: structured tactile messages for non-visual information display. Proceedings of the fifth conference on Australasian user interface-Volume 28. Australian Computer Society, Inc.
Fogtmann MH, Grønbæk K, Ludvigsen MK (2011). Interaction technology for collective and psychomotor training in sports. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology. ACM, p 13
Parikh SP, Esposito JM, Searock J (2014) The role of verbal and nonverbal communication in a two-person, cooperative manipulation task. Adv Hum Comput Interact 2014:9
Gentry S, Murray-Smith R (2003) Haptic dancing: human performance at haptic decoding with a vocabulary. In SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483) Vol. 4. IEEE, pp 3432-3437
Barczewska K, Drozd A (2013) Comparison of methods for hand gesture recognition based on dynamic time warping algorithm. In 2013 Federated Conference on Computer Science and Information Systems. IEEE, pp 207-210
Yurtman A, Barshan B (2014) Automated evaluation of physical therapy exercises using multi-template dynamic time warping on wearable sensor signals. Comput Methods Prog Biomed 117(2):189–207
Chan JC, Leung H, Tang JK, Komura T (2010) A virtual reality dance training system using motion capture technology. IEEE Trans Learn Technol 4(2):187–195
Matsuyama H, Hiroi K, Kaji K, Yonezawa T, Kawaguchi N (2019) Hybrid activity recognition for ballroom dance exercise using video and wearable sensor. In 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, pp 112-117
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Camarillo-Abad, H.M., Sánchez, J.A. & Starostenko, O. An environment for motor skill transfer based on wearable haptic communication. Pers Ubiquit Comput (2020). https://doi.org/10.1007/s00779-020-01425-z
- Skill transfer
- Haptic communication
- Wearable technology
- Technology-mediated learning