Abstract
With the lightning speed of technological evolution, several methods have been proposed with the aim of controlling robots and using them to serve humanity. In this work, we present and evaluate a novel learning-based system to control Pepper, the humanoid robot. We leveraged an existing low-cost surface electromyography (sEMG) sensor, that is in the consumer market, Myo armband. To achieve our goal, we created a dataset including 6 hand gestures recorded from 35 intact people by the usage of the Myo Armband device, which has 8 non-intrusive sEMG sensors. Using raw signals extracted from Myo armband, we have been able to train a gated recurrent unit-based network to perform gesture classification. Afterwards, we integrated our system with a live hand gesture recognition application, transmitting the commands to the robot for implementing a live teleoperation method. In this way, we are able to evaluate in real-time the capabilities of our system. According to the experiments, the teleoperation of a Pepper robot achieved an average of 77.5% accuracy during test.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cook, A.M., Polgar, J.M.: Essentials of Assistive Technologies. ELSEVIER Mosby (2012)
Costa, A., Martinez-Martin, E., Cazorla, M., Julian, V.: Pharos-physical assistant robot system. Sensors 18(8), 2633 (2018)
Kowalczuk, Z., Czubenko, M.: Model of human psychology for controlling autonomous robots. In: 2010 15th International Conference on Methods and Models in Automation and Robotics, pp. 31–36, August 2010
Li, M., Li, W., Zhao, J., Meng, Q., Sun, F., Chen, G.: An adaptive P300 model for controlling a humanoid robot with mind. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1390–1395, December 2013
Lamiroy, B., Espiau, B., Andreff, N., Horaud, R.: Controlling robots with two cameras: how to do it properly. In: IEEE International Conference on Robotics and Automation (ICRA 2000), San Francisco, USA, pp. 2100–2105. IEEE Computer Society, April 2000
Allard, U.C., et al.: A convolutional neural network for robotic arm guidance using sEMG based frequency-features. In: Intelligent Robots and Systems (IROS). IEEE 2016, pp. 2464–2470 (2016)
Kucukyildiz, G., Ocak, H., Karakaya, S., Sayli, O.: Design and implementation of a multi sensor based brain computer interface for a robotic wheelchair. J. Intell. Robot. Syst. 87, 247–263 (2017)
Shin, S., Kim, D., Seo, Y.: Controlling mobile robot using IMU and EMG sensor-based gesture recognition. In: 2014 Ninth International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA), pp. 554–557, November 2015
Bisi, S., De Luca, L., Shrestha, B., Yang, Z., Gandhi, V.: Development of an EMG-controlled mobile robot. Robotics 7, 36 (2018)
Wang, H.-B., Liu, M.: Design of robotic visual servo control based on neural network and genetic algorithm. Int. J. Autom. Comput. 9, 24–29 (2012)
Stanton, C., Bogdanovych, A., Ratanasena, E.: Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning, December 2012
Morris, A.S., Mansor, A.: Finding the inverse kinematics of manipulator arm using artificial neural network with lookup table. Robotica 15, 617–625 (1997)
Yang, C., Chang, S., Liang, P., Li, Z., Su, C.-Y.: Teleoperated robot writing using EMG signals. In: 2015 IEEE International Conference on Information and Automation, pp. 2264–2269 (2015)
Reddivari, H., Yang, C., Ju, Z., Liang, P., Li, Z., Xu, B.: Teleoperation control of Baxter robot using body motion tracking. In: 2014 International Conference on Multisensor Fusion and Information Integration for Intelligent Systems (MFI), pp. 1–6, September 2014
Xu, Y., Yang, C., Liang, P., Zhao, L., Li, Z.: Development of a hybrid motion capture method using MYO armband with application to teleoperation. In: 2016 IEEE International Conference on Mechatronics and Automation, pp. 1179–1184, August 2016
Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R.: Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 515–524. ACM, New York (2008)
Rojas-Martinez, M., Manyanas, M., Alonso, J., Merletti, R.: Identification of isometric contractions based on high density EMG maps. Electromyogr. Kinesiol. 23, 33–42 (2013)
Zhang, X., Zhou, P.: High-density myoelectric pattern recognition toward improved stroke rehabilitation. IEEE Trans. Biomed. Eng. 59, 1649–1657 (2012)
Atzori, M., Cognolato, M., Müller, H.: Deep learning with convolutional neural networks applied to electromyography data: a resource for the classification of movements for prosthetic hands. Front. Neurorobot. 10, 9 (2016)
Geng, W., Du, Y., Jin, W., Wei, W., Hu, Y., Li, J.: Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6, 36571 (2016)
Wang, K.-J., Tung, H.-W., Huang, Z., Thakur, P., Mao, Z.-H., You, M.-X.: EXGbuds: universal wearable assistive device for disabled people to interact with the environment seamlessly, pp. 369–370, March 2018
Nasri, N., Orts-Escolano, S., Gomez-Donoso, F., Cazorla, M.: Inferring static hand poses from a low-cost non-intrusive sEMG sensor. Sensors 19(2), 371 (2019)
Bauer, Z., Escalona, F., Cruz, E., Cazorla, M., Gomez-Donoso, F.: Improving 3D estimation for the pepper robot using monocular depth prediction. In: Workshop de Agentes Físicos (WAF) (2018)
Cruz, E., et al.: Geoffrey: an automated schedule system on a social robot for the intellectually challenged. Comput. Intell. Neurosci. 2018, 17 (2018)
Pomboza-Junez, G., Terriza, J.H.: Hand gesture recognition based on sEMG signals using support vector machines. In: Consumer Electronics-Berlin (2016)
Allard, U.C., et al.: Deep learning for electromyographic hand gesture signal classification by leveraging transfer learning. CoRR, vol. abs/1801.07756 (2018)
Cote-Allard, U., Fall, C.L., Campeau-Lecours, A., Gosselin, C., Laviolette, F., Gosselin, B.: Transfer learning for sEMG hand gestures recognition using convolutional neural networks. In: IEEE International Conference on Systems (2017)
Pizzolato, S., Tagliapietra, L., Cognolato, M., Reggiani, M., Muller, H., Atzori, M.: Comparison of six electromyography acquisition setups on hand movement classification tasks. PloS One 12, e0186132 (2017)
Farina, D., Cescon, C., Merletti, R.: Influence of anatomical, physical, and detection-system parameters on surface emg. Biol. Cybern. 86, 445–456 (2002)
Kuiken, T.A., Lowery, M.M., Stoykov, N.S.: The effect of subcutaneous fat on myoelectric signal amplitude and cross-talk. Prosthet. Orthot. Int. 27(1), 48–54 (2003). PMID: 12812327
Acknowledgements
This work was supported by the Spanish Government TIN2016-76515R grant, supported with Feder funds. It has also been funded by the University of Alicante project GRE16-19, by the Valencian Government project GV/2018/022, and by a Spanish grant for PhD studies ACIF/2017/243. The authors would like to thank all the subjects for their participation in our experiments. We would also like to thank NVIDIA (Santa Clara, California, USA) for the generous donation of a Titan Xp and a Quadro P6000.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Nasri, N., Gomez-Donoso, F., Orts-Escolano, S., Cazorla, M. (2019). Using Inferred Gestures from sEMG Signal to Teleoperate a Domestic Robot for the Disabled. In: Rojas, I., Joya, G., Catala, A. (eds) Advances in Computational Intelligence. IWANN 2019. Lecture Notes in Computer Science(), vol 11507. Springer, Cham. https://doi.org/10.1007/978-3-030-20518-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-20518-8_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20517-1
Online ISBN: 978-3-030-20518-8
eBook Packages: Computer ScienceComputer Science (R0)