Robust Hand Gesture Recognition Using Multimodal Deep Learning for Touchless Visualization of 3D Medical Images

  • Kotaro Furusawa
  • Jiaqing Liu
  • Seiju Tsujinaga
  • Tomoko Tateyama
  • Yutaro Iwamoto
  • Yen-Wei ChenEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)


Three-dimensional (3D) visualization of medical images is an important technology for efficiently conducting a surgery. However, efficient review of 3D anatomical models is required to maintain sterile field conditions. An operation using touchless interface for gesture recognition is one of the review methods. Real-time hand gesture application for supporting a surgery requires a robust recognition of various gestures. This study proposes a robust hand gesture recognition using multimodal deep learning to perform recognition using color and depth images. We evaluated the recognition accuracy of 25 different gestures and compared its recognition accuracy with conventional recognition methods. Resultantly, it was found that the proposed system achieves better real-time robust recognition than conventional methods.


Hand gesture recognition Multimodal deep learning Surgery aid system 



Authors would like to thank Dr. M. Kaibori of KANSAI Medical University for providing medical images and advice on surgical support systems. This work is supported in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant Nos. 18H03267, 18K11454, 17H00754, 17K00420; and in part by Zhejiang Lab Program under the Grant No. 2018DG0ZX01.


  1. 1.
    Kaibori, M., Chen, Y.W., Matsui, K., Ishizaki, M., Tsuda, T., Nakatake, R., Sakaguchi, T., Matsushima, H., Miyawaki, K., Shindo, T., Tateyama, T., Kwon, A.H.: Novel liver visualization and surgical simulation system. J. Gastrointest. Surg. 17, 1422–1428 (2013)CrossRefGoogle Scholar
  2. 2.
    Tateyama, T., Kaibori, M., Chen, Y.W., et al.: Patient-specified 3D-visualization for liver and vascular structures and interactive surgical planning system. Med. Imaging Technol. 31, 176–188 (2013). (in Japanese)Google Scholar
  3. 3.
    Nasr-Esfahani, E., et al.: Hand gesture recognition for contactless device control in operating rooms. arXiv Preprint arXiv1611.04138 (2016)Google Scholar
  4. 4.
    Gallo, L.: Controller-free exploration of medical image data: experiencing the Kinect, National Research Council of Italy Institute for High Performance Computing and Networking (2011)Google Scholar
  5. 5.
    Yoshimitsu, K., Muragaki, Y., Iseki, H., et al.: Development and initial clinical testing of “OPECT”: an innovative device for fully intangible control of the intraoperative image-displaying monitor by the surgeon. Neurosurgery 10(Suppl. 1), 46–50 (2014)Google Scholar
  6. 6.
    Ruppert, G.C., Coares, C., Lopes, V., et al.: Touchless gesture user interface for interactive image visualization in urological surgery. World J. Urol. 30, 687–691 (2012)CrossRefGoogle Scholar
  7. 7.
    Mewes, A., Hensen, B., Wacker, F., et al.: Touchless interaction with software in interventional radiology and surgery: a systematic literature review. Int. J. CARS 12, 291 (2017)CrossRefGoogle Scholar
  8. 8.
    Liu, J.Q., Fujii, R., Tateyama, T., Iwamoto, Y., Chen, Y.W.: Kinect-based gesture recognition for touchless visualization of medical images. Int. J. Comput. Electr. Eng. 9(2), 421–429 (2017)CrossRefGoogle Scholar
  9. 9.
    Liu, J.Q., Tateyama, T., Iwamoto, Y., Chen, Y.W.: A preliminary study of kinect-based real-time hand gesture interaction systems for touchless visualizations of hepatic structures in surgery. J. Med. Imaging Inf. Sci. (2019, in press)Google Scholar
  10. 10.
    Liu, J.Q., Tateyama, T., Iwamoto, Y., Chen, Y.W.: Kinect-based real-time gesture recognition using deep convolutional neural networks for touchless visualization of hepatic anatomical models in surgery. In: KES-IIMSS-18 2018. Smart Innovation, Systems and Technologies, vol. 98. Springer, Cham (2018)Google Scholar
  11. 11.
    Liu, J.Q., Furusawa, K., Tsujinaga, S., Tateyama, T., Iwamoto, Y., Chen, Y.W.: An improved kinect-based real-time gesture recognition using deep convolutional neural networks for touchless visualization of hepatic anatomical models in surgery. J. Image Graph. (2019, in press)Google Scholar
  12. 12.
    Krizhevsky, A., Sutsjever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, p. 9 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Kotaro Furusawa
    • 1
  • Jiaqing Liu
    • 1
  • Seiju Tsujinaga
    • 1
  • Tomoko Tateyama
    • 2
  • Yutaro Iwamoto
    • 1
  • Yen-Wei Chen
    • 1
    • 3
    • 4
    Email author
  1. 1.Ritsumeikan UniversityKusatsuJapan
  2. 2.Hiroshima Institute of TechnologyHiroshimaJapan
  3. 3.Zhejiang LabHangzhouChina
  4. 4.College of Computer Science and TechnologyZhejiang UniversityHangzhouChina

Personalised recommendations