Kinect-Based Real-Time Gesture Recognition Using Deep Convolutional Neural Networks for Touchless Visualization of Hepatic Anatomical Models in Surgery

  • Jia-Qing Liu
  • Tomoko Tateyama
  • Yutaro Iwamoto
  • Yen-Wei ChenEmail author
Conference paper
Part of the Smart Innovation, Systems and Technologies book series (SIST, volume 98)


In this paper, we present a novel touchless interaction system for visualization of hepatic anatomical models in surgery. Real-time visualization is important in surgery, particularly during the operation. However, it often faces the challenge of efficiently reviewing the patient’s 3D anatomy model while maintaining a sterile field. The touchless technology is an attractive and potential solution to address the above problem. We use a Microsoft Kinect sensor as input device to produce depth images for extracting a hand without markers. Based on this representation, a deep convolutional neural network is used to recognize various hand gestures. Experimental results demonstrate that our system can significantly improve the response time while achieve almost same accuracy compared with the previous researches.


Deep learning Kinect Real-time touch-less interaction Visualization of hepatic anatomical models 



Authors would like to thank Dr. M. Kaibori of KANSAI Medical University for providing medical images and advice on surgical support systems. This work is supported in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant Nos. 16H01436, 15K16031, 17H00754, 17K00420, 18H03267; in part by the MEXT Support Program for the Strategic Research Foundation at Private Universities, Grant (2013–2017).


  1. 1.
    Kaibori, M., Chen, Y.W., Matsui, K., Ishizaki, M., Tsuda, T., Nakatake, R., Sakaguchi, T., Matsushima, H., Miyawaki, K., Shindo, T., Tateyama, T., Kwon, A.H.: Novel liver visualization and surgical simulation system. J. Gastrointest. Surg. 17, 1422–1428 (2013)CrossRefGoogle Scholar
  2. 2.
    Tateyama, T., Kaibori, M., Chen, Y.W., et al.: Patient-specified 3D-visualization for liver and vascular structures and interactive surgical planning system. Med. Imaging Technol. 31, 176–188 (2013). (in Japanese)Google Scholar
  3. 3.
    Gallo, L.: Controller-free exploration of medical image data: experiencing the Kinect. National Research Council of Italy Institute for High Performance Computing and Networking (2011)Google Scholar
  4. 4.
    Yoshimitsu, K., Muragaki, Y., Iseki, H., et al.: Development and initial clinical testing of “OPECT”: an innovative device for fully intangible control of the intraoperative image-displaying monitor by the surgeon. Neurosurgery 10(Suppl 1), 46–50 (2014)Google Scholar
  5. 5.
    Ruppert, G.C., Coares, C., Lopes, V., et al.: Touchless gesture user interface for interactive image visualization in urological surgery. World J. Urol. 30, 687–691 (2012)CrossRefGoogle Scholar
  6. 6.
    Mewes, A., Hensen, B., Wacker, F., et al.: Touchless interaction with software in interventional radiology and surgery: a systematic literature review. Int. J. CARS 12, 291 (2017)CrossRefGoogle Scholar
  7. 7.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: NIPS 2012 Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS), vol. 1, pp. 1097–1105 (2012)Google Scholar
  8. 8.
    Liu, J.Q., Fujii, R., Tateyama, T., Iwamoto, Y., Chen, Y.W.: Kinect-based gesture recognition for touchless visualization of medical images. Int. J. Comput. Electr. Eng. 9(2), 421–429 (2017)CrossRefGoogle Scholar
  9. 9.
    Liu, J.Q., et al.: A kinect-based real-time hand gesture interaction system for touchless visualization of hepatic structure in surgery. Med. Imaging Inf. Sci. (2018, submitted)Google Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2323 (1998)CrossRefGoogle Scholar
  11. 11.
    Intel RealSence Depth camera D400 series.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  • Jia-Qing Liu
    • 1
  • Tomoko Tateyama
    • 2
  • Yutaro Iwamoto
    • 1
  • Yen-Wei Chen
    • 1
    Email author
  1. 1.Information Science and EngineeringRitsumeikan UniversityKusatsuJapan
  2. 2.Department of Computer ScienceHiroshima Institute of TechnologyHiroshimaJapan

Personalised recommendations