Virtual Conversation with Real-Time Prediction of Body Moments/Gestures on Video Streaming Data

  • Gopichand Agnihotram
  • Rajesh Kumar
  • Pandurang Naik
  • Rahul YadavEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1101)


The exisitng conversation system where the user interacts with the virtual system with voice and virtual system replies to the user based on what user speaks. In this context whenever user makes some gestures to communicate with the virtual system, the virtual system will miss out those communications. For example, user instead of speaking, may nod head for “yes” or “no” and user can also use hand signals to respond to the virtual system. If these events are not addressed then the conversation is not very interactive and natural human-like interaction will start losing important information. The paper describes how the user body moments/gestures will help effective conversation with the virtual system and virtual conversation system can understand the user misspelled conversation, missed conversation effectively with user gesture/body movements.


Key point detection Gesture classification Events computation Virtual conversation system User conversation Feature extraction Real-time gesture prediction Convolutional neural networks (CNN) 


  1. 1.
    Kaushik, Manju, and Rashmi Jain. 2014. Natural user interfaces: Trend in virtual interaction. arXiv preprint arXiv: 1405.0101.Google Scholar
  2. 2.
    Eshed, Ohn-Bar, and Mohan Manubhai Trivedi. 2014. Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations. IEEE Transactions on Intelligent Transportation Systems 15 (6): 2368–2377.CrossRefGoogle Scholar
  3. 3.
    Molchanov, Pavlo, Shalini Gupta, Kihwan Kim, and Jan Kautz. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1–7.Google Scholar
  4. 4.
    Wu, Erwin, and Hideki Koike. 2018. Real-time human motion forecasting using a RGB camera. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. ACM.Google Scholar
  5. 5.
    Badler, I. Norman. 1997. Real-time virtual humans. In Proceedings of the Fifth Pacific Conference on Computer Graphics and Applications. IEEE.Google Scholar
  6. 6.
    Kuffner, J.James. 1998. Goal-directed navigation for animated characters using real-time path planning and control. International Workshop on Capture Techniques for Virtual Environments, 171–186. Berlin: Springer.CrossRefGoogle Scholar
  7. 7.
    Ng, Kia. 2004. Music via motion: Transdomain mapping of motion and sound for interactive performances. Proceedings of the IEEE 92 (4): 645–655.CrossRefGoogle Scholar
  8. 8.
    Nguyen, H. Katerina. 2001. Method and apparatus for real-time gesture recognition. U.S. Patent No. 6,256,033, 3 July 2001.Google Scholar
  9. 9.
    Zhou Jie, and Pu Cheng. 2016. System and method for gesture recognition. U.S. Patent No. 9,323,337, 26 Apr 2016.Google Scholar
  10. 10.
    Huang, Kuang-Man, Ming-Chang Liu, and Liangyin Yu. 2013. System and method for dynamic gesture recognition using geometric classification. U.S. Patent No. 8,620,024, 31 Dec 2013.Google Scholar
  11. 11.
    Smith Dana, S. 2014. Geometric shape generation using multi-stage gesture recognition. U.S. Patent Application 13/846,469, filed 18 Sept 2014.Google Scholar
  12. 12.
    Kurakin, Alexey, Zhengyou Zhang, and Zicheng Liu. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO). IEEE.Google Scholar
  13. 13.
    Chen, L., H. Wei, and J. Ferryman. 2013. A survey of human motion analysis using depth imagery. Pattern Recognition Letters 34 (15): 1995–2006.CrossRefGoogle Scholar
  14. 14.
    Marin, Giulio, Fabio Dominio, and Pietro Zanuttigh. 2014. Hand gesture recognition with leap motion and kinect devices. In 2014 IEEE International Conference on Image Processing (ICIP). IEEE.Google Scholar
  15. 15.
    Shan, Caifeng, Tieniu Tan, and Yucheng Wei. 2007. Real-time hand tracking using a mean shift embedded particle filter. Pattern Recognition 40 (7): 1958–1970.CrossRefGoogle Scholar
  16. 16.
    Redmon, Joseph, and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv: 1804.02767.Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Gopichand Agnihotram
    • 1
  • Rajesh Kumar
    • 1
  • Pandurang Naik
    • 1
  • Rahul Yadav
    • 1
    Email author
  1. 1.Wipro CTO Office, Wipro Technology LimitedBangaloreIndia

Personalised recommendations