Abstract
In the existing product virtual display system, the interactions are mainly operated by mouse and keyboard. Inevitably, the user experience is often interrupted and disturbed by the continually input and output actions. This paper proposes a multimodal virtual display system which is integrated with speech and gesture recognition technology. First, we established a digital automobile model and set the interactive actions of the automobile display in a Web3D platform. Next, the speech interaction was realized by the re-development of the Microsoft Speech SDK and the gesture recognition was accomplished by Leap Motion controller. At last, the whole system was achieved on a holographic experimental platform. The proposed system makes the interactions more intuitive, natural and comprehensive in product virtual display.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
da Silva Simões, P.D., Gabriel Inacio Ferreira, C.: Military war games edutainment. In: 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), vol. 7. IEEE (2011)
Rizo, A., Parsons, T.D., et al.: Virtual reality goes to war: a brief review of the future of military behavioral healthcare. J. Clin. Psychol. Med. Settings 18(2), 176–187 (2011)
Wu, X., Gao, F.: Research of virtual reality technology in automotive engine assembly teaching. In: 2011 6th IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC), vol. 1. IEEE (2011)
Barrett, M., Blackledge, J., Coyle, E.: Using virtual reality to enhance electrical safety and design in the built environment. ISAST Trans. Intell. Syst. 3, 1 (2011)
Suma, E.A., Lange, B., Rizzo, A., et al.: FAAST: the flexible action and articulated skeleton toolkit. In: 2011 IEEE Virtual Reality Conference (VR), pp. 247–248 (2001)
Obrenovic, Z., Starcevic, D.: Modeling multimodal human-computer interaction. Computer 37(9), 65–72 (2004)
Wang, L., Zhang, B., Dai, G.Z., et al.: PIP: a natural interaction interface based on multimodal interaction technology. In: Proceedings of the 3rd Joint Conference on Harmonious Human Machine Environment (HHME 2007), College of Computer and Science Technology, Shandong University, vol. 7 (2007)
Myers, B., Pausch, R., Pausch, R.: Past present and future of user interface software tools. ACM Trans. Comput. Hum. Interact. 7(1), 1–28 (2000)
Lozano-Quilis, J.A., Gil-Gomez, H., Gil-Gomez, J.A., Albiol-Perez, S., et al.: Virtual reality system for multiple sclerosis rehabilitation using KINECT. In: Proceedings of the 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops, PervasiveHealth, pp. 366–369 (2013)
Gu, Y., Sosnovsky, S., Ullrich, C.: Modeling children’s pedestrian safety skills in an intelligent virtual reality learning environment. In: Conati, C., Heffernan, N., Mitrovic, A., Felisa Verdejo, M. (eds.) AIED 2015. LNCS, vol. 9112, pp. 604–607. Springer, Heidelberg (2015)
Muda, L., Begam, M., Elamvazuthi, I.: Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques. arXiv preprint arXiv:10034083 (2010)
Varas, A., Cornejo, M.D., Mainemer, D., et al.: Cellular automaton model for evacuation process with obstacles. Phys. Stat. Mech. Appl. 382(2), 631–642 (2007)
Acknowledgement
This paper is supported by the National Natural Science Foundation of China (Grant No. 61472314).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Science+Business Media Singapore
About this paper
Cite this paper
Li, X., Ji, L., Han, F., Sun, X. (2016). Application of the Multimodal Human-Computer Interaction Technology in Product Virtual Display. In: Zhang, L., Song, X., Wu, Y. (eds) Theory, Methodology, Tools and Applications for Modeling and Simulation of Complex Systems. AsiaSim SCS AutumnSim 2016 2016. Communications in Computer and Information Science, vol 645. Springer, Singapore. https://doi.org/10.1007/978-981-10-2669-0_45
Download citation
DOI: https://doi.org/10.1007/978-981-10-2669-0_45
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-2668-3
Online ISBN: 978-981-10-2669-0
eBook Packages: Computer ScienceComputer Science (R0)