Design of a virtual reality training system for human–robot collaboration in manufacturing tasks

  • Elias Matsas
  • George-Christopher Vosniakos
Original Paper


This paper presents a highly interactive and immersive Virtual Reality Training System (VRTS) (“beWare of the Robot”) in terms of a serious game that simulates in real-time the cooperation between industrial robotic manipulators and humans, executing simple manufacturing tasks. The scenario presented refers to collaborative handling in tape-laying for building aerospace composite parts. The tools, models and techniques developed and used to build the “beWare of the Robot” application are described. System setup and configuration are presented in detail, as well as user tracking and navigation issues. Special emphasis is given to the interaction techniques used to facilitate implementation of virtual human–robot (HR) collaboration. Safety issues, such as contacts and collisions are mainly tackled through “emergencies”, i.e. warning signals in terms of visual stimuli and sound alarms. Mental safety is of utmost priority and the user is provided augmented situational awareness and enhanced perception of the robot’s motion due to immersion and real-time interaction offered by the VRTS as well as by special warning stimuli. The short-term goal of the research was to investigate users’ enhanced experience and behaviour inside the virtual world while cooperating with the robot and positive pertinent preliminary findings are presented and briefly discussed. In the longer term, the system can be used to investigate acceptability of H–R collaboration and, ultimately, serve as a platform for programming collaborative H–R manufacturing cells.


Virtual reality Human–robot collaboration Safety  Manufacturing training Interaction Serious game 



This research has been co-financed by the European Union (European Social Fund—ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF)—Research Funding Program: Heracleitus II. Investing in knowledge society through the European Social Fund. The authors would also like to thank Dimitrios Batras, MSc, for his generous help in programming, his motivation and his helpful comments during the development of the application.


  1. 1.
    Unhelkar, V.V., Siu, H.C., Shah, J.A.: Comparative performance of human and mobile robotic assistants in collaborative fetch-and-deliver tasks. In: Proc. 2014 ACM/IEEE Int. Conf. Human–robot Interact—HRI ’14, pp. 82–89 (2014). doi: 10.1145/2559636.2559655
  2. 2.
    de Santos, P.G., Garcia, E., Sarria, J., Ponticelli, R., Reviejo, J.: A new manipulator structure for power-assist devices. Ind. Robot. An. Int. J. 37(5), 452–458 (2010). doi: 10.1108/01439911011063272 CrossRefGoogle Scholar
  3. 3.
    Knight, W.: Smart Robots Can Now Work Right Next to Auto Workers. (2013). Accessed 22 April 2014
  4. 4.
    Rethink robotics: baxter robot overview. (2014). Accessed 22 April 2014
  5. 5.
    Coradeschi, S., Cesta, A., Cortellessa, G., Coraci, L., Gonzalez, J., Karlsson, L., Furfari, F., Loutfi, A., Orlandini, A., Palumbo, F., Pecora, F., von Rump, S., Stimec, A., Ullberg, J., Otslund, B.: GiraffPlus: combining social interaction and long term monitoring for promoting independent living, 2013. In: 6th Int. Conf. Hum. Syst. Interact, pp. 578–585 (2013). doi: 10.1109/HSI.2013.6577883
  6. 6.
    Cavallo, F., Aquilano, M., Bonaccorsi, M., Mannari, I., Carrozza, M.C., Dario, P.: Multidisciplinary approach for developing a new robotic system for domiciliary assistance to elderly people. In: Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf., pp. 5327–5330 (2011). doi: 10.1109/HSI.2013.6577883
  7. 7.
    Helander, M.G.: Ergonomics and safety considerations in the design of robotics workplaces: a review and some priorities for research. Int. J. Ind. Ergon. 6(2), 127–149 (1990). doi: 10.1016/0169-8141(90)90018-W CrossRefGoogle Scholar
  8. 8.
    Working with robots: our friends electric. (2013). Accessed 22 April 2014
  9. 9.
    Ogorodnikova, O.: Human Robot Interaction: The safety challenge (an integrated frame work for human safety). PhD dissertation, Budapest University of Technology and Economics, Faculty of Mechanical Engineering, Budapest (2010)Google Scholar
  10. 10.
    Alami, R., Bicchi, A., Bischoff, R., Chatila, R., De Luca, A., De Santis, A.: Safe and dependable physical human–robot interaction in anthropic domains: State of the art and challenges. In: Bicchi, A., De Luca, A. (ed.) Procceedings IROS Workshop on pHRI—Physical Human–Robot Interaction in Anthropic Domains (2006)Google Scholar
  11. 11.
    Hoffman, G., Breazeal, C.: Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team. Proceeding ACM/IEEE Int. Conf. Human-robot Interact. - HRI ’07. 1–8 (2007). doi: 10.1145/1228716.1228718
  12. 12.
    Wickens, C.D.: Situation awareness: review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Hum. Factors. 50(3), 397–403 (2008). doi: 10.1518/001872008X288420 CrossRefGoogle Scholar
  13. 13.
    Fuchs, P., Moreau, G., Moreau, G., Burkhardt, J.-M., Coquillart, S.: Le traité de la réalité virtuelle, vol. 2. L’interfaçage, l’immersion et l’interaction en environnement virtuel. Les Presses de l’Ecole des Mines de Paris, Paris (2006)Google Scholar
  14. 14.
    Lin, F., Ye, L., Duffy, V., Su, C.-J.: Developing virtual environments for industrial training. Inf. Sci. (Ny) 140(1—-2), 153–170 (2002). doi: 10.1016/S0020-0255(01)00185-2 CrossRefMATHGoogle Scholar
  15. 15.
    Shukla, C., Vazquez, M., Frank Chen, F.: Virtual manufacturing: an overview. Comput. Ind. Eng. 31(1–2), 79–82 (1996). doi: 10.1016/0360-8352(96)00083-6 CrossRefGoogle Scholar
  16. 16.
    Burdea, G., Coiffet, P.: Virtual Reality Technology, 2nd edn. Wiley-Interscience, New York (2003)Google Scholar
  17. 17.
    Duguleana, M., Barbuceanu, F.G., Mogan, G.: Virtual and mixed reality—new trends. In: Shumaker, R. (ed.) Virtual and mixed reality—new trends. Springer, Berlin, pp. 164–173 (2011). doi: 10.1007/978-3-642-22021-0_19
  18. 18.
    Zhu, J., Ong, S.K., Nee, A.Y.C.: A context-aware augmented reality system to assist the maintenance operators. Int. J. Interact. Des. Manuf. (2014). doi: 10.1007/s12008-013-0199-7 Google Scholar
  19. 19.
    Nathanael, D., Vosniakos, G.-C., Mosialos, S.: Cognitive task analysis for virtual reality training. In: Proceedings of the 28th Annual European Conference on Cognitive Ergonomics—ECCE ’10. ACM Press, New York, p. 241 (2010). doi: 10.1145/1962300.1962350
  20. 20.
    Burkhardt, J.-M., Bardy, B., Lourdeaux, D.: Immersion, Réalisme et Présence dans la conception et l ’ évaluation des Environnements Virtuels. Psychol. Française. 48(2), 35–42 (2003)Google Scholar
  21. 21.
    Pook, P.K., Ballard, D.H.: Deictic human/robot interaction. Rob. Auton. Syst. 18, 259–269 (1996). doi: 10.1016/0921-8890(95)00080-1 CrossRefGoogle Scholar
  22. 22.
    Morioka, M., Sakakibara, S.: A new cell production assembly system with human–robot cooperation. CIRP Ann. Manuf. Technol. 59, 9–12 (2010). doi: 10.1016/j.cirp.2010.03.044 CrossRefGoogle Scholar
  23. 23.
    Krüger, J., Lien, T.K., Verl, A.: Cooperation of human and machines in assembly lines. CIRP Ann. Manuf. Technol. 58, 628–646 (2009). doi: 10.1016/j.cirp.2009.09.009 CrossRefGoogle Scholar
  24. 24.
    Giesler, B., Steinhaus, P., Walther, M., Dillmann, R.: Sharing skills: using augmented reality for human–robot collaboration. In: Woods, A.J., Merritt, J.O., Benton, S.A., Bolas, M.T. (eds.) Proc. SPIE 5291, Stereoscopic Displays and Virtual Reality Systems XI, pp. 446–453 (2004). doi: 10.1117/12.526689
  25. 25.
    Corrales, Ja, Candelas, Fa, Torres, F.: Safe human–robot interaction based on dynamic sphere-swept line bounding volumes. Robot. Comput. Integr. Manuf. 27(1), 177–185 (2011). doi: 10.1016/j.rcim.2010.07.005 CrossRefGoogle Scholar
  26. 26.
    Arai, T., Kato, R., Fujita, M.: Assessment of operator stress induced by robot collaboration in assembly. CIRP Ann. Manuf. Technol. 59(1), 5–8 (2010). doi: 10.1016/j.cirp.2010.03.043 CrossRefGoogle Scholar
  27. 27.
    Charoenseang, S., Tonggoed, T.: Human–robot collaboration with augmented reality. In: Stephanidis, C. (ed.) HCI International 2011—Posters’ Extended Abstracts. Springer, Berlin, pp. 93–97 (2011). doi: 10.1007/978-3-642-22095-1_19
  28. 28.
    Fang, H.C., Ong, S.K., Nee, A.Y.C.: A novel augmented reality-based interface for robot path planning. Int. J. Interact. Des. Manuf. 8(1), 33–42 (2014). doi: 10.1007/s12008-013-0191-2 CrossRefGoogle Scholar
  29. 29.
    Du, G., Zhang, P.: Markerless human–robot interface for dual robot manipulators using Kinect sensor. Robot. Comput. Integr. Manuf. 30(2), 150–159 (2014). doi: 10.1016/j.rcim.2013.09.003 CrossRefGoogle Scholar
  30. 30.
    Wang, L., Schmidt, B., Nee, A.Y.C.: Vision-guided active collision avoidance for human–robot collaborations. Manuf. Lett. 1(1), 5–8 (2013). doi: 10.1016/j.mfglet.2013.08.001 CrossRefGoogle Scholar
  31. 31.
    Qiu, S., Fan, X., Wu, D., He, Q., Zhou, D.: Virtual human modeling for interactive assembly and disassembly operation in virtual reality environment. Int. J. Adv. Manuf. Technol. 69(9–12), 2355–2372 (2013). doi: 10.1007/s00170-013-5207-3 CrossRefGoogle Scholar
  32. 32.
    Zaeh, M., Roesel, W.: Safety aspects in a human–robot interaction scenario: a human worker is co-operating with an industrial robot. In: Kim, J.-H., et al. (eds.) Progress in robotics. Springer, Berlin, pp. 53–62 (2009). doi: 10.1007/978-3-642-03986-7_7
  33. 33.
    Neuhoefer, J.A., Bernhard, K., Schlick, C.M.: Embedded augmented reality training system for dynamic human–robot cooperation. NATO RTO-MP-HFM-169. RTO Human Factors and Medicine Panel (HFM) Workshop, RDECOM, pp. 1–14, Orlando, Florida, USA (2009)Google Scholar
  34. 34.
    Weistroffer, V., Paljic, A., Callebert, L., Fuchs, P.: A methodology to assess the acceptability of human-robot collaboration using virtual reality. In: Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology—VRST ’13. ACM Press, New York, p. 39 (2013). doi: 10.1145/2503713.2503726
  35. 35.
    Matsas, E., Batras, D., Vosniakos, G.: Beware of the robot: a highly interactive and immersive virtual reality training application in robotic manufacturing systems. In: Emmanouilidis, C., Taisch, M., Kiritsis, D. (eds.) Advances in Production Management Systems. Competitive Manufacturing for Innovative Products and Services. Springer, Berlin, pp. 606–613 (2013). doi: 10.1007/978-3-642-40352-1_76
  36. 36.
    Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D User Interfaces: Theory and Practice. Addison Wesley Longman Publishing Co., Inc, Redwood City (2004)Google Scholar
  37. 37.
    Evolver 3D digital avatar website. Accessed 23 January 2012
  38. 38.
    Usoh, M., Arthur, K., Whitton, M.C., Bastos, R., Steed, A., Slater, M., Brooks, F.P.: Walking, walking-in-place, flying, in virtual environments. In: Proc. 26th Annu. Conf. Comput. Graph. Interact. Tech.—SIGGRAPH ’99, pp. 359–364 (1999). doi: 10.1145/311535.311589
  39. 39.
    Unity online manual. Accessed 25 July 2014
  40. 40.
    Miles, R.: Start here! Learn microsoft kinect API. O’Reilly Media Inc, Sebastopol (2012)Google Scholar
  41. 41.
    Vora, J., Nair, S., Gramopadhye, A.K., Duchowski, A.T., Melloy, B.J., Kanki, B.: Using virtual reality technology for aircraft visual inspection training: presence and comparison studies. Appl. Ergon. 33(6), 559–70 (2002). doi: 10.1145/311535.311589 CrossRefGoogle Scholar
  42. 42.
    Prothero, J.D., Parker, D.E., Furness III, T.A., Wells, M.J.: Towards a robust, quantitative measure for presence. In: Garland, D.J., Endsley, M.R. (eds.) Conference on Experimental Analysis and Measurement of Situational Awareness. Embry-Riddle Aeronautical University Press, Daytona Beach, pp. 359–366 (1995)Google Scholar
  43. 43.
    Schuemie, M.J., van der Straaten, P., Krijn, M., van der Mast, C.A.: Research on presence in virtual reality: a survey. Cyberpsychol. Behav. 4(2), 183–201 (2001). doi: 10.1089/109493101300117884 CrossRefGoogle Scholar
  44. 44.
    Burkhardt, J.-M., Lourdeaux, D., Fuchs, P.: Conception d’un système de RV pour la formation des agents de conduite aux opérations en milieu ferroviaire. In: Richards, A., Grumbach, N. (eds.) Colloque Réalité Virtuelle et Cognition, pp. 123–132. ENST Paris, France (1999)Google Scholar

Copyright information

© Springer-Verlag France 2015

Authors and Affiliations

  1. 1.School of Mechanical EngineeringNational Technical University of AthensZografou-AthensGreece

Personalised recommendations