An augmented reality-based training system with a natural user interface for manual milling operations

  • Chih-Kai Yang
  • Yu-Hsi Chen
  • Tung-Jui Chuang
  • Kalpana Shankhwar
  • Shana SmithEmail author
Original Article


This study developed an augmented reality (AR)-based training system for conventional manual milling operations. An Intel RealSense R200 depth camera and a Leap Motion controller were mounted on an HTC Vive head-mounted display to allow users freely walk around in a room-size AR environment to operate a full-size virtual milling machine with their barehands, using their natural operation behaviors, as if they were operating a real milling machine in the real world, without additional worn or handheld devices. GPU parallel computing was used to handle dynamic occlusions and accelerate the machining simulation to achieve a real-time simulation. Using the developed AR-based training system, novices can receive a hands-on training in a safe environment, without any injury or damage. User test results showed that using the developed AR-based training resulted in lower failure rates and inquiry times than using video training. Users also commented that the AR-based training was interesting and helpful for novices to learn the basic manual milling operation techniques.


Augmented reality Natural operation behavior Manual milling operation Occlusion 



The authors would like to thank the Ministry of Science and Technology, Taiwan, Republic of China for financially supporting this research under Contract MOST 108-2221-E-002-161-MY2.


  1. Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Ind 189(194):4–7Google Scholar
  2. Chardonnet J-R, Fromentin G, Outeiro J (2017) Augmented reality as an aid for the use of machine tools. In: The 15th management and innovative technologies (MIT) conference, Sinaia, Romania pp 1–4Google Scholar
  3. Corbett-Davies S, Green R, Clark A (2012) Physically interactive tabletop augmented reality using the Kinect. In: Proceedings of the 27th conference on image and vision computing Dunedin, New Zealand, pp 210–215Google Scholar
  4. Gheorghe C, Rizzotti D, Tièche F, Carrino F, Khaled OA, Mugellini E (2015) Occlusion management in augmented reality systems for machine-tools. In: International conference on virtual, augmented and mixed reality, Los Angeles, CA, USA, pp 438–446Google Scholar
  5. Khattak S, Cowan B, Chepurna I, Hogue A (2014) A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion. In: 2014 IEEE games media entertainment, Toronto, ON, Canada, pp 1–8Google Scholar
  6. Kiswanto G, Ariansyah D (2013) Development of augmented reality (AR) for machining simulation of 3-axis CNC milling. In: International conference on advanced computer science and information systems (ICACSIS), Bali, Indonesia, pp 143–148Google Scholar
  7. Leal-Meléndrez JA, Altamirano-Robles L, Gonzalez JA (2013) Occlusion handling in video-based augmented reality using the Kinect sensor for indoor registration. In: Proceedings of 18th Iberoamerican congress CIARP 2013 on progress in pattern recognition, image analysis, computer vision, and applications, Havana, Cuba, pp 447–454CrossRefGoogle Scholar
  8. Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput Graphics Anaheim 21(4):163–169CrossRefGoogle Scholar
  9. Lu Y, Smith S (2009) GPU-based real-time occlusion in an immersive augmented reality environment. J Comput Inf Sci Eng 9(2):024501CrossRefGoogle Scholar
  10. Neugebauer R, Klimant P, Wittstock V (2010) Virtual-reality-based simulation of NC programs for milling machines. In: Proceedings of the 20th CIRP design conference on global product development, Ecole Centrale de Nantes, Nantes, France, pp 697–703CrossRefGoogle Scholar
  11. Penelle B, Debeir O (2014) Multi-sensor data fusion for hand tracking using Kinect and Leap Motion. In: Proceedings of the 2014 virtual reality international conference, Laval, France, pp 1–7Google Scholar
  12. Qiu S, Fan X, Wu D, He Q, Zhou D (2013) Virtual human modeling for interactive assembly and disassembly operation in virtual reality environment. Int J Adv Manuf Technol 69:9–12CrossRefGoogle Scholar
  13. Regazzoni D, Rizzi C, Vitali A (2018) Virtual reality applications: guidelines to design natural user interface. In: Proceedings of the ASME 2018 international design engineering technical conferences and computers and information in engineering conference, Quebec, Canada, pp DETC2018-85867Google Scholar
  14. Shim J, Yang Y, Kang N, Seo J, Han T-D (2016) Gesture-based interactive augmented reality content authoring system using HMD. Virtual Real 20(1):57–69CrossRefGoogle Scholar
  15. Sportillo D, Avveduto G, Tecchia F, Carrozzino M (2015) Training in VR: a preliminary study on learning assembly/disassembly sequences. In: International conference on augmented and virtual reality, Lecce, Italy, pp 332–343Google Scholar
  16. Tullis TS, Stetson JN (2004) A comparison of questionnaires for assessing website usability. In: Usability professional association conference, Minneapolis, Minnesota, USA, pp 1–12Google Scholar
  17. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393CrossRefGoogle Scholar
  18. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRefGoogle Scholar
  19. Zhang J, Ong S-K, Nee AY (2008) AR-assisted in situ machining simulation: architecture and implementation. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry, p 26Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Mechanical EngineeringNational Taiwan UniversityTaipeiTaiwan

Personalised recommendations