An augmented reality-based training system with a natural user interface for manual milling operations


This study developed an augmented reality (AR)-based training system for conventional manual milling operations. An Intel RealSense R200 depth camera and a Leap Motion controller were mounted on an HTC Vive head-mounted display to allow users freely walk around in a room-size AR environment to operate a full-size virtual milling machine with their barehands, using their natural operation behaviors, as if they were operating a real milling machine in the real world, without additional worn or handheld devices. GPU parallel computing was used to handle dynamic occlusions and accelerate the machining simulation to achieve a real-time simulation. Using the developed AR-based training system, novices can receive a hands-on training in a safe environment, without any injury or damage. User test results showed that using the developed AR-based training resulted in lower failure rates and inquiry times than using video training. Users also commented that the AR-based training was interesting and helpful for novices to learn the basic manual milling operation techniques.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13


  1. Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Ind 189(194):4–7

    Google Scholar 

  2. Chardonnet J-R, Fromentin G, Outeiro J (2017) Augmented reality as an aid for the use of machine tools. In: The 15th management and innovative technologies (MIT) conference, Sinaia, Romania pp 1–4

  3. Corbett-Davies S, Green R, Clark A (2012) Physically interactive tabletop augmented reality using the Kinect. In: Proceedings of the 27th conference on image and vision computing Dunedin, New Zealand, pp 210–215

  4. Gheorghe C, Rizzotti D, Tièche F, Carrino F, Khaled OA, Mugellini E (2015) Occlusion management in augmented reality systems for machine-tools. In: International conference on virtual, augmented and mixed reality, Los Angeles, CA, USA, pp 438–446

  5. Khattak S, Cowan B, Chepurna I, Hogue A (2014) A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion. In: 2014 IEEE games media entertainment, Toronto, ON, Canada, pp 1–8

  6. Kiswanto G, Ariansyah D (2013) Development of augmented reality (AR) for machining simulation of 3-axis CNC milling. In: International conference on advanced computer science and information systems (ICACSIS), Bali, Indonesia, pp 143–148

  7. Leal-Meléndrez JA, Altamirano-Robles L, Gonzalez JA (2013) Occlusion handling in video-based augmented reality using the Kinect sensor for indoor registration. In: Proceedings of 18th Iberoamerican congress CIARP 2013 on progress in pattern recognition, image analysis, computer vision, and applications, Havana, Cuba, pp 447–454

  8. Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput Graphics Anaheim 21(4):163–169

    Article  Google Scholar 

  9. Lu Y, Smith S (2009) GPU-based real-time occlusion in an immersive augmented reality environment. J Comput Inf Sci Eng 9(2):024501

    Article  Google Scholar 

  10. Neugebauer R, Klimant P, Wittstock V (2010) Virtual-reality-based simulation of NC programs for milling machines. In: Proceedings of the 20th CIRP design conference on global product development, Ecole Centrale de Nantes, Nantes, France, pp 697–703

  11. Penelle B, Debeir O (2014) Multi-sensor data fusion for hand tracking using Kinect and Leap Motion. In: Proceedings of the 2014 virtual reality international conference, Laval, France, pp 1–7

  12. Qiu S, Fan X, Wu D, He Q, Zhou D (2013) Virtual human modeling for interactive assembly and disassembly operation in virtual reality environment. Int J Adv Manuf Technol 69:9–12

    Article  Google Scholar 

  13. Regazzoni D, Rizzi C, Vitali A (2018) Virtual reality applications: guidelines to design natural user interface. In: Proceedings of the ASME 2018 international design engineering technical conferences and computers and information in engineering conference, Quebec, Canada, pp DETC2018-85867

  14. Shim J, Yang Y, Kang N, Seo J, Han T-D (2016) Gesture-based interactive augmented reality content authoring system using HMD. Virtual Real 20(1):57–69

    Article  Google Scholar 

  15. Sportillo D, Avveduto G, Tecchia F, Carrozzino M (2015) Training in VR: a preliminary study on learning assembly/disassembly sequences. In: International conference on augmented and virtual reality, Lecce, Italy, pp 332–343

  16. Tullis TS, Stetson JN (2004) A comparison of questionnaires for assessing website usability. In: Usability professional association conference, Minneapolis, Minnesota, USA, pp 1–12

  17. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393

    Article  Google Scholar 

  18. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  19. Zhang J, Ong S-K, Nee AY (2008) AR-assisted in situ machining simulation: architecture and implementation. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry, p 26

Download references


The authors would like to thank the Ministry of Science and Technology, Taiwan, Republic of China for financially supporting this research under Contract MOST 108-2221-E-002-161-MY2.

Author information



Corresponding author

Correspondence to Shana Smith.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, CK., Chen, YH., Chuang, TJ. et al. An augmented reality-based training system with a natural user interface for manual milling operations. Virtual Reality 24, 527–539 (2020).

Download citation


  • Augmented reality
  • Natural operation behavior
  • Manual milling operation
  • Occlusion