Augmented Reality for Interactive Robot Control

  • Levi Manring
  • John Pederson
  • Dillon Potts
  • Beth Boardman
  • David Mascarenas
  • Troy Harden
  • Alessandro CattaneoEmail author
Conference paper
Part of the Conference Proceedings of the Society for Experimental Mechanics Series book series (CPSEMS)


Robots are widely used to support mission-critical, high-risk and complex operations. Human supervision and remote robot control are often required to operate robots in unpredictable and changing scenarios. Often, robots are controlled remotely by technicians via joystick interfaces which require training and experience to operate. To improve robot usage and practicality, we propose using augmented reality (AR) to create a more intuitive, less training-intensive means of controlling robots than traditional joystick control. AR is a creative platform for developing robot control systems, because AR combines the real world (the environment around the user, the physical robot, etc.) with the digital world (holograms, digital displays, etc.); it can even interpret physical gestures, such as pinching two fingers.

In this research, a Microsoft Hololens headset is used to create an AR environment to control a Yaskawa Motoman SIA5D robot. The control process begins with the user placing an interactable holographic robot in 3D space. The user can then select between two control methods: manual control and automatic control. In manual control, the user can move the end effector of the holographic robot and the physical robot will respond immediately. In automatic control, the user can move the end effector of the holographic robot to a desired location, view a holographic preview of the motion, and select execute if the motion plan is satisfactory. In this preview mode, the user is able to preview both the motion of the robot and the torques experienced by the joints of the manipulator. This gives the user additional feedback on the planned motion. In this project we succeeded in creating an AR control system that makes controlling a robotic manipulator intuitive and effective.


Augmented reality Microsoft Hololens Robotic arm Force feedback Motion planning 



We would like to acknowledge the 2018 Dynamics Summer School at the Los Alamos National Laboratory for sponsoring this project as well as James Riback and Anita Jaramillo for their contributions to this project.


  1. 1.
    Kim, S.L., Suk, H.J., Kang, J.H., Jung, J.M., Laine, T.H., Westlin, J.: Using unity 3D to facilitate mobile augmented reality game development. In: 2014 IEEE World Forum on Internet of Things (WF-IoT), pp. 21–26. IEEE, Hoboken (2014)CrossRefGoogle Scholar
  2. 2.
    Lee, A., Lee, J.-H., Kim, J.: Data-driven kinematic control for robotic spatial augmented reality system with loose kinematic specifications. ETRI J. 38(2), 337–346 (2016)CrossRefGoogle Scholar
  3. 3.
    Kuriya, R., Tsujimura, T., Izumi, K.: Augmented reality robot navigation using infrared marker. In: Robot and Human Interactive Communication (RO-MAN), 2015 24th IEEE International Symposium on, pp. 450–455. IEEE, Hoboken (2015)Google Scholar
  4. 4.
    Zeng, H., Wang, Y., Wu, C., Song, A., Liu, J., Ji, P., Xu, B., Zhu, L., Li, H., Wen, P.: Closed-loop hybrid gaze brain-machine interface based robotic arm control with augmented reality feedback. Front. Neurorobot. 11, 60 (2017)CrossRefGoogle Scholar
  5. 5.
    Yew, A.W.W., Ong, S.K., Nee, A.Y.C.: Immersive augmented reality environment for the teleoperation of maintenance robots. Procedia CIRP. 61, 305–310 (2017)CrossRefGoogle Scholar
  6. 6.
    Hashimoto, S., Ishida, A., Inami, M., Igarashi, T.: Touchme: an augmented reality based remote robot manipulation. In: 21st International Conference on Artificial Reality and Telexistence, Proceedings of ICAT2011. Osaka University, Osaka (2011)Google Scholar
  7. 7.
    Fang, H.C., Ong, S.K., Nee, A.Y.C.: Interactive robot trajectory planning and simulation using augmented reality. Robot. Comput. Integr. Manuf. 28(2), 227–237 (2012)CrossRefGoogle Scholar
  8. 8.
    Fang, H.C., Ong, S.K., Nee, A.Y.C.: Orientation planning of robot end-effector using augmented reality. Int. J. Adv. Manuf. Technol. 67(9–12), 2033–2049 (2013)CrossRefGoogle Scholar
  9. 9.
    Bau, O., Poupyrev, I.: REVEL. ACM Trans. Graph. 31(4), 1–11 (2012)CrossRefGoogle Scholar
  10. 10.
    Zhao, Z., Huang, P., Lu, Z., Liu, Z.: Augmented reality for enhancing tele-robotic system with force feedback. Robot. Auton. Syst. 96, 93–101 (2017)CrossRefGoogle Scholar
  11. 11.
    Chen, R.-J., Lin, H.-W., Chang, Y.-H., Wu, C.-T., Lee, S.-T.: Development of an augmented reality force feedback virtual surgery training platform. Int J. Autom. Smart Technol. 1(1), 41–51 (2011)CrossRefGoogle Scholar
  12. 12.
    Culbertson, H., Kuchenbecker, K.J.: Ungrounded haptic augmented reality system for displaying roughness and friction. IEEE-ASME Trans. Mech. 22(4), 1839–1849 (2017)CrossRefGoogle Scholar
  13. 13.
    Lee, S., Lucas, N.P., Darin Ellis, R., Pandya, A.: Development and human factors analysis of an augmented reality interface for multi-robot tele-operation and control. In: Unmanned Systems Technology XIV, vol. 8387, p. 83870N. International Society for Optics and Photonics, Baltimore, Maryland (2012)CrossRefGoogle Scholar
  14. 14.
    Hu, Y., Meng, W.: Rosunitysim: Development and experimentation of a real-time simulator for multi-unmanned aerial vehicle local planning. Simulation. 92(10), 931–944 (2016)CrossRefGoogle Scholar
  15. 15.
    Mizuchi, Y., Inamura, T.: Cloud-based multimodal human-robot interaction simulator utilizing ros and unity frameworks. In: System Integration (SII), 2017 IEEE/SICE International Symposium on, pp. 948–955. IEEE, Hoboken (2017)CrossRefGoogle Scholar
  16. 16.
    Codd-Downey, R., Mojiri Forooshani, P., Speers, A., Wang, H., Jenkin, M.: From ROS to unity: leveraging robot and virtual environment middleware for immersive teleoperation. In: Information and Automation (ICIA), 2014 IEEE International Conference on, pp. 932–936. IEEE, Hoboken (2014)CrossRefGoogle Scholar
  17. 17.
    Sita, E., Horvath, C.M., Thomessen, T., Korondi, P., Pipe, A.G.: ROS-Unity3D based system for monitoring of an industrial robotic process. In: 2017 IEEE/SICE International Symposium on System Integration (SII). IEEE, Hoboken (2018)Google Scholar

Copyright information

© Society for Experimental Mechanics, Inc. 2020

Authors and Affiliations

  • Levi Manring
    • 1
  • John Pederson
    • 2
  • Dillon Potts
    • 3
  • Beth Boardman
    • 4
  • David Mascarenas
    • 5
  • Troy Harden
    • 4
  • Alessandro Cattaneo
    • 4
    Email author
  1. 1.Department of Mechanical Engineering and Materials Science, Pratt School of EngineeringDuke UniversityDurhamUSA
  2. 2.Department of Mechanical Engineering, The Fu Foundation School of Engineering and Applied ScienceColumbia UniversityNew YorkUSA
  3. 3.Department of Mechanical Engineering, Ira A. Fulton CollegeBrigham Young UniversityProvoUSA
  4. 4.Engineering Technology and Design DivisionLos Alamos National LaboratoryLos AlamosUSA
  5. 5.Engineering InstituteLos Alamos National LaboratoryLos AlamosUSA

Personalised recommendations