Skip to main content
SpringerLink
Log in

Simplifying Robot Programming Using Augmented Reality and End-User Development

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2021 (INTERACT 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12932))

Included in the following conference series:

Abstract

Robots are widespread across diverse application contexts. Teaching robots to perform tasks, in their respective contexts, demands a high domain and programming expertise. However, robot programming faces high entry barriers due to the complexity of robot programming itself. Even for experts robot programming is a cumbersome and error-prone task where faulty robot programs can be created, causing damage when being executed on a real robot. To simplify the process of robot programming, we combine Augmented Reality (AR) with principles of end-user development. By combining them, the real environment is extended with useful virtual artifacts that can enable experts as well as non-professionals to perform complex robot programming tasks. Therefore, Simple Programming Environment in Augmented Reality with Enhanced Debugging (SPEARED) was developed as a prototype for an AR-assisted robot programming environment. SPEARED makes use of AR to project a robot as well as a programming environment onto the target working space. To evaluate our approach, expert interviews with domain experts from the area of industrial automation, robotics, and AR were performed. The experts agreed that SPEARED has the potential to enrich and ease current robot programming processes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/VARobot-PG/application.

  2. 2.

    http://gazebosim.org.

  3. 3.

    https://www.ros.org.

  4. 4.

    https://www.docker.com.

  5. 5.

    https://ni.www.techfak.uni-bielefeld.de/files/URDF-XACRO.pdf.

  6. 6.

    https://www.dobot.us/.

  7. 7.

    https://github.com/microsoft/MixedRealityToolkit-Unity.

  8. 8.

    https://developers.google.com/blockly.

  9. 9.

    https://github.com/RobotWebTools/roslibjs.

References

  1. Aleotti, J., Caselli, S., Reggiani, M.: Leveraging on a virtual environment for robot programming by demonstration. Robot. Auton. Syst. 47(2–3), 153–161 (2004). https://doi.org/10.1016/j.robot.2004.03.009

  2. Aleotti, J., Caselli, S., Reggiani, M.: Evaluation of virtual fixtures for a robot programming by demonstration interface. IEEE Trans. Syst. Man Cybern. Part A 35(4), 536–545 (2005). https://doi.org/10.1109/TSMCA.2005.850604

  3. Aleotti, J., Micconi, G., Caselli, S.: Programming manipulation tasks by demonstration in visuo-haptic augmented reality. In: 2014 IEEE International Symposium on Haptic, Audio and Visual Environments and Games, HAVE 2014, Richardson, TX, USA, 10–11 October 2014, pp. 13–18. IEEE (2014). https://doi.org/10.1109/HAVE.2014.6954324

  4. Alexandrova, S., Cakmak, M., Hsiao, K., Takayama, L.: Robot programming by demonstration with interactive action visualizations. In: Fox, D., Kavraki, L.E., Kurniawati, H. (eds.) Robotics: Science and Systems X, University of California, Berkeley, USA, 12–16 July 2014 (2014). https://doi.org/10.15607/RSS.2014.X.048

  5. Coronado, E., Mastrogiovanni, F., Indurkhya, B., Venture, G.: Visual programming environments for end-user development of intelligent and social robots, a systematic review. J. Comput. Lang. 58, 100970 (2020). https://doi.org/10.1016/j.cola.2020.100970

  6. Fang, H., Ong, S.K., Nee, A.: Orientation planning of robot end-effector using augmented reality. Int. J. Adv. Manuf. Technol. 67(08) (2012). https://doi.org/10.1007/s00170-012-4629-7

  7. Fang, H., Ong, S.K., Nee, A.: Novel ar-based interface for human-robot interaction and visualization. Adv. Manuf. 2(12) (2014). https://doi.org/10.1007/s40436-014-0087-9

  8. Fang, H., Ong, S., Nee, A.Y.C.: Robot programming using augmented reality. In: Ugail, H., Qahwaji, R., Earnshaw, R.A., Willis, P.J. (eds.) 2009 International Conference on CyberWorlds, Bradford, West Yorkshire, UK, 7–11 September 2009, pp. 13–20. IEEE Computer Society (2009). https://doi.org/10.1109/CW.2009.14

  9. Gadre, S.Y., Rosen, E., Chien, G., Phillips, E., Tellex, S., Konidaris, G.D.: End-user robot programming using mixed reality. In: International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, 20–24 May 2019, pp. 2707–2713. IEEE (2019). https://doi.org/10.1109/ICRA.2019.8793988

  10. Gianni, M., Ferri, F., Pirri, F.: ARE: augmented reality environment for mobile robots. In: Natraj, A., Cameron, S., Melhuish, C., Witkowski, M. (eds.) TAROS 2013. LNCS (LNAI), vol. 8069, pp. 470–483. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-43645-5_48

    Chapter  Google Scholar 

  11. Gottschalk, S., Yigitbas, E., Schmidt, E., Engels, G.: Model-based product configuration in augmented reality applications. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) HCSE 2020. LNCS, vol. 12481, pp. 84–104. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64266-2_5

    Chapter  Google Scholar 

  12. Gottschalk, S., Yigitbas, E., Schmidt, E., Engels, G.: ProConAR: a tool support for model-based AR product configuration. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) HCSE 2020. LNCS, vol. 12481, pp. 207–215. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64266-2_14

    Chapter  Google Scholar 

  13. Jovanovikj, I., Yigitbas, E., Sauer, S., Engels, G.: Augmented and virtual reality object repository for rapid prototyping. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) HCSE 2020. LNCS, vol. 12481, pp. 216–224. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64266-2_15

    Chapter  Google Scholar 

  14. Krings, S., Yigitbas, E., Jovanovikj, I., Sauer, S., Engels, G.: Development framework for context-aware augmented reality applications. In: Bowen, J., Vanderdonckt, J., Winckler, M. (eds.) EICS 2020: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Sophia Antipolis, France, 23–26 June 2020, pp. 9:1–9:6. ACM (2020). https://doi.org/10.1145/3393672.3398640

  15. Lambrecht, J., Chemnitz, M., Krüger, J.: Control layer for multi-vendor industrial robot interaction providing integration of supervisory process control and multifunctional control units. In: 2011 IEEE Conference on Technologies for Practical Robot Applications, pp. 115–120 (2011). https://doi.org/10.1109/TEPRA.2011.5753492

  16. Lasi, H., Fettke, P., Kemper, H., Feld, T., Hoffmann, M.: Industry 4.0. Bus. Inf. Syst. Eng. 6(4), 239–242 (2014). https://doi.org/10.1007/s12599-014-0334-4

  17. Newcomer, K.E., Hatry, H.P., Wholey, J.S.: Conducting semi-structured interviews. In: Handbook of Practical Program Evaluation, pp. 492–505 (2015)

    Google Scholar 

  18. Orendt, E.M., Fichtner, M., Henrich, D.: Robot programming by non-experts: Intuitiveness and robustness of one-shot robot programming. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 192–199 (2016). https://doi.org/10.1109/ROMAN.2016.7745110

  19. Pan, Z., Polden, J., Larkin, N., van Duin, S., Norrish, J.: Recent progress on programming methods for industrial robots. In: ISR/ROBOTIK 2010, Proceedings for the joint conference of ISR 2010 (41st Internationel Symposium on Robotics) und ROBOTIK 2010 (6th German Conference on Robotics), Munich, Germany - Parallel to AUTOMATICA, 7–9 June 2010, pp. 1–8. VDE Verlag (2010). http://ieeexplore.ieee.org/document/5756855/

  20. Pettersen, T., Pretlove, J., Skourup, C., Engedal, T., Løkstad, T.: Augmented reality for programming industrial robots. In: 2003 IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR 2003), Tokyo, Japan, 7–10 October 2003, pp. 319–320. IEEE Computer Society (2003). https://doi.org/10.1109/ISMAR.2003.1240739

  21. Rosen, E., Whitney, D., Phillips, E., Chien, G., Tompkin, J., Konidaris, G.D., Tellex, S.: Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays. Int. J. Robot. Res. 38(12–13) (2019). https://doi.org/10.1177/0278364919842925

  22. Shepherd, D.C., Francis, P., Weintrop, D., Franklin, D., Li, B., Afzal, A.: [Engineering paper] an IDE for easy programming of simple robotics tasks. In: 18th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2018, Madrid, Spain, 23–24 September 2018, pp. 209–214. IEEE Computer Society (2018). https://doi.org/10.1109/SCAM.2018.00032

  23. Shepherd, D.C., Kraft, N.A., Francis, P.: Visualizing the hidden variables in robot programs. In: Proceedings of the 2nd International Workshop on Robotics Software Engineering, RoSE 2019, Montreal, QC, Canada, May 27, pp. 13–16. IEEE/ACM (2019). https://doi.org/10.1109/RoSE.2019.00007

  24. Solingen, R., Basili, V., Caldiera, G., Rombach, D.: Goal Question Metric (GQM) Approach, January (2002). https://doi.org/10.1002/0471028959.sof142

  25. Thanigaivel, N.K., Ong, S.K., Nee, A.: Augmented reality-assisted robot programming system for industrial applications. Robot. Comput. Integr. Manuf. 61 (2019). https://doi.org/10.1016/j.rcim.2019.101820

  26. Weintrop, D., Shepherd, D.C., Francis, P., Franklin, D.: Blockly goes to work: Block-based programming for industrial robots. In: 2017 IEEE Blocks and Beyond Workshop (B B), pp. 29–36 (2017). https://doi.org/10.1109/BLOCKS.2017.8120406

  27. Weintrop, D., Afzal, A., Salac, J., Francis, P., Li, B., Shepherd, D.C., Franklin, D.: Evaluating coblox: A comparative study of robotics programming environments for adult novices. In: Mandryk, R.L., Hancock, M., Perry, M., Cox, A.L. (eds.) Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, Canada, 21–26 April 2018, p. 366. ACM (2018). https://doi.org/10.1145/3173574.3173940

  28. Yigitbas, E., Heindörfer, J., Engels, G.: A context-aware virtual reality first aid training application. In: Alt, F., Bulling, A., Döring, T. (eds.) Proceedings of Mensch und Computer 2019, pp. 885–888. GI/ACM (2019)

    Google Scholar 

  29. Yigitbas, E., Jovanovikj, I., Sauer, S., Engels, G.: On the development of context-aware augmented reality applications. In: Abdelnour Nocera, J., Parmaxi, A., Winckler, M., Loizides, F., Ardito, C., Bhutkar, G., Dannenmann, P. (eds.) INTERACT 2019. LNCS, vol. 11930, pp. 107–120. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46540-7_11

    Chapter  Google Scholar 

  30. Yigitbas, E., Jovanovikj, I., Scholand, J., Engels, G.: VR training for warehouse management. In: Teather, R.J., Joslin, C., Stuerzlinger, W., Figueroa, P., Hu, Y.,Batmaz, A.U., Lee, W., Ortega, F.R. (eds.) VRST 2020: 26th ACM Symposium on Virtual Reality Software and Technology. pp. 78:1–78:3. ACM (2020)

    Google Scholar 

  31. Yigitbas, E., Karakaya, K., Jovanovikj, I., Engels, G.: Enhancing human-in-the-loop adaptive systems through digital twins and VR interfaces (2021). https://arxiv.org/abs/2103.10804

  32. Yigitbas, E., Sauer, S., Engels, G.: Using augmented reality for enhancing planning and measurements in the scaffolding business. In: EICS 2021: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Virtual, 8–11 June 2021. ACM (2021). https://doi.org/10.1145/3459926.3464747

  33. Yigitbas, E., Tejedor, C.B., Engels, G.: Experiencing and programming the ENIAC in VR. In: Alt, F., Schneegass, S., Hornecker, E. (eds.) Mensch und Computer 2020, pp. 505–506. ACM (2020)

    Google Scholar 

  34. Yoshizawa, Y., Watanobe, Y.: Logic error detection algorithm for novice programmers based on structure pattern and error degree. In: 9th International Conference on Awareness Science and Technology, iCAST 2018, Fukuoka, Japan, 19–21 September 2018, pp. 297–301. IEEE (2018). https://doi.org/10.1109/ICAwST.2018.8517171

Download references

Acknowledgement

We would like to thank Jonas Eilers and Michael Wieneke for their support during the implementation and evaluation of the presented approach.

Author information

Authors and Affiliations

  1. Paderborn University, Zukunftsmeile 2, 33102, Paderborn, Germany

    Enes Yigitbas, Ivan Jovanovikj & Gregor Engels

Authors
  1. Enes Yigitbas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivan Jovanovikj
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gregor Engels
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enes Yigitbas .

Editor information

Editors and Affiliations

  1. Department of Electrical and Information Engineering, Polytechnic University of Bari, Bari, Italy

    Carmelo Ardito

  2. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Rosa Lanzilotti

  3. Computer Science Department, University of Pisa, Pisa, Italy

    Alessio Malizia

  4. Department of Computer Science, University of York, York, UK

    Helen Petrie

  5. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  6. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Giuseppe Desolda

  7. Microsoft Research, Redmond, WA, USA

    Kori Inkpen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yigitbas, E., Jovanovikj, I., Engels, G. (2021). Simplifying Robot Programming Using Augmented Reality and End-User Development. In: Ardito, C., et al. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science(), vol 12932. Springer, Cham. https://doi.org/10.1007/978-3-030-85623-6_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85623-6_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85622-9

  • Online ISBN: 978-3-030-85623-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation