Skip to main content
SpringerLink
Log in

Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection

  • Chapter
  • First Online:
Book cover Assistive Augmentation

Part of the book series: Cognitive Science and Technology ((CSAT))

Abstract

When ordering a product the options to personalize or customize the items have increased over the last years. This flexibility has lead to an increasing number of variants in manufactured products. As storing produced items is expensive, companies tend to produce their products lean, i.e. in smaller lot sizes just when they are needed. This lean manufacturing creates more flexible production environments. In our work, we investigate how human workers can be assisted to work in such demanding environments. Therefore, Augmented Reality systems can be used to provide work instructions. First, in this chapter we provide a comprehensive overview about Augmented Reality approaches to support workers directly at the workplace and introduce an assistive system for providing in-situ instructions. Through three user studies, we evaluate the general impact of in-situ instructions, evaluate three instruction creation strategies, and finally evaluate the created instructions using a real product assembly task.

The majority of the work has been conducted while he was a researcher at the University of Stuttgart.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://memex-academy.eu/ (last access 03-18-2016).

  2. 2.

    http://www.ops-solutions.com/ (last access 03-18-2016).

  3. 3.

    http://www.emgu.com/ (last access 03-18-2016).

  4. 4.

    http://www.lego.com/en-us/duplo (last access 03-18-2016).

References

  1. Aleotti J, Caselli S (2006) Robust trajectory learning and approximation for robot programming by demonstration. Robot Auton Syst 54(5):409–413

    Article  Google Scholar 

  2. Antifakos S, Michahelles F, Schiele B (2002) Proactive instructions for furniture assembly. In: UbiComp 2002: ubiquitous computing. Springer, pp 351–360

    Google Scholar 

  3. Bannat A, Gast J, Rigoll G, Wallhoff F (2008) Event analysis and interpretation of human activity for augmented reality-based assistant systems. In: 4th international conference on intelligent computer communication and processing, 2008. ICCP 2008. IEEE, pp 1–8

    Google Scholar 

  4. Barna J, NovakovaMarcincinova L, Novak-Marcincin J, Fecova V, Janak M, Torok J (2012) Open source tools in assembling process enriched with elements of augmented reality. In: Proceedings of the 2012 virtual reality international conference. ACM, p 2

    Google Scholar 

  5. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer vision-ECCV 2006. Springer, pp 404–417

    Google Scholar 

  6. Billard A, Calinon S, Dillmann R, Schaal S (2008) Robot programming by demonstration. In: Springer handbook of robotics. Springer, pp 1371–1394

    Google Scholar 

  7. Blanke U, Schiele B, Kreil M, Lukowicz P, Sick B, Gruber T (2010) All for one or one for all? Combining heterogeneous features for activity spotting. In: 2010 8th IEEE international conference on Pervasive computing and communications workshops (PERCOM Workshops). IEEE, pp 18–24

    Google Scholar 

  8. Büttner S, Sand O, Röcker C (2015) Extending the design space in industrial manufacturing through mobile projection. In: Proceedings of the 17th international conference on human-computer interaction with mobile devices and services adjunct. ACM, pp 1130–1133

    Google Scholar 

  9. Büttner S, Mucha H, Funk M, Kosch T, Aehnelt M, Robert S, Röcker C (2017) The design space of augmented and virtual reality applications for assistive environments in manufacturing: a visual approach. In: Proceedings of the 10th international conference on pervasive technologies related to assistive environments. ACM, pp 433–440

    Google Scholar 

  10. Büttner S, Funk M, Sand P, Röcker C (2016) Using head-mounted displays and in-situ projection for assistive systems: a comparison. In: Proceedings of the 9th ACM international conference on pervasive technologies related to assistive environments. ACM, p 44

    Google Scholar 

  11. Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the twenty-fifth hawaii international conference on system sciences, vol 2. IEEE, pp 659–669

    Google Scholar 

  12. Collett T, MacDonald BA (2006) Developer oriented visualisation of a robot program. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction. ACM, pp 49–56

    Google Scholar 

  13. Fiorentino M, de Amicis R, Monno G, Stork A (2002) Spacedesign: a mixed reality workspace for aesthetic industrial design. In: Proceedings of the 1st international symposium on mixed and augmented reality. IEEE Computer Society, p 86

    Google Scholar 

  14. Funk M, Korn O, Schmidt A (2015) Enabling end users to program for smart environments. In: Proceedings of the CHI 2015—workshop on end user development in the internet of things era 12.2, pp 9–14

    Google Scholar 

  15. Funk M, Kosch T, Schmidt A (2016) Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 934–939

    Google Scholar 

  16. Funk M, Mayer S, Schmidt A (2015) Using in-situ projection to support cognitively impaired workers at the workplace. In: Proceedings of the 17th international ACM SIGACCESS conference on computers and accessibility

    Google Scholar 

  17. Funk M, Schmidt A (2015) Cognitive assistance in the workplace. Pervasive Comput IEEE 14(3):53–55

    Article  Google Scholar 

  18. Funk M, Kosch T, Greenwald SW, Schmidt A (2015) A benchmark for interactive augmented reality instructions for assembly tasks. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia. ACM, pp 253–257

    Google Scholar 

  19. Funk M, Bächler A, Bächler L, Korn O, Krieger C, Heidenreich T, Schmidt A (2015) Comparing projected in-situ feedback at the manual assembly workplace with impaired workers. In: Proceedings of the 8th ACM international conference on pervasive technologies related to assistive environments. ACM, p 1

    Google Scholar 

  20. Funk M, Kosch T, Kettner R, Korn O, Schmidt A (2016) Motioneap: an overview of 4 years of combining industrial assembly with augmented reality for industry 4.0. In: Proceedings of the 16th international conference on knowledge technologies and datadriven business

    Google Scholar 

  21. Funk M, Shirazi AS, Mayer S, Lischke L, Schmidt A (2015) Pick from here!: an interactive mobile cart using in-situ projection for order picking. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 601–609

    Google Scholar 

  22. Guo A, Raghu S, Xie X, Ismail S, Luo X, Simoneau J, Gilliland S, Baumann H, Southern C, Starner T (2014) A comparison of order picking assisted by head-up display (HUD), cart-mounted display (CMD), light, and paper pick list. In: Proceedings of the 2014 ACM international symposium on wearable computers. ACM, pp 71–78

    Google Scholar 

  23. Hahn J, Ludwig B, Wolff C (2015) Augmented reality-based training of the PCB assembly process. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia. ACM, pp 395–399

    Google Scholar 

  24. Hardy J, Alexander J (2012) Toolkit support for interactive projected displays. In: Proceedings of the 11th international conference on mobile and ubiquitous multimedia. ACM, p 42

    Google Scholar 

  25. Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol 52:139–183

    Article  Google Scholar 

  26. Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual Comput Graph 17(10):1355–1368

    Article  Google Scholar 

  27. Hermann M, Pentek T, Otto B. Design principles for industrie 4.0 scenarios: a literature review

    Google Scholar 

  28. Klompmaker F, Nebe K, Fast A (2012) dSensingNI: a framework for advanced tangible interaction using a depth camera. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction. ACM, pp 217–224

    Google Scholar 

  29. Korn O, Schmidt A, Hörz T (2013) Augmented manufacturing: a study with impaired persons on assistive systems using in-situ projection. In: Proceedings of the 6th international conference on pervasive technologies related to assistive environments. ACM, p 21

    Google Scholar 

  30. Korn O, Funk M, Abele S, Hörz T, Schmidt A (2014) Context-aware assistive systems at the workplace: analyzing the effects of projection and gamification. In: Proceedings of the 7th international conference on pervasive technologies related to assistive environments. ACM, p 8

    Google Scholar 

  31. Kubitza T, Schmidt A (2015) Towards a toolkit for the rapid creation of smart environments. In: End-user development. Springer, pp 230–235

    Google Scholar 

  32. Lee C-H, Bonnani L, Selker T (2005) Augmented reality kitchen: enhancing human sensibility in domestic life. In: ACM SIGGRAPH 2005 posters. ACM, p 60

    Google Scholar 

  33. Lieberman H (2001) Your wish is my command: programming by example. Morgan Kaufmann

    Google Scholar 

  34. Linder N, Maes P (2010) LuminAR: portable robotic augmented reality interface design and prototype. In: Adjunct proceedings of the 23nd annual ACM symposium on user interface software and technology. ACM, pp 395–396

    Google Scholar 

  35. Lucke D, Constantinescu C, Westkämper E (2008) Smart factory-a step towards the next generation of manufacturing. In: Manufacturing systems and technologies for the new frontier. Springer, pp 115–118

    Google Scholar 

  36. Marinos D, Wöldecke B, Geiger C (2013) Prototyping natural interactions in virtual studio environments by demonstration: combining spatial mapping with gesture following. In: Proceedings of the virtual reality international conference: laval virtual. ACM, p 2

    Google Scholar 

  37. Myers BA (1986) Creating dynamic interaction techniques by demonstration. In: ACM SIGCHI bulletin 17.SI, pp 271–278 (1986)

    Google Scholar 

  38. Pinhanez C (2001) The everywhere displays projector: a device to create ubiquitous graphical interfaces. In: Ubicomp 2001: ubiquitous computing. Springer, pp 315–331

    Google Scholar 

  39. Rüther S, Hermann T, Mracek M, Kopp S, Steil J (2013) An assistance system for guiding workers in central sterilization supply departments. In: Proceedings of the 6th international conference on pervasive technologies related to assistive environments. ACM, p 3

    Google Scholar 

  40. Salonen T, Sääski J, Hakkarainen M, Kannetis T, Perakakis M, Siltanen S, Potamianos A, Korkalo O, Woodward C (2007) Demonstration of assembly work using augmented reality. In: Proceedings of the 6th ACM international conference on image and video retrieval. ACM, pp 120–123

    Google Scholar 

  41. Schmidt A (2000) Implicit human computer interaction through context. Pers Technol 4(2–3):191–199

    Google Scholar 

  42. Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 73–80

    Google Scholar 

  43. Ward JA, Lukowicz P, Troster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567

    Article  Google Scholar 

  44. Wilson AD (2004) TouchLight: an imaging touch screen and display for gesture-based interaction. In: Proceedings of the 6th international conference on Multimodal interfaces. ACM, pp 69–76

    Google Scholar 

  45. Wilson AD (2010) Using a depth camera as a touch sensor. In: ACM international conference on interactive tabletops and surfaces. ACM, pp 69–72

    Google Scholar 

  46. Wohlgemuth W, Triebfürst G (2000) ARVIKA: augmented reality for development, production and service. In: Proceedings of DARE 2000 on designing augmented reality environments. ACM, pp 151–152

    Google Scholar 

  47. Zauner J, Haller M, Brandl A, Hartman W (2003) Authoring of a mixed reality assembly instructor for hierarchical structures. In: The second IEEE and ACM international symposium on mixed and augmented reality, 2003. Proceedings. IEEE, pp 237–246

    Google Scholar 

  48. Zollner R, Rogalla O, Dillmann R, Zollner M (2002) Understanding users intention: programming fine manipulation tasks by demonstration. In: IEEE/RSJ international conference on intelligent robots and systems, 2002, vol 2. IEEE, pp 1114–1119

    Google Scholar 

Download references

Acknowledgements

This work is funded by the German Federal Ministry for Economic Affairs and Energy in the project motionEAP [20], grant no. 01MT12021E. We thank Mathias Hoppe for his work in helping to implement the software and conducting the user study. We further thank Klaus Klein, Michael Spreng, and Johann Hegel from Audi AG.

Author information

Authors and Affiliations

  1. University of Stuttgart, Pfaffenwaldring 5a, 70569, Stuttgart, Germany

    Markus Funk, Lars Lischke, Sven Mayer & Albrecht Schmidt

  2. Yahoo Inc., 701 1st Ave, Sunnyvale, CA, 94089, USA

    Alireza Sahami Shirazi

Authors
  1. Markus Funk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lars Lischke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sven Mayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alireza Sahami Shirazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Albrecht Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Funk .

Editor information

Editors and Affiliations

  1. Synaptics, Zug, Switzerland

    Jochen Huber

  2. Department of Computer Science, Stony Brook University, New York, New York, USA

    Roy Shilkrot

  3. Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, USA

    Pattie Maes

  4. Augmented Human Lab, Singapore University of Technology and Design, Singapore, Singapore

    Suranga Nanayakkara

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Funk, M., Lischke, L., Mayer, S., Shirazi, A.S., Schmidt, A. (2018). Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection. In: Huber, J., Shilkrot, R., Maes, P., Nanayakkara, S. (eds) Assistive Augmentation. Cognitive Science and Technology. Springer, Singapore. https://doi.org/10.1007/978-981-10-6404-3_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-6404-3_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-6402-9

  • Online ISBN: 978-981-10-6404-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation