Enhancing Process Data in Manual Assembly Workflows

  • Sönke KnochEmail author
  • Nico Herbig
  • Shreeraman Ponpathirkoottam
  • Felix Kosmalla
  • Philipp Staudt
  • Peter Fettke
  • Peter Loos
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 342)


The rise of Industry 4.0 and the convergence with BPM provide new potential for the automatic gathering of process-related sensor information. In manufacturing, information about human behavior in manual assembly tasks is rare when no interaction with machines is involved. We suggest technologies to automatically detect material picking and placement in the assembly workflow to gather accurate data about human behavior. For material picking, we use background subtraction; for placement detection image classification with neural networks is applied. The detected fine-grained worker activities are then correlated to a BPMN model of the assembly workflow, enabling the measurement of production time (time per state) and quality (frequency of error) on the shop floor as an entry point for conformance checking and process optimization. The approach has been evaluated in a quantitative case study recording the assembly process 30 times in a laboratory within 4 h. Under these conditions, the classification of assembly states with a neural network provides a test accuracy of 99.25% on 38 possible assembly states. Material picking based on background subtraction has been evaluated in an informal user study with 6 participants performing 16 picks, each providing an accuracy of 99.48%. The suggested method is promising to easily detect fine-grained steps in manufacturing augmenting and checking the assembly workflow.


Manual assembly Computer vision BPM Industry 4.0 



This research was funded in part by the German Federal Ministry of Education and Research (BMBF) under grant number 01IS16022E (project BaSys4.0). The responsibility for this publication lies with the authors.


  1. 1.
    Cavanillas, J.M., Curry, E., Wahlster, W. (eds.): New Horizons for a Data-Driven Economy: A Roadmap for Usage and Exploitation of Big Data in Europe. Springer, Cham (2016). Scholar
  2. 2.
    Grzeszick, R., et al.: Deep neural network based human activity recognition for the order picking process. In: Proceedings of the 4th International Workshop on Sensor-based Activity Recognition and Interaction. iWOAR 2017, pp. 14:1–14:6. ACM Rostock (2017)Google Scholar
  3. 3.
    He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  4. 4.
    Hull, R., Motahari Nezhad, H.R.: Rethinking BPM in a cognitive world: transforming how we learn and perform business processes. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 3–19. Springer, Cham (2016). Scholar
  5. 5.
    Jaroucheh, Z., Liu, X., Smith, S.: Recognize contextual situation in pervasive environments using process mining techniques. J. Ambient Intell. Humaniz. Comput. 2(1), 53–69 (2011)CrossRefGoogle Scholar
  6. 6.
    Kagermann, H., et al.: Recommendations for implementing the strategic initiative INDUSTRIE 4.0: Securing the Future of German Manufacturing Industry; Final Report of the Industrie 4.0 Working Group. Forschungsunion (2013)Google Scholar
  7. 7.
    Knoch, S., et al.: Automatic capturing and analysis of manual manufacturing processes with minimal setup effort. In: International Joint Conference on Pervasive and Ubiquitous Computing. UbiComp, pp. 305–308. ACM, Heidelberg, September 2016Google Scholar
  8. 8.
    Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.: Technology-enhanced process elicitation of worker activities in manufacturing. In: Teniente, E., Weidlich, M. (eds.) BPM 2017. LNBIP, vol. 308, pp. 273–284. Springer, Cham (2018). Scholar
  9. 9.
    Lasi, H., et al.: Industrie 4.0. Wirtschaftsinformatik 56(4), 261–264 (2014)CrossRefGoogle Scholar
  10. 10.
    Lenz, C., et al.: Human workflow analysis using 3D occupancy grid hand tracking in a human-robot collaboration scenario. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3375–3380, September 2011Google Scholar
  11. 11.
    Marrella, A., Mecella, M.: Cognitive business process management for adaptive cyber-physical processes. In: Teniente, E., Weidlich, M. (eds.) BPM 2017. LNBIP, vol. 308, pp. 429–439. Springer, Cham (2018). Scholar
  12. 12.
    Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28(6), 976–990 (2010)CrossRefGoogle Scholar
  13. 13.
    Roitberg, A., et al.: Multimodal human activity recognition for industrial manufacturing processes in robotic workcells. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ICMI 2015. pp. 259–266. ACM, Seattle (2015)Google Scholar
  14. 14.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Stiefmeier, T., et al.: Wearable activity tracking in car manufacturing. IEEE Pervasive Comput. 7(2), 42–50 (2008)CrossRefGoogle Scholar
  16. 16.
    Thoben, K.-D., Pöppelbuß, J., Wellsandt, S., Teucke, M., Werthmann, D.: Considerations on a lifecycle model for cyber-physical system platforms. In: Grabot, B., Vallespir, B., Gomes, S., Bouras, A., Kiritsis, D. (eds.) APMS 2014, Part I. IAICT, vol. 438, pp. 85–92. Springer, Heidelberg (2014). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Sönke Knoch
    • 1
    Email author
  • Nico Herbig
    • 1
  • Shreeraman Ponpathirkoottam
    • 1
  • Felix Kosmalla
    • 1
  • Philipp Staudt
    • 1
  • Peter Fettke
    • 1
  • Peter Loos
    • 1
  1. 1.German Research Center for Artificial Intelligence (DFKI), Saarland Informatics CampusSaarland UniversitySaarbrückenGermany

Personalised recommendations