Increasing Safety by Combining Multiple Declarative Rules in Robotic Perception Systems

  • Johann Thor Mogensen Ingibergsson
  • Dirk Kraft
  • Ulrik Pagh SchultzEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11267)


Advanced cyber-physical systems such as mobile, networked robots are increasingly finding use in everyday society. A critical aspect of mobile robotics is the ability to react to a dynamically changing environment, which imposes significant requirements on the robot perception system. The perception system is key to maintaining safe navigation and operation for the robot and is often considered a safety-critical aspect of the system as a whole. To allow the system to operate in a public area the perception system thus has to be certified. The key issue that we address is how to have safety-compliant systems while keeping implementation transparency high and complexity low. In this paper we present an evaluation of different methods for modelling combinations of simple explicit computer vision rules designed to increase the trustworthiness of the perception system. We utilise the best-performing method, focusing on keeping the models of the perception pipeline transparent and understandable. We find that it is possible to improve the safety of the system with some performance cost, depending on the acceptable risk level.


Safety Computer vision Robotics Functional safety 


  1. 1.
    Baheti, R., Gill, H.: Cyber-physical systems. Impact Control Technol. 12, 161–166 (2011)Google Scholar
  2. 2.
    Bansal, A., Farhadi, A., Parikh, D.: Towards transparent systems: semantic characterization of failure modes. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 366–381. Springer, Cham (2014). Scholar
  3. 3.
    Daigle, M.J., Koutsoukos, X.D., Biswas, G.: Distributed diagnosis in formations of mobile robots. IEEE Trans. Robo. 23(2), 353–369 (2007)CrossRefGoogle Scholar
  4. 4.
    Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006)Google Scholar
  5. 5.
    De Cabrol, A., Garcia, T., Bonnin, P., Chetto, M.: A concept of dynamically reconfigurable real-time vision system for autonomous mobile robotics. Int. J. Autom. Comput. 5(2), 174–184 (2008)CrossRefGoogle Scholar
  6. 6.
    Fields, C., David, R., Nielsen, P.: Defense science board 2016 summer study on autonomy. Defense Science Board (2016)Google Scholar
  7. 7.
    Frese, U., Hirschmüller, H.: Special issue on robot vision: what is robot vision? J. Real-Time Image Process. 10(4), 597–598 (2015)CrossRefGoogle Scholar
  8. 8.
    Gupta, P., Loparo, K., Mackall, D., Schumann, J., Soares, F.: Verification and validation methodology of real-time adaptive neural networks for aerospace applications. In: International Conference on Computational Intelligence for Modeling, Control, and Automation (2004)Google Scholar
  9. 9.
    Hauge, A., Tonnesen, A.: Use of artificial neural networks in safety critical systems. Faculty of Computer Sciences (2004)Google Scholar
  10. 10.
    Heckemann, K., Gesell, M., Pfister, T., Berns, K., Schneider, K., Trapp, M.: Safe automotive software. In: König, A., Dengel, A., Hinkelmann, K., Kise, K., Howlett, R.J., Jain, L.C. (eds.) KES 2011. LNCS (LNAI), vol. 6884, pp. 167–176. Springer, Heidelberg (2011). Scholar
  11. 11.
    IFR: World Robotics 2014 Industrial Robots (2014)Google Scholar
  12. 12.
    Ingibergsson, J.T.M., Schultz, U.P., Kuhrmann, M.: On the use of safety certification practices in autonomous field robot software development: a systematic mapping study. In: Abrahamsson, P., Corral, L., Oivo, M., Russo, B. (eds.) PROFES 2015. LNCS, vol. 9459, pp. 335–352. Springer, Cham (2015). Scholar
  13. 13.
    Ingibergsson, J.T.M., Hanenberg, S., Sunshine, J., Schultz, U.P.: Readability study of a domain specific language: process and outcome. In: Accepted for the 33rd ACM/SIGAPP Symposium on Applied Computing (SAC-18) (2018)Google Scholar
  14. 14.
    Ingibergsson, J.T.M., Kraft, D., Schultz, U.P.: Declarative rule-based safety for robotic perception systems. J. Software Eng. Rob. (JOSER) 8(1), 17–31 (2017)Google Scholar
  15. 15.
    Ingibergsson, J.T.M., Kraft, D., Schultz, U.P.: Explicit image quality detection rules for functional safety in computer vision. In: 12th International Conference on Computer Vision Theory and Applications (VISAPP), p. 12, Marts 2017Google Scholar
  16. 16.
    ISO TC22/SC3/WG16. ISO/IEC 26262:2011: Road vehicles - Functional safety. Technical report, International Organization for Standardization (2011)Google Scholar
  17. 17.
    Klarreich, E.: Learning securely. Commun. ACM 59(11), 12–14 (2016)CrossRefGoogle Scholar
  18. 18.
    Kurd, Z., Kelly, T.: Establishing safety criteria for artificial neural networks. In: Palade, V., Howlett, R.J., Jain, L. (eds.) KES 2003. LNCS (LNAI), vol. 2773, pp. 163–169. Springer, Heidelberg (2003). Scholar
  19. 19.
    Kurd, Z., Kelly, T., Austin, J.: Safety criteria and safety lifecycle for artificial neural networks. In: Proceedings of Eunite, vol. 2003 (2003)Google Scholar
  20. 20.
    Machin, M., Dufossé, F., Blanquart, J.-P., Guiochet, J., Powell, D., Waeselynck, H.: Specifying safety monitors for autonomous systems using model-checking. In: Bondavalli, A., Di Giandomenico, F. (eds.) SAFECOMP 2014. LNCS, vol. 8666, pp. 262–277. Springer, Cham (2014). Scholar
  21. 21.
    Mekki-Mokhtar, A., Blanquart, J.-P., Guiochet, J., Powell, D., Roy, M.: Safety trigger conditions for critical autonomous systems. In: 18th Pacific Rim International Symposium on Dependable Computing, pp. 61–69. IEEE (2012)Google Scholar
  22. 22.
    METI: Trends in the Market for the Robot Industry in 2012, July 2013Google Scholar
  23. 23.
    Murphy, R.R., Hershberger, D.: Handling sensing failures in autonomous mobile robots. Int. J. Robot. Res. 18(4), 382–400 (1999)CrossRefGoogle Scholar
  24. 24.
    Myles, A.J., Feudale, R.N., Liu, Y., Woody, N.A., Brown, S.D.: An introduction to decision tree modeling. J. Chemom. 18(6), 275–285 (2004)CrossRefGoogle Scholar
  25. 25.
    Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427–436. IEEE (2015)Google Scholar
  26. 26.
    Redmon, J., Farhadi, A.: YOLO9000: Better, faster, stronger. arXiv preprint arXiv:1612.08242 (2016)
  27. 27.
    Reichardt, M., Föhst, T., Berns, K.: On software quality-motivated design of a real-time framework for complex robot control systems. In: International Workshop on Software Quality and Maintainability (2013)Google Scholar
  28. 28.
    Safavian, S.R., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Trans. Syst. Man Cybern. 21(3), 660–674 (1991)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Saito, T., Rehmsmeier, M.: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. In: PLoS ONE, pp. 1–21 (2015)Google Scholar
  30. 30.
    Santosuosso, A., Boscarato, C., Caroleo, F., Labruto, R., Leroux, C.: Robots, market and civil liability: a european perspective. In: RO-MAN, pp. 1051–1058. IEEE (2012)Google Scholar
  31. 31.
    Schumann, J., Gupta, P., Liu, Y.: Application of neural networks in high assurance systems: a survey. In: Schumann, J., Liu, Y. (eds.) Applications of Neural Networks in High Assurance Systems, pp. 1–19. Springer, Heidelberg (2010). Scholar
  32. 32.
    SDU: Marken er mejet af en robot (2017)Google Scholar
  33. 33.
    Steen, K.A., Christiansen, P., Karstoft, H., Jørgensen, R.N.: Using deep learning to challenge safety standard for highly autonomous machines in agriculture. J. Imaging 2(1), 6 (2016)CrossRefGoogle Scholar
  34. 34.
    TC 127: Earth-moving machinery - autonomous machine system safety. In: International Standard ISO 17757–2015, International Organization for Standardization (2015)Google Scholar
  35. 35.
    TC 23: Agricultural machinery and tractors - Safety of highly automated machinery. International Standard ISO/DIS 18497, International Organization for Standardization (2014)Google Scholar
  36. 36.
    TC 44: Safety of machinery - electro-sensitive protective equipment. International Standard IEC 61496–2012, International Electronical Commission (2012)Google Scholar
  37. 37.
    TC 65: Safety of machinery - electro-sensitive protective equipment. International Standard IEC 61508–2011, International Electronical Commission (2011)Google Scholar
  38. 38.
    Veres, S.M., Lincoln, N.K., Molnar, L.: Control engineering of autonomous cognitive vehicles-a practical tutorial. Technical report, Faculty of Engineering and the Environment, University of Southampton, Technical report (2011)Google Scholar
  39. 39.
    Yang, Y., Keller, P., Livnat, Y., Liggesmeyer, P.: Improving safety-critical systems by visual analysis. In: OASIcs-OpenAccess Series in Informatics, vol. 27. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2012)Google Scholar
  40. 40.
    Zhang, P., Wang, J., Farhadi, A., Hebert, M., Parikh, D.: Predicting failures of vision systems. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3566–3573 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Johann Thor Mogensen Ingibergsson
    • 1
  • Dirk Kraft
    • 1
  • Ulrik Pagh Schultz
    • 1
    Email author
  1. 1.Mærsk Mc-Kinney Møller Institute, University of Southern DenmarkOdense MDenmark

Personalised recommendations