Improving ML Safety with Partial Specifications

  • Rick SalayEmail author
  • Krzysztof Czarnecki
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11699)


Advanced autonomy features of vehicles are typically difficult or impossible to specify precisely and this has led to the rise of machine learning (ML) from examples as an alternative implementation approach to traditional programming. Developing software without specifications sacrifices the ability to effectively verify the software yet this is a key component of safety assurance. In this paper, we suggest that while complete specifications may not be possible, partial specifications typically are and these could be used with ML to strengthen safety assurance. We review the types of partial specifications that are applicable for these problems and discuss the places in the ML development workflow that they could be used to improve the safety of ML-based components.


Safety Machine learning Specification 



We would like to thank Mark Costin for insightful comments that have contributed to this work.


  1. 1.
    Bhattacharyya, S., Cofer, D., Musliner, D., Mueller, J., Engstrom, E.: Certification considerations for adaptive systems. In: 2015 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 270–279. IEEE (2015)Google Scholar
  2. 2.
    Cohen, T., Welling, M.: Group equivariant convolutional networks. In: International Conference on Machine Learning, pp. 2990–2999 (2016)Google Scholar
  3. 3.
    Cooke, D., Gates, A., Demirörs, E., Demirörs, O., Tanik, M.M., Krämer, B.: Languages for the specification of software. J. Syst. Softw. 32(3), 269–308 (1996)CrossRefGoogle Scholar
  4. 4.
    Czarnecki, K., Salay, R.: Towards a framework to manage perceptual uncertainty for safe automated driving. In: Gallina, B., Skavhaug, A., Schoitsch, E., Bitsch, F. (eds.) SAFECOMP 2018. LNCS, vol. 11094, pp. 439–445. Springer, Cham (2018). Scholar
  5. 5.
    Dreossi, T., Donzé, A., Seshia, S.A.: Compositional falsification of cyber-physical systems with machine learning components. In: Barrett, C., Davies, M., Kahsai, T. (eds.) NFM 2017. LNCS, vol. 10227, pp. 357–372. Springer, Cham (2017). Scholar
  6. 6.
    Dwarakanath, A., et al.: Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 118–128. ACM (2018)Google Scholar
  7. 7.
    Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.: Ai2: safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 3–18. IEEE (2018)Google Scholar
  8. 8.
    Harnad, S.: The symbol grounding problem. Physica D 42(1–3), 335–346 (1990)CrossRefGoogle Scholar
  9. 9.
    Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada. Conference Track Proceedings, 30 April–3 May 2018.
  10. 10.
    International Organization for Standardization: ISO 26262: Road Vehicles - Functional Safety, 2nd edition (2018)Google Scholar
  11. 11.
    International Organization for Standardization: ISO/AWI PAS 21448: Road Vehicles - Safety of the Intended Functionality, 1st Edition (2019)Google Scholar
  12. 12.
    Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). Scholar
  13. 13.
    Koopman, P., Wagner, M.: Challenges in autonomous vehicle testing and validation. SAE Int. J. Transp. Saf. 4(1), 15–24 (2016)CrossRefGoogle Scholar
  14. 14.
    Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.L.: Joint 3D proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ IROS, pp. 1–8. IEEE (2018)Google Scholar
  15. 15.
    Lakoff, G.: Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. University of Chicago press, Chicago (1987)Google Scholar
  16. 16.
    Lamsweerde, A.V.: Formal specification: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering, pp. 147–159. ACM (2000)Google Scholar
  17. 17.
    Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)Google Scholar
  18. 18.
    Meel, K.S., et al.: Constrained sampling and counting: universal hashing meets SAT solving. In: Workshops at the Thirtieth AAAI Conference on Artificial Intelligence (2016)Google Scholar
  19. 19.
    Meyer, B.: Applying ‘design by contract’. Computer 25(10), 40–51 (1992)CrossRefGoogle Scholar
  20. 20.
    Reiter, R.: A logic for default reasoning. Artif. Intell. 13(1–2), 81–132 (1980)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Rouder, J.N., Ratcliff, R.: Comparing exemplar and rule-based theories of categorization. Curr. Dir. Psychol. Sci. 15(1), 9–13 (2006)CrossRefGoogle Scholar
  22. 22.
    von Rueden, L., Mayer, S., Garcke, J., Bauckhage, C., Schuecker, J.: Informed machine learning-towards a taxonomy of explicit integration of knowledge into machine learning. arXiv preprint arXiv:1903.12394 (2019)
  23. 23.
    Salay, R., Czarnecki, K.: Using machine learning safely in automotive software: An assessment and adaption of software process requirements in ISO 26262. arXiv preprint arXiv:1808.01614 (2018)
  24. 24.
    Salay, R., Queiroz, R., Czarnecki, K.: An Analysis of ISO 26262: Machine Learning and Safety in Automotive Software. SAE Technical Paper (2018)Google Scholar
  25. 25.
    Seshia, S.A., Sadigh, D., Sastry, S.S.: Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514 (2016)
  26. 26.
    Sha, L.: Using simplicity to control complexity. IEEE Softw. 4, 20–28 (2001)Google Scholar
  27. 27.
    Spanfelner, B., Richter, D., Ebel, S., Wilhelm, U., Branz, W., Patz, C.: Challenges in applying the ISO 26262 for driver assistance systems. Tagung Fahrerassistenz, München 15(16), 2012 (2012)Google Scholar
  28. 28.
    Varshney, K.R.: Engineering safety in machine learning. arXiv preprint arXiv:1601.04126 (2016)
  29. 29.
    Vedaldi, A., Blaschko, M., Zisserman, A.: Learning equivariant structured output SVM regressors. In: Proceedings of 2011 International Conference on Computer Vision, pp. 959–966. IEEE (2011)Google Scholar
  30. 30.
    Wang, J., Perez, L.: The effectiveness of data augmentation in image classification using deep learning. In: Convolutional Neural Networks Vision Recognition (2017)Google Scholar
  31. 31.
    Wong, S.C., Gatt, A., Stamatescu, V., McDonnell, M.D.: Understanding data augmentation for classification: when to warp? In: 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–6. IEEE (2016)Google Scholar
  32. 32.
    Worrall, D.E., Garbin, S.J., Turmukhambetov, D., Brostow, G.J.: Harmonic networks: deep translation and rotation equivariance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028–5037 (2017)Google Scholar
  33. 33.
    Xu, J., Zhang, Z., Friedman, T., Liang, Y., Broeck, G.V.D.: A semantic loss function for deep learning with symbolic knowledge. arXiv preprint arXiv:1711.11157 (2017)
  34. 34.
    Yan, J., Zhang, X., Lei, Z., Liao, S., Li, S.Z.: Robust multi-resolution pedestrian detection in traffic scenes. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3033–3040. IEEE (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of WaterlooWaterlooCanada

Personalised recommendations