Towards Robust Neural Networks with Lipschitz Continuity

  • Muhammad Usama
  • Dong Eui ChangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11378)


Deep neural networks have shown remarkable performance across a wide range of vision-based tasks, particularly due to the availability of large-scale datasets for training and better architectures. However, data seen in the real world are often affected by distortions that not accounted for by the training datasets. In this paper, we address the challenge of robustness and stability of neural networks and propose a general training method that can be used to make the existing neural network architectures more robust and stable to input visual perturbations while using only available datasets for training. Proposed training method is convenient to use as it does not require data augmentation or changes in the network architecture. We provide theoretical proof as well as empirical evidence for the efficiency of the proposed training method by performing experiments with existing neural network architectures and demonstrate that same architecture when trained with the proposed training method perform better than when trained with conventional training approach in the presence of noisy datasets.


Deep neural networks Robust neural networks Lipschitz continuity 



This research has been in part supported by the ICT R&D program of MSIP/IITP [2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion].


  1. 1.
    Szegedy, C., et al.: Going deeper with convolutions. In: CVPR, June 2015Google Scholar
  2. 2.
    Wang, F., et al.: Residual attention network for image classification. CoRR, abs/1704.06904 (2017)Google Scholar
  3. 3.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, June 2016Google Scholar
  4. 4.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) NIPS 2012, pp. 1097–1105. Curran Associates Inc. (2012)Google Scholar
  5. 5.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR, June 2016Google Scholar
  6. 6.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) NIPS 2015, pp. 91–99. Curran Associates Inc. (2015)Google Scholar
  7. 7.
    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. CoRR, abs/1411.4555 (2014)Google Scholar
  8. 8.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  9. 9.
    Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550, 354 (2017)CrossRefGoogle Scholar
  10. 10.
    Caterini, A.L., Chang, D.E.: Deep Neural Networks in a Mathematical Framework. SCS. Springer, Cham (2018). Scholar
  11. 11.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255, June 2009Google Scholar
  12. 12.
    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  13. 13.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. CoRR, abs/1603.05027 (2016)Google Scholar
  14. 14.
    Paszke, A., et al.: Automatic differentiation in pytorch, Alban Desmaison (2017)Google Scholar
  15. 15.
    Abadi, M., et al.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015). Software
  16. 16.
    Dodge, S.F., Karam, L.J.: Understanding how image quality affects deep neural networks. CoRR, abs/1604.04004 (2016)Google Scholar
  17. 17.
    Vasiljevic, I., Chakrabarti, A., Shakhnarovich, G.: Examining the impact of blur on recognition by convolutional networks. CoRR, abs/1611.05760 (2016)Google Scholar
  18. 18.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  19. 19.
    Rozsa, A., Günther, M., Boult, T.E.: Towards robust deep neural networks with BANG. CoRR, abs/1612.00138 (2016)Google Scholar
  20. 20.
    Zheng, S., Song, Y., Leung, T., Goodfellow, I.J.: Improving the robustness of deep neural networks via stability training. CoRR, abs/1604.04326 (2016)Google Scholar
  21. 21.
    Zhou, Y., Song, S., Cheung, N.-M.: On classification of distorted images with deep convolutional neural networks. CoRR, abs/1701.01924 (2017)Google Scholar
  22. 22.
    Dodge, S.F., Karam, L.J.: Quality resilient deep neural networks. CoRR, abs/1703.08119 (2017)Google Scholar
  23. 23.
    Szegedy, C., et al.: Intriguing properties of neural networks. CoRR, abs/1312.6199 (2013)Google Scholar
  24. 24.
    Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (2015)Google Scholar
  25. 25.
    LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010)Google Scholar
  26. 26.
    Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report (2009)Google Scholar
  27. 27.
    Coates, A., Ng, A., Lee. H.: An analysis of single-layer networks in unsupervised feature learning. In: Gordon, G., Dunson, D., Dudk, M. (eds.) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, vol. 15 of Proceedings of Machine Learning Research, pp. 215–223, Fort Lauderdale, FL, USA, 11–13 Apr 2011. PMLR (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Electrical EngineeringKorea Advanced Institute of Science and TechnologyDaejeonRepublic of Korea

Personalised recommendations