Advertisement

An Interpretation of Forward-Propagation and Back-Propagation of DNN

  • Guotian Xie
  • Jianhuang LaiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11257)

Abstract

Deep neural network (DNN) is hard to understand because the objective loss function is defined on the last layer, not directly on the hidden layers. To best understand DNN, we interpret the forward-propagation and back-propagation of DNN as two network structures, fp-DNN and bp-DNN. Then we introduce the direct loss function for hidden layers of fp-DNN and bp-DNN, which gives a way to interpret the fp-DNN as an encoder and bp-DNN as a decoder. Using this interpretation of DNN, we do experiments to analyze that fp-DNN learns to encode discriminant features in the hidden layers with the supervision of bp-DNN. Further, we use bp-DNN to visualize and explain DNN. Our experiments and analyses show the proposed interpretation of DNN is a good tool to understand and analyze the DNN.

Keywords

Forward-propagation Back-propagation Encoder Decoder 

Notes

Acknowledgments

This project is supported by the Natural Science Foundation of China (61573387) and Guangdong Project (2017B030306018).

References

  1. 1.
    An, S., Boussaid, F., Bennamoun, M., Hu, J.: From deep to shallow: transformations of deep rectifier networks. arXiv preprint arXiv:1703.10355 (2017)
  2. 2.
    Arik, S.O., et al.: Deep voice: real-time neural text-to-speech. arXiv preprint arXiv:1702.07825 (2017)
  3. 3.
    Bengio, Y.: How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906 (2014)
  4. 4.
    Bengio, Y., et al.: Learning deep architectures for Ai. Found. Trends® Mach. Learn. 2(1), 1–127 (2009)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  8. 8.
    Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext.zip: compressing text classification models. arXiv preprint arXiv:1612.03651 (2016)
  9. 9.
    Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730 (2017)
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  11. 11.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  12. 12.
    Lee, D.-H., Zhang, S., Fischer, A., Bengio, Y.: Difference target propagation. In: Appice, A., Rodrigues, P.P., Santos Costa, V., Soares, C., Gama, J., Jorge, A. (eds.) ECML PKDD 2015. LNCS (LNAI), vol. 9284, pp. 498–515. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-23528-8_31CrossRefGoogle Scholar
  13. 13.
    Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37, 421–436 (2016).  https://doi.org/10.1177/0278364917710318CrossRefGoogle Scholar
  14. 14.
    Lillicrap, T.P., Cownden, D., Tweed, D.B., Akerman, C.J.: Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:1411.0247 (2014)
  15. 15.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)zbMATHGoogle Scholar
  16. 16.
    Nøkland, A.: Direct feedback alignment provides learning in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 1037–1045 (2016)Google Scholar
  17. 17.
    Pei, K., Cao, Y., Yang, J., Jana, S.: Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, pp. 1–18. ACM (2017)Google Scholar
  18. 18.
    Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. arXiv preprint arXiv:1612.08242 (2016)
  19. 19.
    Scellier, B., Bengio, Y.: Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Front. Comput. Neurosci. 11, 24 (2017)CrossRefGoogle Scholar
  20. 20.
    Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)
  21. 21.
    Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.The School of Data and Computer ScienceSun Yat-sen UniversityGuangzhouChina
  2. 2.Guangdong Key Laboratory of Information Security TechnologyGuangzhouChina
  3. 3.The School of Information Science and Technology, Xinhua CollegeSun Yat-sen UniversityGuangzhouPeople’s Republic of China

Personalised recommendations