Towards Explaining Deep Neural Networks Through Graph Analysis
Due to its potential to solve complex tasks, deep learning is being used across many different areas. The complexity of neural networks however makes it difficult to explain the whole decision process used by the model, which makes understanding deep learning models an active research topic. In this work we address this issue by extracting the knowledge acquired by trained Deep Neural Networks (DNNs) and representing this knowledge in a graph. The proposed graph encodes statistical correlations between neurons’ activation values in order to expose the relationship between neurons in the hidden layers with both the input layer and output classes. Two initial experiments in image classification were conducted to evaluate whether the proposed graph can help understanding and explaining DNNs. We first show how it is possible to explore the proposed graph to find what neurons are the most important for predicting each class. Then, we use graph analysis to detect groups of classes that are more similar to each other and how these similarities affect the DNN. Finally, we use heatmaps to visualize what parts of the input layer are responsible for activating each neuron in hidden layers. The results show that by building and analysing the proposed graph it is possible to gain relevant insights of the DNN’s inner workings.
KeywordsExplainable AI Deep learning Graph analysis
The Insight Centre for Data Analytics is supported by Science Foundation Ireland under Grant Number 17/RC-PhD/3483.
- 3.Chan, V., Chan, C.W.: Development and application of an algorithm for extracting multiple linear regression equations from artificial neural networks for nonlinear regression problems. In: 2016 IEEE 15th International Conference on Cognitive Informatics Cognitive Computing (ICCI*CC), pp. 479–488, August 2016Google Scholar
- 4.Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Technical report. Univeristé de Montréal, January 2009Google Scholar
- 6.Garcia-Gasulla, D., et al.: Building graph representations of deep vector embeddings. CoRR abs/1707.07465 (2017). http://arxiv.org/abs/1707.07465
- 7.Garcia-Gasulla, D., et al.: An out-of-the-box full-network embedding for convolutional neural networks. In: 2018 IEEE International Conference on Big Knowledge (ICBK), pp. 168–175 (2018)Google Scholar
- 10.LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist/
- 11.Liu, B., Wei, Y., Zhang, Y., Yang, Q.: Deep neural networks for high dimension, low sample size data, pp. 2287–2293, August 2017. https://doi.org/10.24963/ijcai.2017/318
- 14.Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. CoRR abs/1312.6034 (2013). http://arxiv.org/abs/1312.6034
- 15.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). http://arxiv.org/abs/1409.1556
- 16.Towell, G.G., Shavlik, J.W.: Extracting refined rules from knowledge-based neural networks. Mach. Learn. 13(1), 71–101 (1993)Google Scholar
- 17.Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)Google Scholar