Meaningful Clusterings of Recurrent Neural Network Activations for NLP

  • Mihai PomarlanEmail author
  • John Bateman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11308)


Recurrent neural networks have found applications in NLP, but their operation is difficult to interpret. A state automaton that approximates the network would be more interpretable, but for this one needs a method to group network activation states by their behavior. In this paper we propose such a method, and compare it to an existing dimensionality reduction and clustering approach. Our method is better able to group together neural states of similar behavior.


Recurrent neural networks Natural language processing Interpretability 


  1. 1.
    Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., Goldberg, Y.: Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR abs/1608.04207 (2016)Google Scholar
  2. 2.
    Angluin, D.: Identifying languages from stochastic examples. Technical report, YALEU/ DCS/RR-614, Yale University, Department of Computer Science, New Haven, CT (1988)Google Scholar
  3. 3.
    Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2015)Google Scholar
  4. 4.
    Bowman, S.R., Potts, C., Manning, C.D.: Recursive neural networks for learning logical semantics. CoRR abs/1406.1827 (2014)Google Scholar
  5. 5.
    Choo, J., Liu, S.: Visual analytics for explainable deep learning. CoRR abs/1804.02527 (2018)CrossRefGoogle Scholar
  6. 6.
    Carrasco, R.C., Oncina, J.: Learning deterministic regular grammars from stochastic samples in polynomial time. ITA 33(1), 1–20 (1999)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Cohen, M., Caciularu, A., Rejwan, I., Berant, J.: Inducing regular grammars using recurrent neural networks. CoRR abs/1710.10453 (2017)Google Scholar
  8. 8.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. AAAI Press (1996)Google Scholar
  9. 9.
    Evans, R., Saxton, D., Amos, D., Kohli, P., Grefenstette, E.: Can neural networks understand logical entailment? In: Intntl Conf on Learning Representations (2018)Google Scholar
  10. 10.
    Giles, C.L., Miller, C.B., Chen, D., Sun, G.Z., Chen, H.H., Lee, Y.C.: Extracting and learning an unknown grammar with recurrent neural networks. In: Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS 1991. Morgan Kaufmann Publishers Inc., San Francisco (1991)Google Scholar
  11. 11.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  12. 12.
    Hohman, F., Kahng, M., Pienta, R., Chau, D.H.: Visual analytics in deep learning: An interrogative survey for the next frontiers. CoRR abs/1801.06889 (2018)Google Scholar
  13. 13.
    Kansky, K., et al.: Schema networks: zero-shot transfer with a generative causal model of intuitive physics. In: ICML (2017)Google Scholar
  14. 14.
    Karpathy, A.: The unreasonable effectiveness of recurrent neural networks. Accessed 29 Jan 2018
  15. 15.
    Karpathy, A., Johnson, J., Fei-Fei, L.: Visualizing and understanding recurrent networks. CoRR abs/1506.02078 (2015)Google Scholar
  16. 16.
    Kolesnyk, V., Rocktäschel, T., Riedel, S.: Generating natural language inference chains. CoRR abs/1606.01404 (2016)Google Scholar
  17. 17.
    Li, J., Chen, X., Hovy, E., Jurafsky, D.: Visualizing and understanding neural models in NLP. In: Proceedings of NAACL-HLT, pp. 681–691 (2016)Google Scholar
  18. 18.
    Li, J., Monroe, W., Jurafsky, D.: Understanding neural networks through representation erasure. CoRR abs/1612.08220 (2016)Google Scholar
  19. 19.
    Linzen, T., Dupoux, E., Goldberg, Y.: Assessing the ability of LSTMs to learn syntax-sensitive dependencies. TACL 4, 521–535 (2016)Google Scholar
  20. 20.
    Ming, Y., et al.: Understanding hidden memories of recurrent neural networks. CoRR abs/1710.10777 (2017)Google Scholar
  21. 21.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  22. 22.
    Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should i trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York (2016)Google Scholar
  23. 23.
    Strobelt, H., Gehrmann, S., Huber, B., Pfister, H., Rush, A.M.: Visual analysis of hidden state dynamics in recurrent neural networks. CoRR abs/1606.07461 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of BremenBremenGermany

Personalised recommendations