Abstract
Dialogue intent detection and semantic slot filling are two critical tasks in nature language understanding (NLU) for task-oriented dialog systems. In this paper, we present an attention-based encoder-decoder neural network model for joint intent detection and slot filling, which encodes sentence representation with a hybrid Convolutional Neural Networks and Bidirectional Long Short-Term Memory Networks (CNN-BLSTM), and decodes it with an attention-based recurrent neural network with aligned inputs. In the encoding process, our model firstly extracts higher-level phrase representations and local features from each utterance using convolutional neural network, and then propagates historical contextual semantic information with a bidirectional long short-term memory network layer architecture. Accordingly, we could obtain sentence representation by merging the two architectures mentioned above. In the decoding process, we introduce attention mechanism in long short-term memory networks that can provide additional sematic information. We conduct experiment on dialogue intent detection and slot filling tasks with standard data set Airline Travel Information System (ATIS). Experimental results manifest that our proposed model can achieve better overall performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Shen, B., Inkpen, D.: Speech intent recognition for robots (2017)
Liu, B., Lane, I.: Attention-based recurrent neural network models for joint intent detection and slot filling (2016)
Liu, B., Line, I.: Recurrent neural network structured output prediction for spoken language understanding (2015)
Kim, Y.: Convolutional neural networks for sentence classification. Eprint Arxiv (2014)
Zhang, X., Wang, H.: A joint model of intent determination and slot filling for spoken language understanding, pp. 5690–5694 (2016)
Haffner, P., Tur, G., Wright, J.H.: Optimizing SVMs for complex call classification (2003)
Chen, J., Huang, H., Tian, S., et al.: Feature selection for text classification with Naive Bayes. Expert Syst. Appl. Int. J. 36(3), 5432–5435 (2009)
Graves, A., Jaitly, N., Mohamed, A.R.: Hybrid speech recognition with deep bidirectional LSTM (2014)
Xiao, Y., Cho, K.: Efficient character-level document classification by combining convolution and recurrent layers (2016)
Xiao, J., Wang, X., Liu, B.: The study of a nonstationary maximum entropy Markov model and its application on the pos-tagging task. ACM Trans. Asian Lang. Inf. Process. 6(2), 7 (2007)
Raymond, C., Riccardi, G.: Generative and discriminative algorithms for spoken language understanding (2007)
Aliannejadi, M., Kiaeeha, M., Khadivi, S., et al.: Graph-based semi-supervised conditional random fields for spoken language understanding using unaligned data (2017)
Xu, P., Sarikaya, R.: Convolutional neural network based triangular CRF for joint intent detection and slot filling (2014)
Yao, K., Peng, B., Zhang, Y., et al.: Spoken language understanding using long short-term memory neural networks (2015)
Vu, N.T., Gupta, P., Adel, H., et al.: Bi-directional recurrent neural network with ranking loss for spoken language understanding (2016)
Kurata, G., Xiang, B., Zhou, B., et al.: Leveraging sentence-level information with encoder LSTM for natural language understanding (2016)
Zhu, S., Yu, K.: Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding (2017)
Guo, D., Tur, G., Yih, W.T., et al.: Joint semantic utterance classification and slot filling with recursive neural networks (2015)
Liu, B., Lane, I.: Joint online spoken language understanding and language modeling with recurrent neural networks (2016)
Weigelt, S., Hey, T., Landhäußer, M.: Integrating a dialog component into a framework for spoken language understanding (2018)
Zhou, C., Sun, C., Liu, Z., et al.: A C-LSTM neural network for text classification. Comput. Sci. 1(4), 39–44 (2015)
Yao, K., Peng, B., Zhang, Y., et al.: Spoken language understanding using long short-term memory neural networks. In: IEEE – Institute of Electrical & Electronics Engineers, pp. 189–194 (2014)
Word2vec Homepage. http://code.google.com/archive/p/word2vec/
Yin, W., Schütze, H., Xiang, B., et al.: ABCNN: attention-based convolutional neural network for modeling sentence pairs (2015)
Morin, F., Bengio, Y.: Hierarchical probabilistic neural network language model. Aistats (2005)
Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks (2013)
Hemphill, C.T., Godfrey, J.J., Doddington, G.R.: The ATIS spoken language systems pilot corpus. In: Proceedings of the Darpa Speech & Natural Language Workshop, pp. 96–101 (1990)
Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: International Conference on Machine Learning, pp. 2342–2350. JMLR.org (2015)
Acknowledgments
This research is supported by the Fundamental Research Funds for the Central Universities (CCNU18JCK05), the National Natural Science Foundation of China (61532008), the National Science Foundation of China (61572223), the National Key Research and Development Program of China (2017YFC0909502).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Tang, L., He, T. (2018). Attention-Based CNN-BLSTM Networks for Joint Intent Detection and Slot Filling. In: Sun, M., Liu, T., Wang, X., Liu, Z., Liu, Y. (eds) Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. CCL NLP-NABD 2018 2018. Lecture Notes in Computer Science(), vol 11221. Springer, Cham. https://doi.org/10.1007/978-3-030-01716-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-01716-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01715-6
Online ISBN: 978-3-030-01716-3
eBook Packages: Computer ScienceComputer Science (R0)