Abstract
Question-Answering (QA) has become one of the most popular natural language processing (NLP) and information retrieval applications. To be applied in QA systems, this paper presents a question classification technique based on NLP and Bidirectional Encoder Representation from Transformers (BERT). We performed experimental investigation on BERT for question classification with TREC-6 dataset and a Thai sentence dataset. We propose an improved processing technique called “More Than Words – BERT” (MTW – BERT) that is a special NLP Annotation tags for combining Part-Of-Speech tagging and Named Entities Recognition to be able for learning both pattern of grammatical tag sequence and recognized entities together as input before classifying text on BERT model. Experimental results showed that MTW – BERT outperformed existing classification methods and achieved new state-of-the-art performance on question classification for TREC-6 dataset with 99.20%. In addition, MTW-BERT also applied for question classification for Thai sentences in wh-question category. The proposed technique remarkably achieved Thai wh-classification with accuracy rate of 87.50%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Vaswani, A., et al.: Atteion, is all you need. In: 31st Conference on Neural Information Processing Systems (NIPS: Long Beach, CA, USA, p. 2017 (2017)
Xin, L., Xuan-Jing, H., Li-d, W.: Question classification using multiple classifiers. In: Proceedings of the Fifth Workshop on Asian Language Resources (ALR-05) and First Symposium on Asian Language Resources Network (ALRN) (2005)
Haryanto, A.W., Mawardi,E.K.: Muljono. influence of word normalization and chi-squared feature selection on support vector machine (SVM) text classification. In: 2018 International Seminar on Application for Technology of Information and Communication, pp. 229–233 (2018). https://doi.org/10.1109/ISEMANTIC.2018.8549748
Silva, V.A., Bittencourt, I.I., Maldonado, J.C.: Automatic question classifiers: a systematic review. IEEE Trans. Learn. Technol. 12(4), 485–502 (2018)
Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: Proceedings of the 10th European Conference on Machine Learning ECML-98, Chemnitz, Germany, pp. 137–142 (1998)
Pranckevičius, T., Marcinkevičius,V.: Application of Logistic Regression with part-of-the-speech tagging for multi-class text classification. In: IEEE 4th Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE) (2016)
Li, H., Jiang,H., Wang, D., Han, B.: An improved KNN algorithm for text classification. In: 8th International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC), pp. 1081–1085 (2018)
Cer, D., et al.: Universal sentence encoder. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstration, Brussels, Belgium, pp. 169–174 (2018). https://doi.org/10.18653/v1/D18-2029
Chi, S., Qiu, X., Xu, Y., Haung, X.: How to fine-tune BERT for text classification?. In: Chinese Computational Linguistics, pp. 194–206 (2019)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 328–339 (2018). https://doi.org/10.18653/v1/P18-1031
Pasupa, K., Ayutthaya, T.S.N.: Thai sentiment analysis with deep learning techniques: a comparative study based on word embedding, POS-Tag, and sentic features. Sustain. Cities Soc. (2019). https://doi.org/10.1016/j.scs.2019.101615
Manning, C., Surdeanu, M., Bauer,J., Finkel, J.,Bethard, S., McClosky, D.: The stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014). https://doi.org/10.3115/v1/P14-5010
Phatthiyaphaibun, W., Chaovavanich, K., Polpanumas, C., Suriyawongkul, A., Lowphansirikul, L., Chormai, P.: PyThaiNLP: Thai Natural Language Processing in Python. Zenodo (2016). http://doi.org/10.5281/zenodo.3519354
Acknowledgment
The researcher is grateful for funded by the Office of the Permanent Secretary (OPS), Ministry of Higher Education, Science, Research and Innovation, Thailand. We thank King Mongkut’s University of Technology North Bangkok, Thailand for support Information Technology.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chotirat, S., Meesad, P. (2021). Natural Language Processing with “More Than Words – BERT”. In: Meesad, P., Sodsee, D.S., Jitsakul, W., Tangwannawit, S. (eds) Recent Advances in Information and Communication Technology 2021. IC2IT 2021. Lecture Notes in Networks and Systems, vol 251. Springer, Cham. https://doi.org/10.1007/978-3-030-79757-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-79757-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79756-0
Online ISBN: 978-3-030-79757-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)