Chinese Wrong Words Detection and Correction Algorithm Based on Sentence Perplexity and Iterative Correction
- 26 Downloads
With the rapid development of the field of artificial intelligence, the traditional way to check whether there is a typo in the text is very inefficient. People have been looking for a way to automatically detect and correct wrong words used in texts. The use of artificial intelligence technology for detection and correction of typography came into being. The defects and causes of the existing Chinese wrong words detection and correction technology is analyzed in this paper. The algorithm is designed to detect and correct the wrong words twice and judge the interval between the first result and the second result. The sentence perplexity is calculated by combining the deep neural network language model, forming a double filter to reduce the misjudgment in the process of wrong words detection. Experiments in various text styles indicate that the improved algorithm can reduce more than 90% of misjudgments in the process of detection and correction.
KeywordsArtificial intelligence Chinese word segmentation Wrong words detection and correction Language model Sentence perplexity
This work was partially supported by the following research grants: (1) No. 2016NXY46 from the Research Foundation of Nanjing Xiaozhuang University and (2) No. 19AIP07 from the Research Foundation of the Key Laboratory of Intelligent Information Processing of Nanjing Xiaozhuang University.
- 1.Julie. Automatic detection of Chinese typography. Sichuan Foreign Studies University (2016). (in Chinese)Google Scholar
- 2.Xu, M.: Research and implementation of automatic detection and correction of typography in online business user reviews. Beijing University of Technology (2016). (in Chinese)Google Scholar
- 6.Li, F., Ke, J.: Research progress of word vector semantic representation. Inf. Sci. 37(05), 155–165 (2019). (in Chinese)Google Scholar
- 7.Li, H., Zhang, H.: Book review emotion analysis method based on word vector and CNN. J. Test. Technol. (2019). (in Chinese)Google Scholar
- 8.Mikolov, T.: Statistical Language Models Based on Neural Networks. Brno University of Technology (2012)Google Scholar
- 9.Mikolov, T., Yih, W.T., Zweig, G.: Linguistic regularities continuous space word representations. Assoc. Comput. Linguist. (2013)Google Scholar