Advertisement

Chinese Wrong Words Detection and Correction Algorithm Based on Sentence Perplexity and Iterative Correction

  • Wanli SongEmail author
  • Chunxu Zhu
Conference paper
  • 26 Downloads
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1147)

Abstract

With the rapid development of the field of artificial intelligence, the traditional way to check whether there is a typo in the text is very inefficient. People have been looking for a way to automatically detect and correct wrong words used in texts. The use of artificial intelligence technology for detection and correction of typography came into being. The defects and causes of the existing Chinese wrong words detection and correction technology is analyzed in this paper. The algorithm is designed to detect and correct the wrong words twice and judge the interval between the first result and the second result. The sentence perplexity is calculated by combining the deep neural network language model, forming a double filter to reduce the misjudgment in the process of wrong words detection. Experiments in various text styles indicate that the improved algorithm can reduce more than 90% of misjudgments in the process of detection and correction.

Keywords

Artificial intelligence Chinese word segmentation Wrong words detection and correction Language model Sentence perplexity 

Notes

Acknowledgments

This work was partially supported by the following research grants: (1) No. 2016NXY46 from the Research Foundation of Nanjing Xiaozhuang University and (2) No. 19AIP07 from the Research Foundation of the Key Laboratory of Intelligent Information Processing of Nanjing Xiaozhuang University.

References

  1. 1.
    Julie. Automatic detection of Chinese typography. Sichuan Foreign Studies University (2016). (in Chinese)Google Scholar
  2. 2.
    Xu, M.: Research and implementation of automatic detection and correction of typography in online business user reviews. Beijing University of Technology (2016). (in Chinese)Google Scholar
  3. 3.
  4. 4.
    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)zbMATHGoogle Scholar
  5. 5.
  6. 6.
    Li, F., Ke, J.: Research progress of word vector semantic representation. Inf. Sci. 37(05), 155–165 (2019). (in Chinese)Google Scholar
  7. 7.
    Li, H., Zhang, H.: Book review emotion analysis method based on word vector and CNN. J. Test. Technol. (2019). (in Chinese)Google Scholar
  8. 8.
    Mikolov, T.: Statistical Language Models Based on Neural Networks. Brno University of Technology (2012)Google Scholar
  9. 9.
    Mikolov, T., Yih, W.T., Zweig, G.: Linguistic regularities continuous space word representations. Assoc. Comput. Linguist. (2013)Google Scholar
  10. 10.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Information Engineering SchoolNanjing Xiaozhuang UniversityNanjingChina
  2. 2.School of Computer and Information EngineeringHohai UniversityNanjingChina
  3. 3.Key Laboratory of Intelligent Information ProcessingNanjing Xiaozhuang UniversityNanjingChina

Personalised recommendations