Skip to main content
SpringerLink
Log in

A BERT-Based Model for Legal Document Proofreading

  • Conference paper
  • First Online:
Intelligent Information Processing XII (IIP 2024)

Part of the book series: IFIP Advances in Information and Communication Technology ((IFIPAICT,volume 703))

Included in the following conference series:

Abstract

Legal documents require high precision and accuracy in language use, leaving no room for grammatical and spelling errors. To address the issue, this paper proposes a novel application of the BERT pre-trained language model for legal document proofreading. The BERT-based model is trained to detect and correct legal texts’ grammatical and spelling errors. On a dataset of annotated legal documents, we experimentally show that our BERT-based model significantly outperforms state-of-the-art proofreading models in precision, recall, and F1 score, showing its potential as a valuable tool in legal document preparation and revision processes. The application of such advanced deep learning techniques could revolutionise the field of legal document proofreading, enhancing accuracy and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://app.grammarly.com/.

  2. 2.

    https://languagetool.org.

  3. 3.

    https://prowritingaid.com/.

  4. 4.

    https://www.whitesmoke.com.

  5. 5.

    https://github.com/china-ai-law-challenge/CAIL 2022/tree/main/wsjd.

  6. 6.

    https://github.com/china-ai-law-challenge/CAIL 2022/tree/main/wsjd.

  7. 7.

    http://cail.cipsc.org.cn/task_summit.html?raceID=2 &cail_tag=2022.

  8. 8.

    https://github.com/china-ai-law-challenge/CAIL2022/tree/main/wsjd/baseline.

  9. 9.

    https://github.com/HillZhang1999/MuCGEC/tree/main.

  10. 10.

    https://github.com/shibing624/pycorrector/tree/master/examples/t5.

  11. 11.

    https://huggingface.co/bert-base-chinese.

  12. 12.

    Since the competition phase for testing is now closed, we cannot make a direct comparison.

  13. 13.

    https://chat.openai.com/.

  14. 14.

    https://www.anthropic.com/.

  15. 15.

    https://xinghuo.xfyun.cn/desk.

  16. 16.

    https://neice.tiangong.cn.

References

  1. Bai, Y., et al.: Constitutional AI: harmlessness from AI feedback (2022). arXiv preprint arXiv:2212.08073

  2. Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning. Springer (2006)

    Google Scholar 

  3. Bryant, C., Yuan, Z., Qorib, M.R., Cao, H., Ng, H.T., Briscoe, T.: Grammatical error correction: a survey of the state of the art. Comput. Linguist. 1–59 (2022)

    Google Scholar 

  4. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)

    Google Scholar 

  5. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)

    Article  Google Scholar 

  6. Fang, T., et al.: Is ChatGPT a highly fluent grammatical error correction system? A comprehensive evaluation (2023). arXiv preprint arXiv:2304.01746

  7. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    Google Scholar 

  8. Gu, J., Wang, C., Zhao, J.: Levenshtein transformer. In: Advances in Neural Information Processing Systems. vol. 32 (2019)

    Google Scholar 

  9. Hong, Y., Yu, X., He, N., Liu, N., Liu, J.: FASPell: a fast, adaptable, simple, powerful Chinese spell checker based on DAE-decoder paradigm. In: Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pp. 160–169 (2019)

    Google Scholar 

  10. Katsumata, S., Komachi, M.: Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 827–832 (2020)

    Google Scholar 

  11. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, pp. 1–15 (2014)

    Google Scholar 

  12. Korre, K., Pavlopoulos, J.: Enriching grammatical error correction resources for modern Greek. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4984–4991 (2022)

    Google Scholar 

  13. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  14. Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. Sov. Phys.-Dokl. 10(8), 707–710 (1966)

    MathSciNet  Google Scholar 

  15. Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880 (2020)

    Google Scholar 

  16. Malmi, E., Krause, S., Rothe, S., Mirylenka, D., Severyn, A.: Encode, tag, realize: high-precision text editing. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5054–5065 (2019)

    Google Scholar 

  17. Musyafa, A., Gao, Y., Solyman, A., Wu, C., Khan, S.: Automatic correction of Indonesian grammatical errors based on transformer. Appl. Sci. 12(20), 10380 (2022)

    Article  Google Scholar 

  18. Omelianchuk, K., Atrasevych, V., Chernodub, A., Skurzhanskyi, O.: GECToR – grammatical error correction: Tag, not rewrite. In: Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 163–170 (2020)

    Google Scholar 

  19. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

    MathSciNet  Google Scholar 

  20. Rothe, S., Mallinson, J., Malmi, E., Krause, S., Severyn, A.: A simple recipe for multilingual grammatical error correction. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 702–707 (2021)

    Google Scholar 

  21. Solyman, A., Wang, Z., Tao, Q., Elhag, A.A.M., Zhang, R., Mahmoud, Z.: Automatic Arabic grammatical error correction based on expectation-maximization routing and target-bidirectional agreement. Knowl.-Based Syst. 241, 108180 (2022)

    Article  Google Scholar 

  22. Stahlberg, F., Kumar, S.: Synthetic data generation for grammatical error correction with tagged corruption models (2021). arXiv preprint arXiv:2105.13318

  23. Sun, K., Luo, X., Luo, M.Y.: A survey of pretrained language models. In: Memmi, G., Yang, B., Kong, L., Zhang, T., Qiu, M. (eds.) Knowledge Science, Engineering and Management. KSEM 2022. LNCS(), vol. 13369. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-10986-7_36

  24. Tarnavskyi, M., Chernodub, A., Omelianchuk, K.: Ensembling and knowledge distilling of large sequence taggers for grammatical error correction (2022). arXiv preprint arXiv:2203.13064

  25. Trinh, V.A., Rozovskaya, A.: New dataset and strong baselines for the grammatical error correction of Russian. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4103–4111 (2021)

    Google Scholar 

  26. Vaswani, A., et al.: Attention is all you need. Adv. Neural. Inf. Process. Syst. 30, 1–10 (2017)

    Google Scholar 

  27. Wang, C., Yang, L., Wang, Y., Du, Y., Yang, E.: Chinese grammatical error correction method based on transformer enhanced architecture. J. Chin. Inf. Process. 34(6), 106–114 (2020)

    Google Scholar 

  28. Xu, H.D., et al.: Read, listen, and see: Leveraging multimodal information helps Chinese spell checking. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 716–728 (2021)

    Google Scholar 

  29. Yuan, Z., Briscoe, T.: Grammatical error correction using neural machine translation. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 380–386 (2016)

    Google Scholar 

  30. Yuan, Z., Bryant, C.: Document-level grammatical error correction. In: Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 75–84 (2021)

    Google Scholar 

  31. Zhang, S., Huang, H., Liu, J., Li, H.: Spelling error correction with soft-masked BERT. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 882–890 (2020)

    Google Scholar 

  32. Zhang, Y., et al.: MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error correction. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3118–3130 (2022)

    Google Scholar 

  33. Zhang, Y., Zhang, B., Li, Z., Bao, Z., Li, C., Zhang, M.: SynGEC: Syntax-enhanced grammatical error correction with a tailored GEC-oriented parser (2022). arXiv preprint arXiv:2210.12484

  34. Zhao, Y., Jiang, N., Sun, W., Wan, X.: Overview of the NLPCC 2018 shared task: grammatical error correction. In: Zhang, M., Ng, V., Zhao, D., Li, S., Zan, H. (eds.) NLPCC 2018. LNCS (LNAI), vol. 11109, pp. 439–445. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99501-4_41

    Chapter  Google Scholar 

  35. Zhao, Z., Wang, H.: MaskGEC: improving neural grammatical error correction via dynamic masking. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 1226–1233 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Guangxi Normal University, Guilin, 541004, China

    Jinlong Liu & Xudong Luo

  2. Key Lab of Education Blockchain and Intelligent Technology of Ministry of Education, Guangxi Normal University, Guilin, 541004, China

    Xudong Luo

  3. Guangxi Key Lab of Multi-Source Information Mining & Security, Guangxi Normal University, Guilin, 541004, China

    Xudong Luo

Authors
  1. Jinlong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xudong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xudong Luo .

Editor information

Editors and Affiliations

  1. Chinese Academy of Sciences, Beijing, China

    Zhongzhi Shi

  2. University of Oslo, Oslo, Norway

    Jim Torresen

  3. De Montfort University, Leicester, UK

    Shengxiang Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, J., Luo, X. (2024). A BERT-Based Model for Legal Document Proofreading. In: Shi, Z., Torresen, J., Yang, S. (eds) Intelligent Information Processing XII. IIP 2024. IFIP Advances in Information and Communication Technology, vol 703. Springer, Cham. https://doi.org/10.1007/978-3-031-57808-3_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-57808-3_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-57807-6

  • Online ISBN: 978-3-031-57808-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation