Abstract
We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annotated by law experts. The CJRC dataset can help researchers extract elements by reading comprehension technology. Element extraction is an important task in the legal field. However, it is difficult to predefine the element types completely due to the diversity of document types and causes of action. By contrast, machine reading comprehension technology can quickly extract elements by answering various questions from the long document. We build two strong baseline models based on BERT and BiDAF. The experimental results show that there is enough space for improvement compared to human annotators.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Fawei, B., Pan, J.Z., Kollingbaum, M., Wyner, A.Z.: A methodology for a criminal law and procedure ontology for legal question answering. In: Ichise, R., Lecue, F., Kawamura, T., Zhao, D., Muggleton, S., Kozaki, K. (eds.) JIST 2018. LNCS, vol. 11341, pp. 198–214. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04284-4_14
Chen, D., Bolton, J., Manning, C.D.: A thorough examination of the CNN/daily mail reading comprehension task. CoRR abs/1606.02858 (2016). http://arxiv.org/abs/1606.02858
Cui, Y., Liu, T., Chen, Z., Wang, S., Hu, G.: Consensus attention-based neural networks for Chinese reading comprehension, July 2016
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
Dhingra, B., Liu, H., Cohen, W.W., Salakhutdinov, R.: Gated-attention readers for text comprehension. CoRR abs/1606.01549 (2016). http://arxiv.org/abs/1606.01549
He, W., et al.: DuReader: a Chinese machine reading comprehension dataset from real-world applications. CoRR abs/1711.05073 (2017). http://arxiv.org/abs/1711.05073
Hill, F., Bordes, A., Chopra, S., Weston, J.: The goldilocks principle: reading children’s books with explicit memory representations, November 2015
Hu, Z., Li, X., Tu, C., Liu, Z., Sun, M.: Few-shot charge prediction with discriminative legal attributes. In: Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, 20–26 August 2018, pp. 487–498 (2018). https://aclanthology.info/papers/C18-1041/c18-1041
Joshi, M., Choi, E., Weld, D.S., Zettlemoyer, L.: TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. CoRR abs/1705.03551 (2017). http://arxiv.org/abs/1705.03551
Kadlec, R., Schmid, M., Bajgar, O., Kleindienst, J.: Text understanding with the attention sum reader network. CoRR abs/1603.01547 (2016). http://arxiv.org/abs/1603.01547
Kanapala, A., Pal, S., Pamula, R.: Text summarization from legal documents: a survey. Artif. Intell. Rev. 1–32 (2017)
Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.H.: Race: large-scale reading comprehension dataset from examinations. CoRR abs/1704.04683 (2017). http://arxiv.org/abs/1704.04683
Luo, B., Feng, Y., Xu, J., Zhang, X., Zhao, D.: Learning to predict charges for criminal cases with legal basis. In: Proceedings of EMNLP (2017)
Natural Language Computing Group, Microsoft Research Asia: R-net: machine reading comprehension with self-matching networks. In: Proceedings of ACL (2017)
Quaresma, P., Rodrigues, I.P.: A question answer system for legal information retrieval (2005)
Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for squad. CoRR abs/1806.03822 (2018). http://arxiv.org/abs/1806.03822
Reddy, S., Chen, D., Manning, C.D.: CoQA: a conversational question answering challenge. CoRR abs/1808.07042 (2018). http://arxiv.org/abs/1808.07042
Seo, M.J., Kembhavi, A., Farhadi, A., Hajishirzi, H.: Bidirectional attention flow for machine comprehension. CoRR abs/1611.01603 (2016). http://arxiv.org/abs/1611.01603
Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: Weakly supervised memory networks. CoRR abs/1503.08895 (2015). http://arxiv.org/abs/1503.08895
Tran, A.H.N.: Applying deep neural network to retrieve relevant civil law articles. In: Proceedings of the Student Research Workshop Associated with RANLP 2017, pp. 46–48. INCOMA Ltd., Varna, September 2017. https://doi.org/10.26615/issn.1314-9156.2017_007
Trischler, A., et al.: NewsQA: a machine comprehension dataset. CoRR abs/1611.09830 (2016). http://arxiv.org/abs/1611.09830
Xiao, C., et al.: CAIL 2018: a large-scale legal dataset for judgment prediction. CoRR abs/1807.02478 (2018). http://arxiv.org/abs/1807.02478
Yin, X., Zheng, D., Lu, Z., Liu, R.: Neural entity reasoner for global consistency in NER (2018)
Zhang, N., Pu, Y.F., Yang, S.Q., Zhou, J.L., Gao, J.K.: An ontological Chinese legal consultation system. IEEE Access 5, 18250–18261 (2017)
Zhong, H., Guo, Z., Tu, C., Xiao, C., Liu, Z., Sun, M.: Legal judgment prediction via topological learning. In: Proceedings of EMNLP (2018)
Zhong, H., et al.: Overview of CAIL 2018: legal judgment prediction competition. CoRR abs/1810.05851 (2018). http://arxiv.org/abs/1810.05851
Acknowledgements
This work is supported by the National Key R&D Program of China under Grant No. 2018YFC0832103.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Duan, X. et al. (2019). CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds) Chinese Computational Linguistics. CCL 2019. Lecture Notes in Computer Science(), vol 11856. Springer, Cham. https://doi.org/10.1007/978-3-030-32381-3_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-32381-3_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32380-6
Online ISBN: 978-3-030-32381-3
eBook Packages: Computer ScienceComputer Science (R0)