Recall-Oriented Evaluation Metrics for Consistent Translation of Japanese Legal Sentences

  • Yasuhiro Ogawa
  • Masaki Mori
  • Katsuhiko Toyama
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7258)


We propose new evaluation metrics for statutory sentences translated from Japanese to English. Since translation variety is unacceptable and consistency is crucial in legal translation, a new metric called CIEL has been proposed that evaluates translation consistency. That metric is based on precision with a recall strategy. However, since we believe that recall-oriented metrics are more suitable for consistent evaluation, we propose recall-oriented metrics to evaluate the consistency of legal translations and illustrate their performances by comparing them with other metrics.


ROSSO translation evaluation metric legal translation consistency ROUGE 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgements. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72 (2005)Google Scholar
  2. 2.
    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum (1988)Google Scholar
  3. 3.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press (2001)Google Scholar
  4. 4.
    Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the Second International Conference on Human Language Technology Research, pp. 138–145 (2002)Google Scholar
  5. 5.
    Giménez, J., Amigó, E.: IQMT: A Framework for Automatic Machine Translation Evaluation. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, pp. 685–690 (2006)Google Scholar
  6. 6.
    Hashimoto, S.: Kokugo-hou Kenkyuu. Iwanami Shoten (1948) (in Japanese)Google Scholar
  7. 7.
    Jain, A.K., Murty, M.N., Flynn, P.J.: Data clustering: a review. ACM Computing Surveys 31(3), 264–323 (1999)CrossRefGoogle Scholar
  8. 8.
    Kudo, T., Matsumoto, Y.: Japanese Dependency Analysis using Cascaded Chunking. In: Proceedings of the International Conference on Computational Linguistics, pp. 1–7 (2002)Google Scholar
  9. 9.
    Leusch, G., Ueffing, N., Ney, H.: A novel string-to-string distance measure with applications to machine translation evaluation. In: Proceedings MT Summit IX, pp. 240–247 (2003)Google Scholar
  10. 10.
    Lin, C.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of Text Summarization Branches Out, Workshop at the ACL 2004, pp. 74–81 (2004)Google Scholar
  11. 11.
    Lin, C., Och, F.J.: Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, Main Volume, pp. 605–612 (2004)Google Scholar
  12. 12.
    Lin, C., Och, F.J.: ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In: Proceedings of the 20th International Conference on Computational Linguistics, pp. 501–507 (2004)Google Scholar
  13. 13.
    Nakane, F.: EHS Law Bulletin Series: Japan, vol. VIII. Eibun-Horei-Sha, Inc. (2002)Google Scholar
  14. 14.
    Ogawa, Y., Imai, K., Toyama, K.: Evaluation Metrics for Consistent Translation of Japanese Legal Sentences. In: Francesconi, E., Montemagni, S., Peters, W., Tiscornia, D. (eds.) Semantic Processing of Legal Texts. LNCS, vol. 6036, pp. 235–248. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318 (2002)Google Scholar
  16. 16.
    Study Council for Promoting Translation of Japanese Laws and Regulations into Foreign Languages: Final report (2006),
  17. 17.
    Toyama, K., Ogawa, Y., Imai, K., Matsuura, Y.: Application of word alignment for supporting translation of Japanese statutes into English. In: van Engers, T.M. (ed.) Legal Knowledge and Information Systems JURIX 2006: The Nineteenth Annual Conference, pp. 141–150 (2006)Google Scholar
  18. 18.
    Toyama, K., Saito, D., Sekine, Y., Ogawa, Y., Kakuta, T., Kimura, T., K., Matsuura, Y.: Design and Development of Japanese Translation Database System. Law via the Internet 2011, 12 pages (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Yasuhiro Ogawa
    • 1
  • Masaki Mori
    • 1
  • Katsuhiko Toyama
    • 1
  1. 1.Graduate School of Information ScienceNagoya UniversityNagoyaJapan

Personalised recommendations