Machine Translation

, Volume 19, Issue 1, pp 1–36 | Cite as

Controlled Translation in an Example-based Environment: What do Automatic Evaluation Metrics Tell Us?

  • Andy Way
  • Nano Gough


This paper presents an extended, harmonised account of our previous work on integrating controlled language data in an Example-based Machine Translation system. Gough and Way in MT Summit pp. 133–140 (2003) focused on controlling the output text in a novel manner, while Gough and Way (9th Workshop of the EAMT, (2004a), pp. 73–81) sought to constrain the input strings according to controlled language specifications. Our original sub-sentential alignment algorithm could deal only with 1:1 matches, but subsequent refinements enabled n:m alignments to be captured. A direct consequence was that we were able to populate the system’s databases with more than six times as many potentially useful fragments. Together with two simple novel improvements – correcting a small number of mistranslations in the lexicon, and allowing multiple translations in the lexicon – translation quality improves considerably. We provide detailed automatic and human evaluations of a number of experiments carried out to test the quality of the system. We observe that our system outperforms the rule-based on-line system Logomedia on a range of automatic evaluation metrics, and that the ‘best’ translation candidate is consistently highly ranked by our system. Finally, we note in a number of tests that the BLEU metric gives objectively different results than other automatic evaluation metrics and a manual evaluation. Despite these conflicting results, we observe a preference for controlling the source data rather than the target translations.


controlled translation example-based MT Marker Hypothesis evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Akiba, Y., K. Imamura and E. Sumita: 2001, ‘Using multiple edit distances to automatically rank Machine Translation output’. In MT Summit (2001), pp. 15–20.Google Scholar
  2. Almqvist, I. and A. Sågvall Hein: 1996, ‘Defining ScaniaSwedish – Controlled language for truck maintenance’. In CLAW (1996), pp. 159–164.Google Scholar
  3. Adriens, G. and D. Schreurs: 1992, ‘From Cogram to Alcogram:Toward a controlled English grammar checker’. In COLING (1992), pp. 595–601.Google Scholar
  4. Barthe, K.: 1998, ‘GIFAS Rationalised French: Designing one controlled language to match another’. In CLAW (1998), pp. 87–102.Google Scholar
  5. Bennett, S., Slocum, J. 1985The LRC translation system’Computational Linguistics11111121Google Scholar
  6. Bernth, A.: 2003, ‘Controlled generation for speech-to-speech MT systems’. In CLAW (2003), pp. 1–7.Google Scholar
  7. Block, H.-U. 2000‘Example-based incremental synchronous interpretation’Wahlster, W. eds. Verbmobil: Foundations of Speech-to-Speech TranslationSpringer VerlagBerlin411417Google Scholar
  8. Brown, R.: 1999, ‘Adding linguistic knowledge to a lexical example-based translation system’. In TMI (1999), pp. 22–32.Google Scholar
  9. Brown, R.: 2000, ‘Automated generalization of translation examples’. In COLING 2000 in Europe: Proceedings of the 18th International Conference on Computational Linguistics, Saarbrücken, Germany, pp. 125–131.Google Scholar
  10. Brown, R.: 2003, ‘Clustered transfer rule induction for example-based translation’. In Carl and Way (2003), pp. 287–305.Google Scholar
  11. Carl, M.: 2003, ‘Data-assisted Controlled Translation’. In CLAW (2003), pp. 16–24.Google Scholar
  12. Carl, M.Way, A. eds. 2003Recent Advances in Example-based Machine TranslationKluwer Academic PublishersDordrechtGoogle Scholar
  13. Chandioux, J.: 1976, ‘MÉTÉO: un système opérationel pour la traduction automatique des bulletins météorologiques destinés au grand public [MÉTÉO: an operational system for the machine translation of weather bulletins aimed at the general public]. META 21, 127–133.Google Scholar
  14. Cicekli, I. and H.A. Güvenir: 2003, ‘Learning translation templates from bilingual translation examples’. In Carl and Way (2003), pp. 255–86.Google Scholar
  15. CLAW 2003, EAMT-CLAW 03, Joint Conference combining the 8th International Workshop of the European Association for Machine Translation and the 4th Controlled Language Applications Workshop, Controlled Translation, Dublin, Ireland.Google Scholar
  16. COLING: 1992, Proceedings of the fifteenth [sic] International Conference on Computational Linguistics, COLING-92, Nantes, France.Google Scholar
  17. Coughlin, D.: 2003, ‘Correlating automated and human assessments of Machine Translation quality’. In MT Summit (2003), pp. 63–70.Google Scholar
  18. Doddington, G.: 2002, ‘Automatic evaluation of Machine Translation quality using n-gram co-occurrence statistics’. In Proceedings of the Second International Conference on Human Language Technology Research, San Diego, CA, pp. 138–145.Google Scholar
  19. EAMT-CLAW: 1996, Proceedings of the First International Workshop on Controlled Language Applications, Leuven, Belgium.Google Scholar
  20. Furuse, O., and H. Iida: 1992a, ‘An example-based method for transfer-driven machine translation’. In Fourth International Conference on Theoretical and Methodological Issues in Machine Translation: Empiricist vs. Rationalist Methods in MT (TMI-92), Montréal, Canada, pp. 139–150.Google Scholar
  21. Furuse, O., and H. Iida: 1992b, ‘Cooperation between transfer and analysis in example-based framework’. In COLING (1992), pp. 645–651.Google Scholar
  22. Gough, N. 2005Example-based Machine Translation Using the Marker HypothesisDublin City UniversityDublin, IrelandPhD ThesisGoogle Scholar
  23. Gough, N., and A. Way: 2003, ‘Controlled generation in Example-Based Machine Translation’. In MT Summit (2003), pp. 133–140.Google Scholar
  24. Gough, N., and A. Way: 2004a, ‘Example-based Controlled Translation’. In Proceedings of the 9th Workshop of the European Association for Machine Translation (EAMT), Valetta, Malta, pp. 73–81.Google Scholar
  25. Gough, N., and A. Way: 2004b, ‘Robust large-scale EBMT with Marker-based segmentation’. In TMI (2004), pp. 95–104.Google Scholar
  26. Green, T.R.G. 1979‘The necessity of syntax markers Two experiments with artificial languages’Journal of Verbal Learning and Behavior18481496Google Scholar
  27. Güvenir, H.A., Cicekli, I. 1998‘Learning translation templates from examples’Information Systems23353363CrossRefGoogle Scholar
  28. Güvenir, H.A., Tunç, A. 1996‘Corpus-based Learning of Generalized Parse Tree Rules for Translation’McCalla, G. eds. Advances in Artificial IntelligenceSpringer VerlagBerlin121132Google Scholar
  29. Hartley, A., D. Scott, J. Bateman, and D. Dochev: 2001, ‘AGILE - A system for multilingual generation of technical instructions’. In MT Summit (2001), pp. 145–150.Google Scholar
  30. Hearne, M. and A. Way: 2003, ‘Seeing the wood for the trees: Data-oriented translation’. In MT Summit (2003), pp. 165–172.Google Scholar
  31. Juola, P.: 1994, ‘A psycholinguistic approach to corpus-based Machine Translation’. In CSNLP 1994; 3rd International Conference on the Cognitive Science of Natural Language Processing, Dublin, Ireland, [pages not numbered].Google Scholar
  32. Kaji, H., Y. Kida, and Y. Morimoto: 1992, ‘Learning translation templates from bilingual bext’. In COLING (1992), pp. 672–678.Google Scholar
  33. Kamprath, C., E. Adolphson, T. Mitamura, and E. Nyberg: 1998, ‘Controlled Language for multilingual document production: Experience with Caterpillar Technical English’. In CLAW (1998), pp. 51–61.Google Scholar
  34. Kulesza, A., and S. Shieber: 2004, ‘A learning approach to improving sentence-level MT evaluation’. In TMI (2004), pp. 75–84.Google Scholar
  35. Lehrndorfer, A., and S. Schachtl: 1998, ‘Controlled Siemens Documentary German and TopTrans’. TC-Forum 3-98:T09. Available on-line at, last accessed 21 July, 2005.Google Scholar
  36. Lin, C-Y., and F.J. Och: 2004, ‘ORANGE: A method for evaluating automatic evaluation metrics for Machine Translation’. In Coling, 20th International Conference on Computational Linguistics, Geneva, Switzerland, pp. 501–507.Google Scholar
  37. Lux, V., and É. Dauphin: 1996, ‘Corpus Studies: A contribution to the definition of a controlled language’. In CLAW (1996), pp. 193–204.Google Scholar
  38. Matsumoto, Y., Kitamura, M. 1995‘Acquisition of translation rules from parallel corpora’Mitkov, R.Nicolov, N. eds. Recent Advances in Natural Language Processing: Selected Papers from the ConferenceJohn BenjaminsAmsterdam405416Google Scholar
  39. McTait, K.: 2003, ‘Translation patterns, linguistic knowledge and complexity in EBMT’. In Carl and Way (2003), pp. 307–338.Google Scholar
  40. McTait, K., and A. Trujillo: 1999, ‘A language-neutral sparse-data algorithm for extracting translation patterns’. In TMI (1999), pp. 98–108.Google Scholar
  41. Means, L., and K. Godden: 1996, ‘The Controlled Automotive Service Language (CASL) project’. In CLAW (1996), pp. 106–114.Google Scholar
  42. Mitamura T., Nyberg E. 1995. ‘Controlled English for Knowledge Based MT: Experience with the KANT System’. In Proceedings of the Sixth International Conference on Theoretical and Methodological Issues in Machine Translation, Leuven, Belgium, pp. 158–172.Google Scholar
  43. MT Summit: 2001, MT Summit VIII: Machine Translation in the Information Age, Proceedings, Santiago de Compostela, Spain.Google Scholar
  44. MT Summit: 2003, MT Summit IX: Proceedings of the Ninth Machine Translation Summit, New Orleans, LA.Google Scholar
  45. Nyberg, E., and T. Mitamura: 1992, ‘The KANT system: fast, accurate, high-quality translation in practical domains’. In COLING (1992), pp. 1254–1258.Google Scholar
  46. Papineni, K., S. Roukos, T. Ward, and W-J. Zhu: 2002, ‘BLEU: A method for automatic evaluation of Machine Translation’. In 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia PA, pp. 31l–318.Google Scholar
  47. Power, R., D. Scott, and A. Hartley: 2003, ‘Multilingual generation of controlled languages’. In CLAW (2003), pp. 115–123.Google Scholar
  48. Sågvall Hein, A. 1996‘Preference mechanisms of the Multra Machine Translation system’Partee, B.Sgall, P. eds. Discourse and Meaning. Papers in Honor of Eva HajičováJohn BenjaminsAmsterdam321333Google Scholar
  49. Schachtl, S.: 1996, ‘Requirements for Controlled German inindustrial applications’. In CLAW (1996), pp. 143–149.Google Scholar
  50. Schäler, R., A. Way, and M. Carl: 2003, ‘Example-based Machine Translation in a controlled environment’. In Carl and Way (2003), pp. 83–114.Google Scholar
  51. TMI: 1999, Proceedings of the 8th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI-99), Chester, England.Google Scholar
  52. TMI: 2004, Proceedings of the Tenth Conference on Theoretical and Methodological Issues in Machine Translation (TMI-04), Baltimore, MD.Google Scholar
  53. Turian, J., L. Shen, and D. Melamed: 2003, ‘Evaluation of Machine Translation and its evaluation’. In MT Summit (2003), pp. 386–393.Google Scholar
  54. Van der Eijk, P., M. de Koning, and G. van der Steen: 1996, ‘Controlled language correction and translation’. In CLAW (1996), pp. 64–73.Google Scholar
  55. Veale, T., and A. Way: 1997, `Gaijin: A bootstrapping, template-driven approach to Example-based Machine Translation’. In International Conference, Recent Advances in Natural Language Processing, Tzigov Chark, Bulgaria, pp. 239–244.Google Scholar
  56. Watanabe, H., S. Kurohashi, and E. Aramaki: 2003, ‘Finding translation patterns from paired source and target dependency structures’. In Carl and Way (2003), pp. 397–420.Google Scholar
  57. Way, A., Gough, N. 2003`wEBMT: Developing and validating an Example-based Machine Translation system using the World Wide Web’Computational Linguistics29421457CrossRefGoogle Scholar
  58. Wells-Akis, J., and W. Sisson: 2002, ‘Improving translatability - A case study at Sun Microsystems Inc’. The LISA Newsletter: Globalization Insider 4.5.Google Scholar
  59. Yamada, S., E. Sumita and H. Kashioka: 2000, ‘Translation using information on dialogue participants’. In Proceedings of the 6th Applied Natural Language Conference and 1st Meeting of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, pp. 37–43.Google Scholar
  60. Zhang, Y., and S. Vogel: 2004, ‘Measuring confidence intervals for the Machine Translation evaluation metrics’. In TMI (2004), pp. 85–94.Google Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  1. 1.School of ComputingDublin City UniversityDublin 9Ireland

Personalised recommendations