Human Post-editing in Hybrid Machine Translation Systems: Automatic and Manual Analysis and Evaluation

  • Juncal Gutiérrez-Artacho
  • María-Dolores Olvera-Lobo
  • Irene Rivera-Trigueros
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 745)

Abstract

This study assesses, automatically and manually, the performance of two hybrid machine translation (HMT) systems, via a text corpus of questions in the Spanish and English languages. The results show that human evaluation metrics are more reliable when evaluating HMT performance. Further, there is evidence that MT can streamline the translation process for specific types of texts, such as questions; however, it does not yet rival the quality of human translations, to which post-editing is key in this process.

Keywords

Hybrid machine translation Automatic evaluation Human evaluation Post-editing 

Notes

Acknowledgements

This work was supported by the University of Granada Special Research Programme - Starting Research Grants for Master Students.

References

  1. 1.
    Gutiérrez Artacho, J.: Recursos y herramientas lingüísticos para los sistemas de búsqueda de respuestas monolingües y multilingües (2015)Google Scholar
  2. 2.
    Zhou, D., Truran, M., Brailsford, T., Wade, V., Ashman, H.: Translation techniques in cross-language information retrieval. ACM Comput. Surv. 45(1), 1–44 (2012)CrossRefGoogle Scholar
  3. 3.
    Olvera-Lobo, M., Gutierrez-Artacho, J.: Language resources used in multi-lingual question-answering systems. Online Inf. Rev. 35, 543–557 (2011)CrossRefGoogle Scholar
  4. 4.
    Olvera-Lobo, M.D., Garcia-Santiago, L.: Analysis of errors in the automatic translation of questions for translingual QA systems. J. Doc. 66, 434–455 (2010)CrossRefGoogle Scholar
  5. 5.
    García-Santiago, L., Olvera-Lobo, M.-D.: Analysis of automatic translation of questions for question-answering systems. Inf. Res. 15 (2010). http://www.informationr.net/ir/15-4/paper450.html
  6. 6.
    Madankar, M., Chandak, M.B., Chavhan, N.: Information retrieval system and machine translation: a review. Procedia Comput. Sci. 78, 845–850 (2016)CrossRefGoogle Scholar
  7. 7.
    Gupta, M., Bendersky, M.: Information retrieval with verbose queries. Found. Trends Inf. Retrieval 9(3–4), 200–354 (2015)Google Scholar
  8. 8.
    Olvera-Lobo, M.D., Castro-Prieto, M.R., Quero-Gervilla, E., Munoz-Martin, R., Munoz-Raya, E., Murillo-Melero, M., Robinson, B., Senso-Ruiz, A., Vargas-Quesada, B., Dominguez-Lopez, C.: Translator training and modern market demands. Perspect. Transl. 13, 132–142 (2005)Google Scholar
  9. 9.
    Koponen, M.: Is machine translation post-editing worth the effort? a survey of research into post-editing and effort. J. Spec. Transl 23, 131–148 (2016)Google Scholar
  10. 10.
    Mesa-lao, B.: Introduction to post-editing – The CasMaCat GUI 1 (2013)Google Scholar
  11. 11.
    Allen, J.: Post-editing. In: Somers, H.L. (ed.) Computers and Translation: A Translators Guide. John Benjamins, Amsterdam/Philadelphia (2003). 297–317Google Scholar
  12. 12.
    Lagarda, A.L., Ortiz-Martinez, D., Alabau, V., Casacuberta, F.: Translating without in-domain corpus: machine translation post-editing with online learning techniques. Comput. Speech Lang. 32, 109–134 (2015)CrossRefGoogle Scholar
  13. 13.
    Shaw, F., Gros, X.: Survey of Machine Translation Evaluation, Saarbrücken (2007)Google Scholar
  14. 14.
    Mauser, A., Hasan, S., Ney, H.: Automatic evaluation measures for statistical machine translation system optimization. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC 2008), Marraketch, Morocco (2008)Google Scholar
  15. 15.
    Hutchins, W.J., Somers, H.L.: An Introduction to Machine Translation. Academic Press, London (1992). 57, 377MATHGoogle Scholar
  16. 16.
    Arnold, D., Balkan, L., Meijer, S., Humphreys, R.L., Sadler, L.: An Introductory Guide. NCC Blackwell, London (1994)Google Scholar
  17. 17.
    Costa-Jussa, M.R., Fonollosa, J.A.R.: Latest trends in hybrid machine translation and its applications. Comput. Speech Lang. 32, 3–10 (2015)CrossRefGoogle Scholar
  18. 18.
    Labaka, G., España-Bonet, C., Màrquez, L., Sarasola, K.: A hybrid machine translation architecture guided by syntax. Mach. Transl. 28, 91–125 (2014)CrossRefGoogle Scholar
  19. 19.
    Hunsicker, S., Yu, C., Federmann, C.: Machine learning for hybrid machine translation. In: Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, pp. 312–316 (2012)Google Scholar
  20. 20.
    Tambouratzis, G.: Conditional random fields versus template-matching in MT phrasing tasks involving sparse training data. PATTERN Recognit. Lett. 53, 44–52 (2015)CrossRefGoogle Scholar
  21. 21.
    Kamran, A.: Hybrid Machine Translation Panel (2013)Google Scholar
  22. 22.
    Leusch, G.: Evaluation Measures in Machine Translation (2005)Google Scholar
  23. 23.
    Niessen, S., Och, F.J., Leusch, G., Ney, H.: An evaluation tool for machine translation: Fast evaluation for MT research. ACM Trans. Inf. Syst. 20, 39–45 (2000)Google Scholar
  24. 24.
    Mayor, A., Alegria, I., Díaz Ilarraza, A., Labaka, G., Lersundi, M., Sarasola, K.: Evaluación de un sistema de traducción automática basado en reglas o por qué BLEU sólo sirve para lo que sirve. Proces. leng. nat. 43, 197–205 (2009)Google Scholar
  25. 25.
    Homola, P., Kubon, V., Pecina, P., Pecina, P.: A simple automatic MT evaluation metric. In: Association for Computational Linguistics (ed.) Proceedings of the 4th EACL Workshop on Statistical Machine Translation, Athens, pp. 33–36 (2009)Google Scholar
  26. 26.
    Turian, J.P., Shen, L., Melamed, I.D.: Evaluation of machine translation and its evaluation. In: Proceedings of MT Summit IX, New Orleans (2003)Google Scholar
  27. 27.
    Boitet, C., Bey, Y., Tomokiyo, M., Cao, W., Blanchon, H.: IWSLT-06 : experiments with commercial MT systems and lessons from subjective evaluations (2006)Google Scholar
  28. 28.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics, Stroudsburg (2002)Google Scholar
  29. 29.
    Bouillon, P. (Coord.): Analysis of existing metrics and proposal for a task-oriented metric. In: ACCEPT- Automated Community Content Editing PorTal (2012)Google Scholar
  30. 30.
    Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, pp. 65–72 (2005)Google Scholar
  31. 31.
    Goutte, C.: Learning Machine Translation. MIT Press, Cambridge (2009)Google Scholar
  32. 32.
    Ferrández, O., Micol, D., Muñoz, R., Palomar, M.: Técnicas léxico- sintácticas para el reconocimiento de Implicación Textual. Proces. del Leng. Nat. 38, 53–60 (2007)Google Scholar
  33. 33.
    Popovic, M., Ney, H.: Word error rates: decomposition over POS classes and applications for error analysis. In: Proceedings of the Second Workshop on Statistical Machine Translation, pp. 48–55. Association for Computational Linguistics, Prague (2007)Google Scholar
  34. 34.
    Helle, A.: Hibridación en lenguas distantes (2013)Google Scholar
  35. 35.
    Olvera‐Lobo, M.D., Quero‐Gervilla, E., Robinson, B., Senso‐Ruiz, J.A., Castro-Prieto, M.R., Muñoz‐Martín, R., Muñoz‐Raya, E., Murillo‐Melero, M: Presentation of a distance training model for introduction into the practice of teaching translation according to the requirements of the Bologna declaration. Vestn. MGU. Ser. (26) (2010). Pedagog. Sci. 605, 196–208Google Scholar
  36. 36.
    Olvera-Lobo, M.-D., Gutiérrez-Artacho, J.: Evaluación de los sistemas QA de dominio abierto frente a los de dominio especializado en el ámbito biomédico. In: I Congreso Español de Recuperación de Información (CERI 2010), Madrid, pp. 161–169 (2010)Google Scholar
  37. 37.
    Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Open - vs. restricted-domain QA systems in the biomedical field. J. Inf. Sci. 37, 152–162 (2011)CrossRefGoogle Scholar
  38. 38.
    Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Multilingual question-answering system in biomedical domain on the web: an evaluation. In: Forner, P., Gonzalo, J., Kekalainen, J., Lalmas, M., DeRijke, M. (eds.) Multilingual and Multimodal Information Access Evaluation, pp. 83–88 (2011)CrossRefGoogle Scholar
  39. 39.
    Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Performance analysis in web-based question answering systems. Rev. Esp. Doc. Cient. 36, e009 (2013)CrossRefGoogle Scholar
  40. 40.
    Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Searching health information in question-answering systems. In: Cruz-Cunha, M.M., Miranda, I.M., Gonçalves, P. (eds.) Handbook of Research on ICTs for Human-Centered Healthcare and Social Care Services, pp. 474–490. IGI Global, Hershey (2013)Google Scholar
  41. 41.
    Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Nuevas tendencias en recuperación de información: la búsqueda de respuestas desde la perspectiva de la traducción. In: VI Congreso Internacional de la Asociación Ibérica de Estudios de Traducción e Interpretación: ‘Traducimos desde el sur’. Asociación Ibérica de Estudios de Traducción e Interpretación (AIETI), Las Palmas de Gran Canaria (2013)Google Scholar
  42. 42.
    Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Satisfacción de los usuarios en la búsqueda multilingüe de respuestas como recursos de información terminológica. In: Vargas-Sierra, C. (ed.) TIC, trabajo colaborativo e interacción en terminología y traducción, Comares, Granada, pp. 191–200 (2014)Google Scholar
  43. 43.
    Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Overview of translation techniques in cross-language question answering during the last decade. In: Khosrow-Pour, M. (ed.) Encyclopedia of Information Science and Technology, pp. 4747–4755. IGI Global, Hershey (2015)Google Scholar
  44. 44.
    Olvera-Lobo, M.-D., Gutierrez-Artacho, J.: Question answering track evaluation in TREC, CLEF and NTCIR. In: Rocha, A., Correia, A.M., Costanzo, S., Reis, L.P. (ed.) New Contributions in Information Systems and Technologies, vol. 1, pp. 13–22 (2015)CrossRefGoogle Scholar
  45. 45.
    Koehn, P.: Europarl: a parallel corpus for statistical machine translation. MT Summit. 11, 79–86 (2005)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Translation and Interpreting, Faculty of Translation and InterpretingUniversity of GranadaGranadaSpain
  2. 2.Department of Information and Communication, Colegio Máximo de CartujaUniversity of GranadaGranadaSpain
  3. 3.CSIC, Unidad Asociada Grupo SCImagoMadridSpain
  4. 4.University of GranadaGranadaSpain

Personalised recommendations