Abstract
The combination of temporal, technical and cognitive effort has been proposed as metrics to evaluate the feasibility of post-editing on machine-translation (MT) output (Krings, 2001). In this study, we investigate the impact of interactive machine translation on the post-editing effort required to post-edit two specialized texts under experimental conditions and correlate it with Translation Edit Rate (TER) scores. Using the CasMaCat workbench as a post-editing tool in conjunction with a Tobii T60 eye tracker, process data were collected from 16 participants with some training on postediting. They were asked to carry out post-editing tasks under two different conditions: i) traditional post-editing (MT) and ii) interactive post-editing (IMT). In the IMT condition, as the user types, the MT system suggests alternative target translations which the post-editor can interactively accept or overwrite, whereas in the traditional MT condition no aids are provided to the user while editing the raw MT output. Temporal effort is measured by the total time spent to complete the task whereas technical effort is measured by the number of keystrokes and mouse events performed by each participant. In turn, cognitive effort is measured by fixation duration and the number of eye fixations (fixation count) in each task. Results show that IMT post-editing had significantly lower fixation duration and fewer fixation counts in comparison to traditional post-editing.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
CASMACAT: Cognitive Analysis and Statistical Methods for Advanced Computer Aided Translation was a project (2011–2014) co-funded by the European Union under the Seventh Framework Programme Project 287576 (ICT-2011.4.2).
- 2.
- 3.
- 4.
Provided that this non-parametric test is computed with the median, such value is given for each condition (PE and ITP) as follows: PE = 901,637 and ITP = 1,199,500.
References
Barrachina, S., Bender, O., Casacuberta, F., Civera, J., Cubel, E., Khadivi, S., Lagarda, A., Ney, H., Tomás, J., Vidal, E., & Vilar, J. M. (2009). Statistical approaches to computer-assisted translation. Computational Linguistics, 35(1), 3–28.
Carl, M., Dragsted, B., Elming, J., Hardt, D., & Jakobsen, A. L. (2011).The process of post-editing: A pilot study. In B. Sharp, M. Zock, M. Carl, & A. L. Jakobsen (orgs.), Proceedings of the 8th natural language processing and cognitive science workshop (Copenhagen studies in language series, Vol. 41, pp. 131–142).
Casacuberta, F., Civera, J., Cubel, E., Lagarda, A. L., Lapalme, G., Macklovitch, E., & Vidal, E. (2009). Human interaction for high quality machine translation. Communications of the ACM, 52(10), 135–138.
Duchowski, A. (2007). Eye tracking methodology: theory and practice. Clemson: Springer.
Federico, M., Cattelan, A., & Trombetti, M. (2012). Measuring user productivity in machine translation enhanced computer assisted translation. In Proceedings of the tenth conference of the association for machine translation in the americas (AMTA). AMTA 2012. Retrieved October 30, 2014.
Flournoy, R., & Duran, C. (2009). Machine translation and document localization at adobe: From pilot to production. In MT Summit XII: Proceedings of the twelfth machine translation summit.
Green, S., Heer, J., & Manning, C. D. (2013). The efficacy of human post-editing for language translation. In SIGCHI conference on human factors in computing systems (pp. 439–448). ACM.
Isabelle, P., & Church, K. (1998). Special issue on: New tools for human translators. Machine Translation, 12(1/2).
Jakobsen, A. L., & Jensen, K. T. H. (2008). Eye movement behaviour across four different types of reading task. Copenhagen Studies in Language, 36, 103–124.
Kay, M., Gawron, J. M., & Norvig, P. (1994). Verbmobil: A translation system for face-to face dialog. Stanford: Center for the Study of Language and Information.
Koehn, P. (2009). A process study of computer-aided translation. Machine Translation, 23(4), 241–263.
Krings, H. (2001). Repairing texts: Empirical investigations of machine translation port-editing processes (Trans. G. Koby, G. Shreve, K. Mischericow, & S.~Litzar). Ohio: Kent State University Press.
Lacruz, I., Gregory, M. S., & Angelone, E. (2012). Average pause ratio as an indicator of cognitive effort in post-editing: A case study. In S. O’Brien, M. Simard, & L. Specia (Eds), Proceedings of the AMTA 2012 workshop on post-editing technology and practice (WPTP 2012). Retrieved from http://amta2012.amtaweb.org/AMTA2012Files/html/2/2_paper
Langlais, P., & Lapalme, G. (2002). TransType: development-evaluation cycles to boost translator’s productivity. Machine Translation, 17(2), 77–98.
Mesa-Lao, B. (2013). Introduction to post-editing – The CasMaCat GUI. Retrieved from http://bridge.cbs.dk/projects/seecat/material/hand-out_post-editing_bmesa-lao.pdf
O’Brien, S. (2004) Machine translatability and post-editing effort: How do they relate? In Translating and the computer. London: Aslib.
O’Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19, 37–58.
O’Brien, S. (2007). An empirical investigation of temporal and technical post-editing effort. Translation and Interpreting Studies, 2(1), 83–136.
O’Brien, S. (2006). Pauses as indicators of cognitive effort in post-editing machine translation output. Across Language and Cultures, 7(1), 1–21.
Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. In The Prague bulletin of mathematical linguistics no. 93 (pp. 7–16). ISBN 978-80-904175-4-0. doi:10.2478/v10108-010-0010-x.
Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52, 591–611.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of AMTA-2006 (pp. 223–231).
Acknowledgements
The work described in this chapter was carried out within the framework of the EU project CASMACAT: Cognitive Analysis and Statistical Methods for Advanced Computer Aided Translation, funded by the European Union 7th Framework Programme Project 287576 (ICT-2011.4.2). Website: http://www.casmacat.eu. Brazilian researchers were funded by CNPq, the Brazilian Research Council (grant 307964/2011-6), and FAPEMIG, the Research Agency of the State of Minas Gerais (grant SHA/PPM-00170-14).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Alves, F. et al. (2016). Analysing the Impact of Interactive Machine Translation on Post-editing Effort. In: Carl, M., Bangalore, S., Schaeffer, M. (eds) New Directions in Empirical Translation Process Research. New Frontiers in Translation Studies. Springer, Cham. https://doi.org/10.1007/978-3-319-20358-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-20358-4_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20357-7
Online ISBN: 978-3-319-20358-4
eBook Packages: Computer ScienceComputer Science (R0)