Skip to main content
SpringerLink
Log in

Measuring Difficulty in Translation and Post-editing: A Review

  • Chapter
  • First Online:
Book cover Researching Cognitive Processes of Translation

Part of the book series: New Frontiers in Translation Studies ((NFTS))

Abstract

Difficulty (or called mental load, cognitive effort) has been an importance topic in translation and interpreting process research. This article first clarifies conceptual issues and reviews difficulty, mental workload, cognitive load and other related terms, their histories and theories. Under the umbrella of cognitive science, it then reviews two lines of research, i.e., difficulty in human translation and in postediting of machine translation. Studies concerning methods for measuring difficulty in human translation and post-editing are presented and critically examined. Two assumptions in translation difficulty research are described towards the end of this article.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Alamargot, D., Chesnet, D., Dansac, C., & Ros, C. (2006). Eye and pen: A new device for studying reading during writing. Behavior Research Methods, 38(2), 287–299.

    Article  Google Scholar 

  • Alamargot, D., Dansac, C., Chesnet, D., & Fayol, M. (2007). Parallel processing before and after pauses: A combined analysis of graphomotor and eye movements during procedural text production. In M. Torrance, L. Van Waes, & D. Galbraith (Eds.), Writing and cognition: Research and applications (pp. 13–29). Amsterdam: Elsevier.

    Google Scholar 

  • Allen, J. (2003). Post-editing. In H. Somers (Ed.), Computers and translation: A translator’s guide (pp. 297–318). Amsterdam: John Benjamins.

    Chapter  Google Scholar 

  • Alves, F. (2015). Translation process research at the interface. In A. Ferreira & J. W. Schwieter (Eds.), Psycholinguistic and cognitive inquiries into translation and interpreting (pp. 17–39). Amsterdam: John Benjamins.

    Google Scholar 

  • Alves, F., Pagano, A., & da Silva, I. (2014). Effortful text production in translation: A study of grammatical (de)metaphorization drawing on product and process data. Translation and Interpreting Studies, 9(1), 25–51.

    Article  Google Scholar 

  • Anagnostou, N. K., & Weir, G. R. S. (2007). From corpus-based collocation frequencies to readability measure. In G. R. S. Weir & T. Ozasa (Eds.), Texts, textbooks and readability (pp. 34–48). Glasgow: University of Stratchclyde Publishing.

    Google Scholar 

  • Arenas, A. G. (2014). The role of professional experience in post-editing from a quality and productivity perspective. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 51–76). Newcastle: Cambridge Scholars Publishing.

    Google Scholar 

  • Aziz, W., & Specia, L. (2012). PET: A standalone tool for assessing machine translation through post-editing. Paper presented at the Translating and The Computer 34, London.

    Google Scholar 

  • Aziz, W., Mitkov, R., & Specia, L. (2013). Ranking machine translation systems via post-editing. In I. Habernal & V. Matoušek (Eds.), Text, speech, and dialogue (pp. 410–418). London: Springer.

    Chapter  Google Scholar 

  • Aziz, W., Koponen, M., & Specia, L. (2014). Sub-sentence level analysis of machine translation post-editing effort. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 170–199). Newcastle: Cambridge Scholars Publishing.

    Google Scholar 

  • Baddeley, A. D., Eysenck, M. W., & Anderson, M. C. (2015). Memory (2nd ed.). London: Psychology Press.

    Book  Google Scholar 

  • Baker, M. (2011). In other words: A coursebook on translation (2nd ed.). New York: Routledge.

    Book  Google Scholar 

  • Balling, L. W., Hvelplund, K. T., & Sjørup, A. C. (2014). Evidence of parallel processing during translation. Meta, 59(2), 234–259.

    Article  Google Scholar 

  • Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Paper presented at the Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43rd Annual Meeting of the Association of Computational Linguistics (ACL-2005), Ann Arbor, Michigan.

    Google Scholar 

  • Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276–292.

    Article  Google Scholar 

  • Bermúdez, J. L. (2014). Cognitive science: An introduction to the science of the mind (2nd ed.). Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Block, R. A., Hancock, P. A., & Zakay, D. (2010). How cognitive load affects duration judgments: A meta-analytic review. Acta Psychologica, 134(3), 330–343.

    Article  Google Scholar 

  • Bradshaw, J. L. (1968). Load and pupillary changes in continuous processing tasks. British Journal of Psychology, 59(3), 265–271.

    Article  Google Scholar 

  • Broadbent, D. E. (1958). Perception and communication. London: Pergamon Press.

    Book  Google Scholar 

  • Brünken, R. E., Plass, J. L., & Moreno, R. E. (2010). Current issues and open questions in cognitive load research. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Busse, L. M., & Buhmann, J. M. (2011). Model-based clustering of inhomogeneous paired comparison data. In M. Pelillo & E. R. Hancock (Eds.), Similarity-based pattern recognition (pp. 207–221). Berlin: Springer.

    Chapter  Google Scholar 

  • Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J. (2008). Further meta-evaluation of machine translation. In Proceedings of the third workshop on statistical machine translation (pp. 70–106). Columbus: Association for Computational Linguistics.

    Chapter  Google Scholar 

  • Campbell, S. (1999). A cognitive approach to source text difficulty in translation. Target, 11(1), 33–63.

    Article  Google Scholar 

  • Campbell, S. (2000). Choice network analysis in translation research. In M. Olohan (Ed.), Intercultural faultlines: Research models in translation studies: Textual and cognitive aspects (pp. 29–42). Manchester: St. Jerome.

    Google Scholar 

  • Campbell, S., & Hale, S. (1999). What makes a text difficult to translate? Refereed Proceedings of the 23rd Annual ALAA Congress. Retrieved March 1, 2015, from http://www.atinternational.org/forums/archive/index.php/t-887.html

  • Cara, F. (1999). Cognitive ergonomics. In R. A. Wilson & F. C. Keil (Eds.), The MIT encyclopedia of the cognitive sciences (pp. 130–132). Cambridge: MIT Press.

    Google Scholar 

  • Carl, M., Bangalore, S., & Schaeffer, M. (2015). New directions in empirical translation process research: Exploring the CRITT TPR-DB. New York: Springer.

    Google Scholar 

  • Cassenti, D. N., & Kelley, T. D. (2006). Towards the shape of mental workload. Paper presented at the Human Factors and Ergonomics Society Annual Meeting, Boston, MA.

    Article  Google Scholar 

  • Cassenti, D. N., Kelley, T. D., & Carlson, R. A. (2013). Differences in performance with changing mental workload as the basis for an IMPRINT plug-in proposal. Paper presented at the 22nd Annual Conference on Behavior Representation in Modeling and Simulation, Ottawa, Canada.

    Google Scholar 

  • Cassin, B. (2014). Dictionary of untranslatables: A philosophical lexicon. Princeton: Princeton University Press.

    Book  Google Scholar 

  • Chall, J. S., & Dale, E. (1995). Readability revisited: The new Dale-Chall readability formula. Cambridge: Brookline Books.

    Google Scholar 

  • Charlton, S. G. (2002). Measurement of cognitive states in test and evaluation. In S. G. Charlton & T. G. O’Brien (Eds.), Handbook of human factors testing and evaluation (2nd ed., pp. 97–126). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Common Sense Advisory. (2014). Ten concepts and data points to remember in 2014. MultiLingual, 1, 37-38.

    Google Scholar 

  • Conway, A. R. A., Cowan, N., Bunting, M. F., Therriault, D. J., & Minkoff, S. R. B. (2002). A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence, 30(2), 163–183.

    Article  Google Scholar 

  • Conway, A. R. A., Kane, M. J., Bunting, M. F., Hambrick, D. Z., Wilhelm, O., & Engle, R. W. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12(5), 769–786.

    Article  Google Scholar 

  • Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19(4), 450–466.

    Article  Google Scholar 

  • De Almeida, G. (2013). Translating the post-editor: An investigation of post-editing changes and correlations with professional experience across two Romance languages. PhD thesis. Dublin City University, Dublin.

    Google Scholar 

  • De Waard, D., & Lewis-Evans, B. (2014). Self-report scales alone cannot capture mental workload. Cognition, Technology & Work, 16(3), 303–305.

    Article  Google Scholar 

  • DeKeyser, R. (2003). Implicit and explicit learning. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 313–348). Oxford: Blackwell.

    Google Scholar 

  • Denkowski, M., & Lavie, A. (2011). Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the 6th workshop on statistical machine translation (pp. 85–91). Edinburgh: Association for Computational Linguistics.

    Google Scholar 

  • Denkowski, M., & Lavie, A. (2012). TransCenter: Web-based translation research suite. Retrieved April 1, 2015, from https://www.cs.cmu.edu/~mdenkows/pdf/transcenter-amta2012.pdf

  • DePalma, D. A., & Hegde, V. (2010). The market for MT post-editing. Lowell: Common Sense Advisory.

    Google Scholar 

  • DePalma, D. A., & Kelly, N. (2009). The business case for machine translation. Lowell: Common Sense Advisory.

    Google Scholar 

  • Doherty, S., O’Brien, S., & Carl, M. (2010). Eye tracking as an MT evaluation technique. Machine Translation, 24(1), 1–13.

    Article  Google Scholar 

  • Dorr, B., Olive, J., McCary, J., & Christianson, C. (2011). Machine translation evaluation and optimization. In J. Olive, C. Christianson, & J. McCary (Eds.), Handbook of natural language processing and machine translation (pp. 745–843). New York: Springer.

    Chapter  Google Scholar 

  • Dragsted, B. (2004). Segmentation in translation and translation memory systems: An empirical investigation of cognitive segmentation and effects of integrating a TM system into the translation process. PhD thesis. Copenhagen Business School, Frederiksberg, Denmark.

    Google Scholar 

  • Dragsted, B. (2012). Indicators of difficulty in translation: Correlating product and process data. Across Languages and Cultures, 13(1), 81–98.

    Article  Google Scholar 

  • Embrey, D., Blackett, C., Marsden, P., & Peachey, J. (2006). Development of a human cognitive workload assessment tool: MCA final report. Dalton: Human Reliability Associates.

    Google Scholar 

  • Engle, R. W., & Kane, M. J. (2004). Executive attention, working memory capacity, and a two-factor theory of cognitive control. In B. Ross (Ed.), The psychology of learning and motivation (pp. 145–199). New York: Elsevier.

    Google Scholar 

  • Englund Dimitrova, B. (2005). Expertise and explicitation in the translation process. Amsterdam: John Benjamins.

    Book  Google Scholar 

  • Felice, M., & Specia, L. (2012). Linguistic features for quality estimation. In Proceedings of the 7th workshop on statistical machine translation (pp. 96–103). Montréal: Association for Computational Linguistics.

    Google Scholar 

  • Fiedler, S., Glöckner, A., & Nicklisch, A. (2012). The influence of social value orientation on information processing in repeated voluntary contribution mechanism games: An eye-tracking analysis. In A. Innocenti & A. Sirigu (Eds.), Neuroscience and the Economics of Decision Making (pp. 21–53). London: Routledge.

    Google Scholar 

  • Frankish, K., & Ramsey, W. (Eds.). (2012). The Cambridge handbook of cognitive science. Cambridge: Cambridge University Press.

    Google Scholar 

  • Freeman, G. L., & Giese, W. J. (1940). The relationship between task difficulty and palmar skin resistance. The Journal of General Psychology, 23(1), 217–220.

    Article  Google Scholar 

  • Freixa, J. (2006). Causes of denominative variation in terminology: A typology proposal. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 12(1), 51–77.

    Google Scholar 

  • Fry, E. B. (1988). Writeability: The principles of writing for increased comprehension. In B. L. Zakaluk & S. J. Samuels (Eds.), Readability: Its past, present, and future (pp. 77–95). Newark: International Reading Association.

    Google Scholar 

  • Gallupe, R. B., DeSanctis, G., & Dickson, G. W. (1988). Computer-based support for group problem-finding: An experimental investigation. MIS Quarterly, 12(2), 277–296.

    Article  Google Scholar 

  • Gile, D. (1999). Testing the Effort Models’ tightrope hypothesis in simultaneous interpreting – A contribution. Hermes, 23, 153–172.

    Google Scholar 

  • Gile, D. (2009). Basic concepts and models for interpreter and translator training (Rev. Ed.). Amsterdam: John Benjamin.

    Google Scholar 

  • Goldberg, J. H., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner’s guide. In R. Radach, J. Hyona, & H. Deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 493–516). Amsterdam: Elsevier.

    Chapter  Google Scholar 

  • Göpferich, S., Jakobsen, A. L., & Mees, I. M. (Eds.). (2008). Looking at eyes: Eye-tracking studies of reading and translation processing. Copenhagen: Sammfundslitteratur.

    Google Scholar 

  • Gopher, D. (1994). Analysis and measurement of mental load. In G. d’Ydewalle, P. Eelen, & P. Bertelson (Eds.), International perspectives on psychological science, Vol. II: The state of the art (pp. 265–292). East Sussex: Lawrence Erlbaum.

    Google Scholar 

  • Gray, W. S., & Leary, B. E. (1935). What makes a book readable. Chicago: The University of Chicago Press.

    Google Scholar 

  • Hale, S., & Campbell, S. (2002). The interaction between text difficulty and translation accuracy. Babel, 48(1), 14–33.

    Article  Google Scholar 

  • Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.

    Chapter  Google Scholar 

  • Hummel, K. M. (2002). Second language acquisition and working memory. In F. Fabbro (Ed.), Advances in the neurolinguistics of bilingualism (pp. 95–117). Udine: Forum.

    Google Scholar 

  • Hvelplund, K. T. (2011). Allocation of cognitive resources in translation: An eye-tracking and key-logging study. PhD thesis. Copenhagen Business School, Frederiksberg, Denmark.

    Google Scholar 

  • Ilkowska, M., & Engle, R. W. (2010). Trait and state differences in working memory capacity. In A. Gruszka, G. Matthews, & B. Szymura (Eds.), Handbook of individual differences in cognition (pp. 295–320). New York: Springer.

    Chapter  Google Scholar 

  • International Ergonomics Association. (2015). Definition and domains of ergonomics. Retrieved March 1, 2015, from http://www.iea.cc/whats/

  • Ivir, V. (1981). Formal correspondence vs. translation equivalence revisited. Poetics Today, 2(4), 51–59.

    Article  Google Scholar 

  • Jakobsen, A. L. (2011). Tracking translators’ keystrokes and eye movements with Translog. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and strategies of process research (pp. 37–55). Amsterdam: John Benjamins.

    Chapter  Google Scholar 

  • Jensen, K. T. (2009). Indicators of text complexity. In S. Göpferich, A. L. Jakobsen, & I. M. Mees (Eds.), Behind the mind: Methods, models and results in translation process research (pp. 61–80). Amsterdam: John Benjamins.

    Google Scholar 

  • Jex, H. R. (1988). Measuring mental workload: Problems, progress, and promises. In P. A. Hancock & N. Meshkati (Eds.), Hman mental workload (pp. 5–38). Amsterdam: North-Holland.

    Chapter  Google Scholar 

  • Just, M. A., & Carpenter, P. A. (1976). Eye fixations and cognitive processes. Cognitive Psychology, 8(4), 441–480.

    Article  Google Scholar 

  • Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354.

    Article  Google Scholar 

  • Kalsbeek, J. W. H., & Sykes, R. N. (1967). Objective measurement of mental load. Acta Psychologica, 27, 253–261.

    Article  Google Scholar 

  • Kalyuga, S. (2009). Managing cognitive load in adaptive multimedia learning. Hershey: Information Science Reference.

    Book  Google Scholar 

  • Kanazawa, S. (1998). In defense of unrealistic assumptions. Sociological Theory, 16(2), 193–204.

    Article  Google Scholar 

  • Karwowski, W. (2012). The discipline of human factors and ergonomics. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 1–37). Hoboken: Wiley.

    Google Scholar 

  • Kit, C. Y., & Wong, B. T. M. (2015). Evaluation in machine translation and computer-aided translation. In S. W. Chan (Ed.), Routledge encyclopedia of translation technology (pp. 213–236). London: Routledge.

    Google Scholar 

  • Klare, G. R. (1984). Readability. In P. D. Pearson & R. Barr (Eds.), Handbook of reading research (pp. 681–744). New York: Longman.

    Google Scholar 

  • Koehn, P. (2010). Statistical machine translation. New York: Cambridge University Press.

    Google Scholar 

  • Koehn, P., & Germann, U. (2014). The impact of machine translation quality on human post-editing. Paper presented at the Workshop on Humans and Computer-Assisted Translation (HaCaT), Gothenburg, Sweden.

    Google Scholar 

  • Koponen, M. (2012). Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the 7th Workshop on Statistical Machine Translation (pp. 181–190). Montreal: Association for Computational Linguistics.

    Google Scholar 

  • Koponen, M., Aziz, W., Ramos, L., & Specia, L. (2012). Post-editing time as a measure of cognitive effort. Paper presented at the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), San Diego.

    Google Scholar 

  • Krings, H. P. (2001). Repairing texts: Empirical investigations of machine translation post-editing processes. (G. Koby, G. Shreve, K. Mischerikow & S. Litzer, Trans.). Kent, Ohio: Kent State University Press.

    Google Scholar 

  • Kuiken, F., & Vedder, I. (2007). Task complexity needs to be distinguished from task difficulty. In M. D. P. GarcíaMayo (Ed.), Investigating tasks in formal language learning (pp. 117–135). Clevedon: Multilingual Matters.

    Google Scholar 

  • Lacruz, I., & Shreve, G. M. (2014). Pauses and cognitive effort in post-editing. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 246–272). Newcastle: Cambridge Scholars Publishing.

    Google Scholar 

  • Lacruz, I., Shreve, G. M., & Angelone, E. (2012). Average pause ratio as an indicator of cognitive effort in post-editing: A case study. Paper presented at the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), San Diego.

    Google Scholar 

  • Liu, M., & Chiu, Y.-H. (2009). Assessing source material difficulty for consecutive interpreting: Quantifiable measures and holistic judgment. Interpreting, 11(2), 244–266.

    Article  Google Scholar 

  • Liu, Q., & Zhang, X. (2015). Machine translation: General. In S. W. Chan (Ed.), Routledge encyclopedia of translation technology (pp. 105–119). London: Routledge.

    Google Scholar 

  • Lusk, M. M., & Atkinson, R. K. (2007). Animated pedagogical agents: Does their degree of embodiment impact learning from static or animated worked examples? Applied Cognitive Psychology, 21(6), 747–764.

    Article  Google Scholar 

  • Martins, D. B., & Caseli, H. (2015). Automatic machine translation error identification. Machine Translation, 29(1), 1–24.

    Article  Google Scholar 

  • Mesa-Lao, B. (2013). Introduction to post-editing–The CasMaCat GUI. Retrieved March 1, 2015 from http://bridge.cbs.dk/projects/seecat/material/hand-out_post-editing_bmesa-lao.pdf

  • Meshkati, N. (1988). Toward development of a cohesive model of workload. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 305–314). Amsterdam: North-Holland.

    Chapter  Google Scholar 

  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.

    Article  Google Scholar 

  • Mishra, A., Bhattacharyya, P., & Carl, M. (2013, August 4–9). Automatically predicting sentence translation difficulty. Paper presented at the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria.

    Google Scholar 

  • Moors, A., & De Houwer, J. (2006). Automaticity: A theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326.

    Article  Google Scholar 

  • Moran, J., & Lewis, D. (2011). Unobtrusive methods for low-cost manual evaluation of machine translation. Retrieved April 1, 2015 from http://lodel.irevues.inist.fr/tralogy/index.php?id=141&format=print

  • Moray, N. (1977). Models and measures of mental workload. In N. Moray (Ed.), Mental workload: Its theory and measurement (pp. 13–21). New York: Springer.

    Google Scholar 

  • Muñoz Martín, R. (2010). Leave no stone unturned: On the development of cognitive translatology. Translation and Interpreting Studies, 5(2), 145–162.

    Article  Google Scholar 

  • Muñoz Martín, R. (2012). Just a matter of scope. Translation Spaces, 1(1), 169–188.

    Article  Google Scholar 

  • Muñoz Martín, R. (2014). A blurred snapshot of advances in translation process research. MonTI. Special Issue (Minding Translation), 1, 49–84.

    Google Scholar 

  • Newell, W. H. (2001). A theory of interdisciplinary studies. Issues in Integrative Studies, 19, 1–25.

    Google Scholar 

  • Nimon, K. F. (2012). Statistical assumptions of substantive analyses across the general linear model: A mini-review. Frontiers in Psychology, 3, 1–5.

    Google Scholar 

  • Nkwake, A. M. (2013). Working with assumptions in international development program evaluation. New York: Springer.

    Book  Google Scholar 

  • Nord, C. (2005). Text analysis in translation: Theory, methodology, and didactic application of a model for translation-oriented text analysis (2nd ed.). Amsterdam: Rodopi.

    Google Scholar 

  • O’Brien, S. (2004). Machine Translatability and Post-Editing Effort: How do they relate? Paper presented at the 26th Translating and the Computer Conference (ASLIB), London.

    Google Scholar 

  • O’Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19(1), 37–58.

    Article  Google Scholar 

  • O’Brien, S. (2006). Pauses as indicators of cognitive effort in post-editing machine translation output. Across Languages and Cultures, 7(1), 1–21.

    Article  Google Scholar 

  • O’Brien, S. (2007a). An empirical investigation of temporal and technical post-editing effort. Translation and Interpreting Studies, 2(1), 83–136.

    Article  Google Scholar 

  • O’Brien, S. (2007b). Eye-tracking and translation memory matches. Perspectives, 14(3), 185–205.

    Article  Google Scholar 

  • O’Brien, S. (2010). Controlled language and readability. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 143–165). Amsterdam: John Benjamins.

    Chapter  Google Scholar 

  • O’Brien, S. (2011). Cognitive explorations of translation. London: Continuum.

    Google Scholar 

  • O’Brien, S., Balling, L. W., Carl, M., Simard, M., & Specia, L. (Eds.). (2014). Post-editing of machine translation: Processes and applications. Newcastle: Cambridge Scholars Publishing.

    Google Scholar 

  • O’Donnell, R. D., & Eggemeier, F. T. (1986). Workload assessment methodology. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, Vol. II: Cognitive processes and performance (pp. 42/41–42–49). New York: Wiley.

    Google Scholar 

  • Onwuegbuzie, A. J., & Daniel, L. G. (1999, November 17–19). Uses and misuses of the correlation coefficient. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, Point Clear, AL.

    Google Scholar 

  • Ortiz-Martínez, D., Sanchis-Trilles, G., Casacuberta, F., Alabau, V., Vidal, E., Benedı, J.-M … González, J. (2012). The CASMACAT project: The next generation translator’s workbench. Paper presented at the 7th Jornadas en Tecnologıa del Habla and the 3rd Iberian SLTech Workshop (IberSPEECH), Madrid.

    Google Scholar 

  • Orzechowski, J. (2010). Working memory capacity and individual differences in higher-level cognition. In G. Matthews & B. Szymura (Eds.), Handbook of individual differences in cognition (pp. 353–368). New York: Springer.

    Chapter  Google Scholar 

  • Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434.

    Article  Google Scholar 

  • Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994a). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6(4), 351–371.

    Article  Google Scholar 

  • Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994b). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122–133.

    Article  Google Scholar 

  • Paas, F. G. W. C., Ayres, P., & Pachman, M. (2008). Assessment of cognitive load in multimedia learning. In D. H. Robinson & G. Schraw (Eds.), Assessment of cognitive load in multimedia learning: Theory, methods and applications (pp. 11–35). Charlotte, NC: Information Age Publishing.

    Google Scholar 

  • Palumbo, G. (2009). Key terms in translation studies. London: Continuum.

    Google Scholar 

  • Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (pp. 311–318). University of Pennsylvania, Philadelphia: Association for Computational Linguistics.

    Google Scholar 

  • Poole, A., & Ball, L. J. (2006). Eye tracking in HCI and usability research. In C. Ghaoui (Ed.), Encyclopedia of human computer interaction (pp. 211–219). London: Idea Group.

    Chapter  Google Scholar 

  • Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422.

    Article  Google Scholar 

  • Rayner, K., & Pollatsek, A. (1989). Psychology of reading. Hillsdale: Lawrence Erlbaum.

    Google Scholar 

  • Redfield, C. L. (1922). Mental levels. Journal of Education, 95(8), 214–216.

    Article  Google Scholar 

  • Reichle, E. D., Reineberg, A. E., & Schooler, J. W. (2010). Eye movements during mindless reading. Psychological Science, 21(9), 1300–1310.

    Article  Google Scholar 

  • Robinson, P. (2001). Task complexity, cognitive resources, and syllabus design: A triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 287–318). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Rosa, R. (2014). Depfix, a tool for automatic rule-based post-editing of SMT. The Prague Bulletin of Mathematical Linguistics, 102(1), 47–56.

    Article  Google Scholar 

  • Rost, M. (2006). Areas of research that influence L2 listening instruction. In E. Usó Juan & A. Martínez Flor (Eds.), Current trends in the development and teaching of the four language skills (pp. 47–74). Berlin: Mouton de Gruyter.

    Chapter  Google Scholar 

  • Schaeffer, M., & Carl, M. (2013). Shared representations and the translation process: A recursive model. Translation and Interpreting Studies, 8(2), 169–190.

    Article  Google Scholar 

  • Schaeffer, M., & Carl, M. (2014). Measuring the cognitive effort of literal translation processes. Paper presented at the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden.

    Google Scholar 

  • Schilperoord, J. (1996). It’s about time: Temporal aspects of cognitive processes in text production. Amsterdam: Rodopi.

    Google Scholar 

  • Schleiermacher, F. (2012). On the different methods of translating (S. Bernofsky, Trans.). In L. Venuti (Ed.), The translation studies reader (3rd ed., pp. 43–63). London: Routledge.

    Google Scholar 

  • Sharmin, S., Špakov, O., Räihä, K.-J., & Jakobsen, A. L. (2008). Where on the screen do translation students look while translating, and for how long? In S. Göpferich, A. L. Jakobsen, & I. M. Mees (Eds.), Looking at eyes: Eye-tracking studies of reading and translation processing (pp. 31–51). Copenhagen: Samfundslitteratur.

    Google Scholar 

  • Shreve, G. M. (2002). Knowing translation: Cognitive and experiential aspects of translation expertise from the perspective of expertise studies. In A. Ruiccardi (Ed.), Translation studies: Perspectives on an emerging discipline (pp. 150–173). Cambridge: Cambridge University Press.

    Google Scholar 

  • Shreve, G. M., & Angelone, E. (Eds.). (2010). Translation and cognition. Amsterdam: John Benjamins.

    Google Scholar 

  • Silveira, F. d. S. D. d. (2011). Working memory capacity and lexical access in advanced students of L2 English. PhD thesis. Universidade Federal do Rio Grande do Sul, Brazil. Retrieved from http://www.lume.ufrgs.br/bitstream/handle/10183/39423/000824076.pdf?sequence=1

  • Sirén, S., & Hakkarainen, K. (2002). Expertise in translation. Across Languages and Cultures, 3(1), 71–82.

    Article  Google Scholar 

  • Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (pp. 223–231). Cambridge, MA.

    Google Scholar 

  • Sousa, S. C. M. d., Aziz, W. F., & Specia, L. (2011). Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In Proceedings of the International Conference of Recent Advances in Natural Language Processing (pp. 97–103). Bulgaria.

    Google Scholar 

  • Specia, L. (2011). Exploiting objective annotations for measuring translation post-editing effort. Paper presented at the 15th Conference of the European Association for Machine Translation, Leuven.

    Google Scholar 

  • Specia, L., Raj, D., & Turchi, M. (2010). Machine translation evaluation versus quality estimation. Machine Translation, 24(1), 39–50.

    Article  Google Scholar 

  • Stymne, S. (2011). Blast: A tool for error analysis of machine translation output. Paper presented at the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon.

    Google Scholar 

  • Sun, S. (2011). Think-aloud-based translation process research: Some methodological considerations. Meta, 56(4), 928–951.

    Article  Google Scholar 

  • Sun, S. (2015). Measuring translation difficulty: Theoretical and methodological considerations. Across Languages and Cultures, 16(1), 29–54.

    Article  Google Scholar 

  • Sun, S., & Shreve, G. M. (2014). Measuring translation difficulty: An empirical study. Target, 26(1), 98–127.

    Article  Google Scholar 

  • Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.

    Article  Google Scholar 

  • Tatler, B. W., Kirtley, C., Macdonald, R. G., Mitchell, K. M., & Savage, S. W. (2014). The active eye: Perspectives on eye movement research. In M. Horsley, M. Eliot, B. A. Knight, & R. Reilly (Eds.), Current trends in eye tracking research (pp. 3–16). London: Springer.

    Chapter  Google Scholar 

  • TAUS. (2010). MT post-editing guidelines. Retrieved March 1, 2015, from https://www.taus.net/think-tank/best-practices/postedit-best-practices/machine-translation-post-editing-guidelines

  • TAUS. (2014). Post-editing: Championing MT. Retrieved March 1, 2015 from https://postedit.taus.net/

  • Thorndike, E. L., Bregman, E. O., Cobb, M. V., & Woodyard, E. (1927). The measurement of intelligence. New York: Bureau of Publications, Columbia University.

    Google Scholar 

  • Tirkkonen-Condit, S. (2005). The monitor model revisited: Evidence from process research. Meta, 50(2), 405–414.

    Article  Google Scholar 

  • Tokowicz, N., Kroll, J. F., De Groot, A. M. B., & Van Hell, J. G. (2002). Number-of-translation norms for Dutch – English translation pairs: A new tool for examining language production. Behavior Research Methods, Instruments, & Computers, 34(3), 435–451.

    Article  Google Scholar 

  • Tomporowski, P. D. (2003). Performance and perceptions of workload among young and older adults: Effects of practice during cognitively demanding tasks. Educational Gerontology, 29(5), 447–466.

    Article  Google Scholar 

  • Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory and Language, 28(2), 127–154.

    Article  Google Scholar 

  • Vasconcellos, M. (1987). A comparison of MT post-editing and traditional revision. In K. Kummer (Ed.), Proceedings of the 28th annual conference of the American Translators Association (pp. 409-416). Medford: Learned Information.

    Google Scholar 

  • Vidulich, M. A., & Tsang, P. S. (2012). Mental workload and situation awareness. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (4th ed., pp. 243–273). Hoboken: Wiley.

    Chapter  Google Scholar 

  • Vieira, L. N. (2014). Indices of cognitive effort in machine translation post-editing. Machine Translation, 28(3-4), 187–216.

    Article  Google Scholar 

  • Vonk, W., & Cozijn, R. (2003). On the treatment of saccades and regressions in eye movement measures of reading time. In J. Hyona, R. Radach, & H. deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 291–312). London: Elsevier.

    Chapter  Google Scholar 

  • Wierwille, W. W., & Williges, B. H. (1980). An annotated bibliography on operator mental workload assessment (Naval Air Test Center Report No. SY-27R-80). Patuxent River: Naval Air Test Center, System Engineering Test Directorate.

    Google Scholar 

  • Wilson, R. A., & Keil, F. C. (Eds.). (1999). The MIT encyclopedia of the cognitive sciences. Cambridge: MIT Press.

    Google Scholar 

  • Wilss, W. (1982). The science of translation: Problems and methods. Tübingen: Gunter Narr.

    Google Scholar 

  • Wisniewski, G., Kübler, N., & Yvon, F. (2014). A corpus of machine translation errors extracted from translation students exercises. Paper presented at the International Conference on Language Resources and Evaluation (LREC), Iceland. http://www.lrec-conf.org/proceedings/lrec2014/pdf/1115_Paper.pdf

  • Woodrow, H. (1936). The measurement of difficulty. Psychological Review, 43(4), 341–365.

    Article  Google Scholar 

  • Zipf, G. K. (1935). The psycho-biology of language: An introduction to dynamic philology. Boston: Houghton Mifflin.

    Google Scholar 

Download references

Acknowledgment

This work was supported by the Young Faculty Research Fund of Beijing Foreign Studies University (Grant No. 2016JT004) and by the Fundamental Research Funds for the Central Universities (Grant No. 2015JJ003).

Author information

Authors and Affiliations

  1. School of English and International Studies, Beijing Foreign Studies University, Beijing, China

    Sanjun Sun

Authors
  1. Sanjun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Centre for Studies of Translation, Interpreting and Cognition, University of Macau, Macau SAR, China

    Defeng Li

  2. University of Macau, Macau, China

    Victoria Lai Cheng Lei

  3. University of South China, Hunan, China

    Yuanjian He

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Sun, S. (2019). Measuring Difficulty in Translation and Post-editing: A Review. In: Li, D., Lei, V., He, Y. (eds) Researching Cognitive Processes of Translation. New Frontiers in Translation Studies. Springer, Singapore. https://doi.org/10.1007/978-981-13-1984-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-1984-6_7

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-1983-9

  • Online ISBN: 978-981-13-1984-6

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation