Skip to main content
SpringerLink
Log in

Using texts generated by STR and CAT to facilitate student comprehension of lecture content in a foreign language

  • Published:
Journal of Computing in Higher Education Aims and scope Submit manuscript

Abstract

In this study, we applied a combination of speech-to-text recognition (STR) and computer-aided translation (CAT) technologies during lectures in English as a foreign language to facilitate student comprehension of the lecture content. The instructor lectured in English, the STR system generated texts from the voice input, and the CAT system then simultaneously translated the STR texts into the students’ native language. We aimed to test the feasibility of applying combined STR and CAT technologies to facilitate student comprehension of lecture content in a foreign language. To this end, we designed an experiment. Three groups with twenty students each were formed. All students attended the same lectures: (a) students in the control group attended lectures without any support, (b) students in experimental group 1 attended lectures with STR support (i.e., they were presented with texts in English generated from the instructor’s speech by STR), and (c) students in experimental group 2 attended lectures with STR and CAT support (i.e., they were presented with texts in their native language that were translated from English by STR and CAT). We compared the posttest results of the students in the three groups. We also explored the effects of our approach with respect to different levels of foreign language ability. Finally, we surveyed the perceptions of students in experimental group 2 about the usefulness of the translated texts for their learning. Our results showed that applying STR and CAT technologies together was a useful approach: the translated texts helped significantly improve student learning performance compared to that of the students in the control condition. Translated texts were beneficial for students, as the students were able (a) to confirm some words that were not clearly spoken by the instructor or to find the meaning of some words with which the students were not familiar and (b) to complement spoken lecture content with translated content to help information processing and enhance comprehension. When comparing students with different language abilities, we found that students with low language abilities benefited from the translated texts the most. The students’ language ability was significantly lower than that of the high-ability students before the experiment; however, the low-ability students’ learning performance showed no significant difference from the high-ability students after the experiment. Finally, most students perceived translated texts as useful for their learning, and they intended to use the texts in the future for learning purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bloomfield, A., Wayland, S. C., Rhoades, E., Blodgett, A., Linck, J., & Ross, S. (2010). What makes listening difficult? Factors affecting second language listening comprehension. College Park: University of Maryland.

    Book  Google Scholar 

  • Camiciottoli, B. C. (2005). Adjusting a business lecture for an international audience: A case study. English for Specific Purposes, 24(2), 183–199.

    Article  Google Scholar 

  • Chen, S. J. (2004). Linguistic dimensions of subtitling. Perspectives from Taiwan. Meta: Journal des traducteursMeta:/Translators’ Journal, 49(1), 115–124.

    Article  Google Scholar 

  • Chen, I. J., & Chang, C. C. (2009). Cognitive load theory: An Empirical study of anxiety and task performance in language learning. Electronic Journal of Research in Educational Psychology, 7(2), 729–746.

    Google Scholar 

  • Chen, I. J., Chang, C. C., & Yen, J. C. (2012). Effects of presentation mode on mobile language learning: A performance efficiency perspective. Australasian Journal of Educational Technology, 28(1), 122–137.

    Article  Google Scholar 

  • Cheng, X. (2000). Asian students’ reticence revisited. System, 28(3), 435–446.

    Article  Google Scholar 

  • Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (4th ed.). San Francisco, CA: Wiley.

    Book  Google Scholar 

  • Debuse, J. C., Hede, A., & Lawley, M. (2009). Learning efficacy of simultaneous audio and on-screen text in online lectures. Australasian Journal of Educational Technology, 25(5), 748–762.

    Article  Google Scholar 

  • Diao, Y., Chandler, P., & Sweller, J. (2007). The effect of written text on comprehension of spoken English as a foreign language. The American Journal of Psychology, 120(2), 237–261.

    Google Scholar 

  • ElShiekh, A. A. A. (2012). Google translate service: transfer of meaning, distortion or simply a new creation? An investigation into the translation process & problems at google. English Language and Literature Studies, 2(1), 56–68.

    Google Scholar 

  • Ferreira, F., Bailey, K. G., & Ferraro, V. (2002). Good-enough representations in language comprehension. Current Directions in Psychological Science, 11(1), 11–15.

    Article  Google Scholar 

  • Ferreira, F., Engelhardt, P. E., & Jones, M. W. (2009). Good enough language processing: A satisficing approach. In Proceedings of the 31st annual conference of the cognitive science society. Austin: Cognitive Science Society.

  • First, E. (2013). EF English proficiency index. Retrieved on March 17, 2017.

  • Godwin-Jones, R. (2011). Emerging technologies: Mobile apps for language learning. Language Learning & Technology, 15(2), 2–11.

    Google Scholar 

  • Graham, S. (2011). Self-efficacy and academic listening. Journal of English for Academic Purposes, 10(2), 113–117.

    Article  Google Scholar 

  • Hermet, M., & Desilets, A. (2009). Using first and second language models to correct preposition errors in second language authoring. In Proceedings of the fourth workshop on innovative use of NLP for building educational applications (pp. 64–72). Association for Computational Linguistics.

  • Hwang, W. Y., Shadiev, R., Kuo, T. C. T., & Chen, N. S. (2012). Effects of speech-to-text recognition application on learning performance in synchronous cyber classrooms. Educational Technology & Society, 15(1), 367–380.

    Google Scholar 

  • Kalyuga, S. (2014). The expertise reversal principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 576–597). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.

    Article  Google Scholar 

  • Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases. Psychological Science, 23(6), 661–668.

    Article  Google Scholar 

  • Kheir, R., & Way, T. (2006, June). Improving speech recognition to assist real-time classroom note taking. In Rehabilitation Engineering Society of North America (RESNA). Conference, Atlanta.

  • Krashen, S. D. (1985). The input hypothesis: Issues and implications. Reading: Addison-Wesley Longman Ltd.

    Google Scholar 

  • Krashen, S. (2014). Case histories and the comprehension hypothesis. TESOL Journal, 9(1), 70–91.

    Google Scholar 

  • Kuo, T. C., Shadiev, R., Hwang, W. Y., & Chen, N. S. (2012). Effects of applying STR for group learning activities on learning performance in a synchronous cyber classroom. Computers & Education, 58(1), 600–608.

    Article  Google Scholar 

  • Kurz, I. (2009). The impact of non-native English on students’ interpreting performance. In G. Hansen, A. Chesterman, & H. Gerzymisch-Arbogast (Eds.), Efforts and models in interpreting and translation research (pp. 179–192). Amsterdam: John Benjamins.

    Chapter  Google Scholar 

  • Lee, C. H., & Kalyuga, S. (2011). Effectiveness of on-screen pinyin in learning Chinese: An expertise reversal for multimedia redundancy effect. Computers in Human Behavior, 27(1), 11–15.

    Article  Google Scholar 

  • Mayer, R. E. (2009). Multimedia learning. New York: Cambridge University Press.

    Book  Google Scholar 

  • Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52.

    Article  Google Scholar 

  • Mellebeek, B., Khasin, A., Owczarzak, K., Van Genabith, J., & Way, A. (2005). Improving online machine translation systems. In Proceedings of the X MT summit (pp. 290–297). Phuket: Thailand.

  • Miller, L. (2007). Issues in lecturing in a second language: Lecturer’s behaviour and students’ perceptions. Studies in Higher Education, 32(6), 747–760.

    Article  Google Scholar 

  • Nisbet, P., Wilson, A., & Aitken, S. (2005). Speech Recognition for Students with Disabilities. In Proceedings of the inclusive and supportive education congress, ISEC 2005 conference. Delph: Inclusive Technology.

  • Omar, H., Embi, M. A., & Yunus, M. M. (2012). ESL learners’ interaction in an online discussion via Facebook. Asian Social Science, 8(11), 67–74.

    Article  Google Scholar 

  • Paas, F., & Sweller, J. (2014). Implication of cognitive load theory for multimedia. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 27–42). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Parmar, J. R., Tejada, F. R., Lang, L. A., Purnell, M., Acedera, L., & Ngonga, F. (2015). Assessment of communications-related admissions criteria in a three-year pharmacy program. American Journal of Pharmaceutical Education. https://doi.org/10.5688/ajpe79686.

    Article  Google Scholar 

  • Pearce, K., & Scutter, S. (2010). Podcasting of health sciences lectures: Benefits for students from a non-English speaking background. Australasian Journal of Educational Technology, 26(7), 1028–1041.

    Article  Google Scholar 

  • Ranchal, R., Taber-Doughty, T., Guo, Y., Bain, K., Martin, H., Robinson, J. P., et al. (2013). Using speech recognition for real-time captioning and lecture transcription in the classroom. IEEE Transactions on Learning Technologies, 6(4), 299–311.

    Article  Google Scholar 

  • Ryba, K., McIvor, T., Shakir, M., & Paez, D. (2006). Liberated Learning: Analysis of University students’ perceptions and experiences with continuous automated speech recognition. Journal of Instructional Science and Technology, 9(1), 1–19.

    Google Scholar 

  • Shadiev, R., & Huang, Y. M. (2016). Facilitating cross-cultural understanding with learning activities supported by speech-to-text recognition and computer-aided translation. Computers & Education, 98, 130–141.

    Article  Google Scholar 

  • Shadiev, R., Hwang, W. Y., Chen, N. S., & Huang, Y. M. (2014). Review of Speech-to-Text Recognition technology for enhancing learning. Educational Technology & Society, 17(4), 65–84.

    Google Scholar 

  • Shadiev, R., Hwang, W. Y., Huang, Y. M., & Liu, C. J. (2016). Investigating applications of speech to text recognition for face to face seminar to assist learning of non-native English participants. Technology, Pedagogy and Education, 25(1), 119–134.

    Article  Google Scholar 

  • Shadiev, R., Wu, T. T., & Huang, Y. M. (2017). Enhancing learning performance, attention, and meditation using a speech-to-text recognition application: Evidence from multiple data sources. Interactive Learning Environments, 25(2), 249–261.

    Article  Google Scholar 

  • Shadiev, R., Wu, T. T., Sun, A., & Huang, Y. M. (2018). Applications of speech-to-text recognition and computer-aided translation for enhancing cross-cultural learning: Issues and their solutions. Educational Technology Research and Development, 66(1), 191–214.

    Article  Google Scholar 

  • Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295–312.

    Article  Google Scholar 

  • Sweller, J. (2017). Cognitive load theory and teaching English as a second language to adult learners. Contact Magazine, 43(1), 10–14.

    Google Scholar 

  • Sweller, J., Ayres, P., & Kalyuga, S. (Eds.) (2011a). The redundancy effect. In Cognitive load theory (pp. 141–154). Springer New York.

  • Sweller, J., Ayres, P., & Kalyuga, S. (Eds.) (2011b). Cognitive load theory. New York: Springer.

    Book  Google Scholar 

  • Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186–204.

    Article  Google Scholar 

  • Wald, M., & Bain, K. (2008). Universal access to communication and learning: The role of automatic speech recognition. International Journal Universal Access in the Information Society, 6(4), 435–447.

    Article  Google Scholar 

  • Yeung, A. S. (1999). Cognitive load and learner expertise: Split-attention and redundancy effects in reading comprehension tasks with vocabulary definitions. The Journal of Experimental Education, 67(3), 197–217.

    Article  Google Scholar 

Download references

Funding

There is no funding to report for this study.

Author information

Authors and Affiliations

  1. School of Education Science, Nanjing Normal University, No. 122, Ninghai Road, Nanjing, 210097, Jiangsu Province, China

    Rustam Shadiev

  2. School of Humanities and Social Sciences, Harbin Institute of Technology in Shenzhen, The University Town of Shenzhen, Nanshan District, Shenzhen, 518055, China

    Ai Sun

Authors
  1. Rustam Shadiev
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ai Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ai Sun.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

  1. 1.

    The treatment improves my understanding of a lecture.

  2. 2.

    The treatment increases my productivity during a lecture.

  3. 3.

    The treatment enhances my learning effectiveness during a lecture.

  4. 4.

    The treatment improves my learning performance during a lecture.

  5. 5.

    The treatment helps me accomplish a learning task more quickly.

  6. 6.

    Overall, I found the treatment to be useful during a lecture.

  7. 7.

    I intend to continue using the treatment for learning in the future.

  8. 8.

    I plan to use the treatment for learning often.

  9. 9.

    I will strongly recommend others to use the treatment for learning.

Appendix 2

STR-text

Photography

Hello everyone, today I am going to talk about photography. Do you have a camera? Do you enjoy taking pictures? Daniel and Winnie are taking pictures today, so we are learning about photography.

Anyone can be a photographer. You just need a camera. Daniel has a small camera. Winnie has a big camera.

CAT-text

ФОТОГРАФИЯ

Пpивeт вceм, ceгoдня я coбиpaюcь пoгoвopить o фoтoгpaфии. У тeбя ecть кaмepa? Baм нpaвитcя фoтoгpaфиpoвaть? Дэниeл и Bинни ceгoдня фoтoгpaфиpyютcя, пoэтoмy мы yчимcя фoтoгpaфии.

Любoй мoжeт быть фoтoгpaфoм. Baм пpocтo нyжнa кaмepa. У Дaниэля мaлeнькaя кaмepa. У Bинни бoльшaя кaмepa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shadiev, R., Sun, A. Using texts generated by STR and CAT to facilitate student comprehension of lecture content in a foreign language. J Comput High Educ 32, 561–581 (2020). https://doi.org/10.1007/s12528-019-09246-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12528-019-09246-7

Keywords

Search

Navigation