Detection of Input-Difficult Words by Automatic Speech Recognition for PC Captioning

  • Yoshinori TakeuchiEmail author
  • Daiki Kojima
  • Shoya Sano
  • Shinji Kanamori
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10896)


Hearing-impaired students often need complementary technologies to assist them in understanding college lectures. Several universities already use PC captioning. Captionists sometime input unfamiliar technical terms and proper nouns in a lecture inaccurately. We call these words “input-difficult words (IDWs).” In this research, we evaluate performance-detecting IDWs by using real lectures from our university. We propose a method to automatically extract IDWs from lecture materials. We conducted an experiment to measure performance-detecting IDWs from lectures by changing the interpolation weight of the language model. In this experiment, we used four real lectures. A high F-measure of 0.889 was achieved.


PC captioning Automatic speech recognition Lecture Hearing impaired 


  1. 1.
    Kato, N., Kawano, S., Miyoshi, S., Nishioka, T., Murakami, H., Minagawa, H., Wakatsuki, D., Shirasawa, M., Ishihara, Y., Naito, I.: Subjective evaluation of displaying keywords for speech to text service operators. Trans. Hum. Interface Soci. 9(2), 195–203 (2007). (in Japanese)Google Scholar
  2. 2.
    Akita, Y., Kuwahara, N., Kawahara, T.: Automatic classification of usability of ASR result for real-time captioning of lectures. In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 19–22 (2015)Google Scholar
  3. 3.
    Gaur, Y., Metze, F., Miao, Y., Bigham, J. P.: Using keyword spotting to help humans correct captioning faster. In: 16th Annual Conference of the International Speech Communication Association, pp. 2829–2833 (2015)Google Scholar
  4. 4.
    Ikeda, N., Takeuchi, Y., Matsumoto, T., Kudo, H., Ohnishi, N.: Support system for lecture captioning using keyword detection by automatic speech recognition. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9759, pp. 377–383. Springer, Cham (2016). Scholar
  5. 5.
    Qin, L.: Learning out-of-vocabulary words in automatic speech recognition. Ph. D. thesis, CMU University (2013)Google Scholar
  6. 6.
    Illina, I., Fohr, D.: Out-of-vocabulary word probability estimation using RNN language model. In: Proceedings of the 8th Language & Technology Conference (2017)Google Scholar
  7. 7.
    Mirzaei, M.S., Meshgi, K., Kawahara, T.: Listening difficulty detection to foster second language listening with a partial and synchronized caption system. In: Borthwick, K., Bradley, L., Thouësny, S. (eds) CALL in a Climate of Change: Adapting to Turbulent Global Conditions Short Papers From EUROCALL 2017, pp. 211–216 (2017)CrossRefGoogle Scholar
  8. 8.
    Munteanu, C., Penn, G., Beacker, R.: Web-based language modelling for automatic lecture transcription. In: Proceedings of the 8th Annual Conference of the International Speech Communication Association, No.ThD.P3a-2, pp. 2353–2356 (2007)Google Scholar
  9. 9.
    Kawahara, T., Nemoto, Y., Akita, Y.: Automatic lecture transcription by exploiting presentation slide information for language model adaptation, In: Proceedings of the ICASSP, pp. 4929–4932 (2008). (in Japanese)Google Scholar
  10. 10.
    Ito, A.: Palmkit (2009).
  11. 11.
    Stolcke, A.: SRILM–an extensible language modeling toolkit. In: Proceedings of the ICSLP (2002)Google Scholar
  12. 12.
    Furui, S.: Recent advances in spontaneous speech recognition and understanding. In: Proceedings of the ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, pp. 1–6 (2003)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Yoshinori Takeuchi
    • 1
    Email author
  • Daiki Kojima
    • 1
  • Shoya Sano
    • 1
  • Shinji Kanamori
    • 1
  1. 1.Department of Information Systems, School of InformaticsDaido UniversityNagoyaJapan

Personalised recommendations