Skip to main content
SpringerLink
Log in

Lip-Reading Using Pixel-Based and Geometry-Based Features for Multimodal Human–Robot Interfaces

  • Conference paper
  • First Online:
Book cover Proceedings of 14th International Conference on Electromechanics and Robotics “Zavalishin's Readings”

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 154))

Abstract

Automatic lip-reading (ALR) is a challenging task and a significant amount of research has been devoted to this topic in recent years. However, continuous Russian speech recognition still remains a not well-investigated area. In this paper, we present the results of Russian visual speech recognition (VSR) system using pixel-based and advanced geometry-based features. A HAVRUS video database, comprising of 4000 utterances of continuous Russian speech, collected from 20 speakers, is used in this study. Pixel-based features (principal component analysis-based or PCA) and geometry-based features (active appearance model-based or AAM) were used for the feature representation, and a Gaussian mixture hidden Markov models (HMM) were used for classification. Our evaluation experiments show a significant improvement (up to 9%) in recognition accuracy by using proposed geometry-based features when compared to pixel-based PCA features. The combined VSR is planned for future studies to augment the performance of audio-based automatic speech recognition systems in human–robot interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Karpov, A.: Dr. Sc. Tech. (SPIIRAS, Computer Science), thesis Audio-visual speech interfaces in assistive information technologies (2013)

    Google Scholar 

  2. McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976)

    Article  Google Scholar 

  3. Katsaggelos, K., Bahaadini, S., Molina, R.: Audiovisual fusion: challenges and new approaches. Proc. IEEE 103(9), 1635–1653 (2015)

    Article  Google Scholar 

  4. Akhtiamov, O., Sidorov, M., Karpov, A., Minker, W.: Speech and text analysis for multimodal address detection in human-human-computer interaction. Proc. Interspeech 2017, 2521–2525 (2017)

    Article  Google Scholar 

  5. Ivanko, D., Fedotov, D., Karpov, A.: Accuracy increase for automatic visual Russian speech recognition. Sci. Tech. J. Inf. Technol., Me-Chanics Opt. 18(3), 346–349 (2018)

    Google Scholar 

  6. Ivanko, D., et al.: Multimodal speech recognition: increasing accuracy using high speed video data. J. Multimodal User Interfaces 12(4), 319–328 (2018)

    Article  Google Scholar 

  7. Zhou, Z., Zhao, G., Hong, X., Pietikainen, M.: A Review of Recent Advances in Visual Speech Decoding. Image Vis. Comput. 32, 590–605 (2014)

    Article  Google Scholar 

  8. Hassanat, A.: Visual words for automatic lip-reading. Ph.D. thesis, Department of Applied Computing, University of Buckingham, UK (2009)

    Google Scholar 

  9. Verkhodanova, V., et al.: HAVRUS corpus: high-speed recordings of audio-visual Russian speech. In: Speech and Computer. SPECOM 2016. Lecture Notes in Computer Science 9811, pp. 338–345 (2016)

    Chapter  Google Scholar 

  10. Zhou, Z., Hong, X., Zhao, G., Pietikainen, M.: A compact representation of visual speech data using latent variables. Proc. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 181–187 (2014)

    Google Scholar 

  11. Zhao, G., Barnard, M., Pietikainen, M.: Lipreading with local spatiotemporal descriptors. Proc. IEEE Trans. Multimed. 11(7), 1254–1265 (2009)

    Article  Google Scholar 

  12. Lan, Y., Theobald, B., Harvey, R.: View independent computer lip-reading. In: Proceedings of International Conference Multimedia Expo (ICME), pp. 432–437 (2012)

    Google Scholar 

  13. Hong, X., Yao, H., Wan, Y., Chen, R.: A PCA Based visual DCT feature extraction method for lip-reading. In: Proc. Int. Conf. Intell. Inf. Hiding Multimed., Signal Process, pp. 321–326 (2006)

    Google Scholar 

  14. Cetingul, H., Yemez, Y., Erzin, E., Tekalp, A.: Discriminative analysis of lip motion features for speaker identification and speech-reading. Proc. IEEE Trans. Image Process. 15(10), 2879–2891 (2006)

    Article  Google Scholar 

  15. Ivanko, D., Ryumin, D., Axyonov, A., Karpov, A., Zelezny, M.: Designing advanced geometric features for automatic Russian visual speech recognition. In: International Conference on Speech and Computer. SPECOM 2018. LNAI 11096, pp. 245–254 (2018)

    Chapter  Google Scholar 

  16. Estellers, V., Thiran, J.-P.: Multi-pose lipreading and audio-visual speech recognition. EURASIP J. Adv. Signal Process. 51. https://doi.org/10.1186/1687-6180-2012-51 (2012)

  17. Siwar, J., Ben, Y. B., Arnaud, M.: Belief hidden markov model for speech recognition. In: 5th International Conference on Modeling, Simulation and Applied Optimization (ICMSAO) (2013)

    Google Scholar 

  18. Sak, H., Senior, A. W., Rao, K., Beaufays, F.: Fast and accurate recurrent neural network. Acoust. Model. Speech Recognit. CoRR, vol. abs/1507.06947 (2015)

    Google Scholar 

  19. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  20. Hoang, L.T., Nhat, V.T.: Face alignment using active shape model and support vector machine. Int. J. Biom. Bioinform. 4(6), 224–234 (2011)

    Google Scholar 

  21. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2008)

    Article  Google Scholar 

  22. Lan, Y., et al.: Improving visual features for lip reading. AVSP (2010)

    Google Scholar 

  23. Vahid, K., Josephine, S.: One millisecond face alignment with an ensemble of regression trees. CVPR (2014)

    Google Scholar 

  24. Hidden Markov Model Toolkit (HTK). Available at: http://htk.eng.cam.ac.uk/. Accessed 15 Feb 2019

  25. Karpov, A.: An automatic multimodal speech recognition system with audio and video information. Autom. Remote. Control. 75(12), 2190–2200 (2014)

    Article  MathSciNet  Google Scholar 

  26. Ivanko, D., et al.: Using a high-speed video Camera for robust audio-visual speech recognition in acoustically noisy conditions. SPECOM 2017, LNAI 10458, pp. 757–766 (2017)

    Google Scholar 

Download references

Acknowledgements

This research is supported by the Russian Foundation for Basic Research (project No. 18-37-00306 and project No. 18-07-01216) and by the Government of Russian Federation (Grant 08-08).

Author information

Authors and Affiliations

  1. St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

    Denis Ivanko, Dmitry Ryumin, Irina Kipyatkova, Alexandr Axyonov & Alexey Karpov

  2. ITMO University, St. Petersburg, Russia

    Denis Ivanko, Alexandr Axyonov & Alexey Karpov

  3. St. Petersburg State University of Aerospace Instrumentation (SUAI), St. Petersburg, Russia

    Irina Kipyatkova

Authors
  1. Denis Ivanko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry Ryumin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Irina Kipyatkova
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexandr Axyonov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexey Karpov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Ivanko .

Editor information

Editors and Affiliations

  1. St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

    Andrey Ronzhin

  2. Saint Petersburg State University of Aerospace Instrumentation, St. Petersburg, Russia

    Vladislav Shishlakov

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ivanko, D., Ryumin, D., Kipyatkova, I., Axyonov, A., Karpov, A. (2020). Lip-Reading Using Pixel-Based and Geometry-Based Features for Multimodal Human–Robot Interfaces. In: Ronzhin, A., Shishlakov, V. (eds) Proceedings of 14th International Conference on Electromechanics and Robotics “Zavalishin's Readings”. Smart Innovation, Systems and Technologies, vol 154. Springer, Singapore. https://doi.org/10.1007/978-981-13-9267-2_39

Download citation

Publish with us

Policies and ethics

Search

Navigation