Abstract
This paper proposed a novel sequence image representation method called concatenated frame image (CFI), two types of data augmentation methods for CFI, and a framework of CFI-based convolutional neural network (CNN) for visual speech recognition (VSR) task. CFI is a simple, however, it contains spatial-temporal information of a whole image sequence. The proposed method was evaluated with a public database OuluVS2. This is a multi-view audio-visual dataset recorded from 52 subjects. The speaker independent recognition tasks were carried out with various experimental conditions. As the result, the proposed method obtained high recognition accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Dupont, S., Luettin, J.: Audio-visual speech modeling for continuous speech recognition. IEEE Trans. Multimed. 2, 141–151 (2000)
Zhou, Z., Zhao, G., Hong, X., Pietikainen, M.: A review of recent advances in visual speech decoding. Image Vis. Comput. 32, 590–605 (2014)
Bregler, C., Konig, Y.: “Eigenlips” for robust speech recognition. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1994), pp. 669–672 (1994)
Lucey, P.J., Potamianos, G., Sridharan, S.: Patch-based analysis of visual speech from multiple views. In: Proceedings of International Conference on Auditory-Visual Speech Processing (AVSP 2008), pp. 69–73 (2008)
Shiraishi, J., Saitoh, T.: Optical flow based lip reading using non rectangular ROI and head motion reduction. In: 11th IEEE International Conference on Automatic Face and Gesture Recognition (FG2015) (2015)
Matthews, I., Cootes, T.F., Bangham, J.A., Cox, S., Harvey, R.: Extraction of visual features for lipreading. IEEE Trans. Pattern Anal. Mach. Intell. 24, 198–213 (2002)
Shin, J., Lee, J., Kim, D.: Real-time lip reading system for isolated Korean word recognition. Pattern Recogn. 44, 559–571 (2011)
Saitoh, T.: Efficient face model for lip reading. In: International Conference on Auditory-Visual Speech Processing (AVSP), pp. 227–232 (2013)
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: 28th International Conference on Machine Learning, pp. 689–696 (2011)
Hu, D., Li, X., Lu, X.: Temporal multimodal learning in audiovisual speech recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3574–3582 (2016)
Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H.G., Ogata, T.: Lipreading using convolutional neural network. In: INTERSPEECH, pp. 1149–1153 (2014)
Amer, M.R., Siddiquie, B., Khan, S., Divakaran, A., Sawhney, H.: Multimodal fusion using dynamic hybrid models. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 556–563 (2014)
Takashima, Y., Kakihara, Y., Aihara, R., Takiguchi, T., Araki, Y., Mitani, N., Omori, K., Nakazono, K.: Audio-visual speech recognition using convolutive bottleneck networks for a person with severe hearing loss. IPSJ Trans. Comput. Vis. Appl. 7, 64–68 (2015)
Anina, I., Zhou, Z., Zhao, G., Pietikainen, M.: OuluVS2: a multi-view audiovisual database for non-rigid mouth motion analysis. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG) (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)
Lin, M., Chen, Q., Yan, S.: Network in network. In: International Conference on Learning Representations (ICLR) (2014)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.: ImageNet classification with deep convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Zhao, G., Barnard, M., Pietikainen, M.: Lipreading with local spatiotemporal descriptors. IEEE Trans. Multimed. 11, 1254–1265 (2009)
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep learning for human action recognition. In: International Workshop on Human Behavior Understanding (HBU 2011) (2011)
Acknowledgement
This work was supported by JSPS KAKENHI Grant Number 15K12601 and 16H03211.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Saitoh, T., Zhou, Z., Zhao, G., Pietikäinen, M. (2017). Concatenated Frame Image Based CNN for Visual Speech Recognition. In: Chen, CS., Lu, J., Ma, KK. (eds) Computer Vision – ACCV 2016 Workshops. ACCV 2016. Lecture Notes in Computer Science(), vol 10117. Springer, Cham. https://doi.org/10.1007/978-3-319-54427-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-54427-4_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54426-7
Online ISBN: 978-3-319-54427-4
eBook Packages: Computer ScienceComputer Science (R0)