Two-Stage Temporal Multimodal Learning for Speaker and Speech Recognition
Temporal information prevails in multimodal sequence data, such as video data and speech signals. In this paper, we propose a two-stage learning to model the temporal information in multimodal sequences. At the first learning stage, static representative features are extracted from each modality at every time step. Then joint representations across various modalities are effectively learned within a joint fusion layer. The second one is to transfer the static features into corresponding dynamical features by jointly learning the temporal information and dependencies between different time steps with a Long Short-Term Memory (LSTM). Compared with previous multimodal methods, the proposed model is efficient in learning temporal joint representations. Evaluated on Big Bang Theory speaker recognition dataset and AVLetters speech recognition dataset, our model proves to outperform other methods.
KeywordsTemporal multimodal learning Speaker recognition Speech recognition
This work is supported by the National Natural Science Foundation of China (Grant No. 61502174, 61402181), the Natural Science Foundation of Guangdong Province (Grant No. S2012010009961, 2015A030313215), the Science and Technology Planning Project of Guangdong Province (Grant No. 2016A040403046), the Guangzhou Science and Technology Planning Project (Grant No. 201704030051, 2014J4100006), the Opening Project of Guangdong Province Key Laboratory of Big Data Analysis and Processing (Grant No. 2017014), and the Fundamental Research Funds for the Central Universities (Grant No. D2153950).
- 1.Amer, M.R., Siddiquie, B., Khan, S., Divakaran, A., Sawhney, H.: Multimodal fusion using dynamic hybrid models. In: IEEE Winter Conference on Applications of Computer Vision, pp. 556–563, March 2014Google Scholar
- 2.Bauml, M., Tapaswi, M., Stiefelhagen, R.: Semi-supervised learning with constraints for person identification in multimedia data. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013Google Scholar
- 6.Hu, D., Li, X., Lu, X.: Temporal multimodal learning in audiovisual speech recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016Google Scholar
- 7.Hu, Y., Ren, J.S., Dai, J., Yuan, C., Xu, L., Wang, W.: Deep multimodal speaker naming. In: Proceedings of the 23rd ACM International Conference on Multimedia, MM 2015, NY, USA, pp. 1107–1110 (2015). doi: 10.1145/2733373.2806293
- 8.Huang, J., Kingsbury, B.: Audio-visual deep learning for noise robust speech recognition. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7596–7599 (2013)Google Scholar
- 9.Jamil, M., Rahman, G.R.S.: Speaker identification using Mel frequency cepstral coefficients (2004)Google Scholar
- 10.Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28–July 2, pp. 689–696 (2011)Google Scholar
- 11.Ren, J., Hu, Y., Tai, Y.W., Wang, C., Xu, L., Sun, W., Yan, Q.: Look, listen and learn – a multimodal LSTM for speaker identification. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 3581–3587. AAAI Press (2016)Google Scholar
- 12.Sohn, K., Shang, W., Lee, H.: Improved multimodal deep learning with variation of information. In: International Conference on Neural Information Processing Systems, pp. 2141–2149 (2014)Google Scholar
- 14.Stiefelhagen, R.: “Knock! knock! who is it?” probabilistic person identification in TV-series. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR 2012, pp. 2658–2665 (2012). http://dl.acm.org/citation.cfm?id=2354409.2354974