Learning Joint Multimodal Representation Based on Multi-fusion Deep Neural Networks

  • Zepeng GuEmail author
  • Bo Lang
  • Tongyu Yue
  • Lei Huang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10635)


Recently, learning joint representation of multimodal data has received more and more attentions. Multimodal features are concept-level compositive features which are more effective than those single-modality features. Most existing methods only mine interactions between modalities on the top of their networks for one time to learn multi-modal representation. In this paper, we propose a multi-fusion deep learning framework which learns multimodal features richer in semantic. The framework sets multiple fusing points in different level of feature spaces, and then integrates and passes the fusing information step by step from the low level to higher levels. Moreover, we propose a multi-channel decoding network with alternate fine-tuning strategy to fully mine the modality–specific information and cross-modality correlations. We are also the first to introduce deep learning features into multimodal deep learning, alleviating the semantic and statistical property differences between modalities to learn better features. Extensive experiments on real-world datasets demonstrate that, our proposed method achieves superior performance compared with the state-of-the-art methods.


Multimodal Deep learning Multi-fusion Semantic integration 



This work was supported in part by the grants from the National Natural Science Foundation of China (Grant No: 61370125, the foundation of the State Key Lab of Software Development Environment (Grant No: SKLSDE-2017ZX-03).


  1. 1.
    Blei, D.M., Jordan, M.I: Modeling annotated data. In: International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 127–134. ACM (2003)Google Scholar
  2. 2.
    Xing, E.P., Yan, R., Hauptmann, A.G.: Mining associated text and images with dual-wing harmoniums. In: Computer Science (2005)Google Scholar
  3. 3.
    Rasiwasia, N., Pereira, J.C., Coviello, E., Doyle, G., Lanckriet, G.R.G., Levy, R.: A new approach to cross-modal multimedia retrieval. In: International Conference on Multimedia, pp. 251–260. ACM (2010)Google Scholar
  4. 4.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: ICML 2011, pp. 689–696. DBLP, Washington (2011)Google Scholar
  5. 5.
    Srivastava, N., Salakhutdinov, R.: Learning representations for multimodal data with deep belief nets. In: International Conference on Machine Learning Workshop (2012)Google Scholar
  6. 6.
    Srivastava, N., Salakhutdinov, R., Hinton, G.: Modeling documents with deep boltzmann machines. In: Uncertainty in Artificial Intelligence, Seattle (2013)Google Scholar
  7. 7.
    Fangxiang, F., Xiaojie, W., Ruifan, L.: Cross-modal retrieval with correspondence Auto-encoder. In: Multimedia 2014, Orlando (2014)Google Scholar
  8. 8.
    Bosch, A., Zisserman, A., Munoz, X.: Image classification using random forests and ferns. In: International Conference on Computer Vision, pp. 1–8. IEEE (2007)Google Scholar
  9. 9.
    Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from National University of Singapore. In: ACM International Conference on Image and Video Retrieval, p. 48. ACM (2009)Google Scholar
  10. 10.
    Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using Amazon’s Mechanical Turk. NAACL Hlt 2010, 139–147 (2010)Google Scholar
  11. 11.
    Raiko, T., Valpola, H., Lecun, Y.: Deep learning made easier by linear transformations. In: Conference on AI and Statistics, Vol. 22, pp. 924–932 (2011)Google Scholar
  12. 12.
    Kaiming, H.: Deep Residual Learning for Image Recognition (2015)Google Scholar
  13. 13.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. ICONIP 25, 1097–1105 (2012)Google Scholar
  14. 14.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. Adv. Neural. Inf. Process. Syst. 26, 3111–3119 (2013)Google Scholar
  15. 15.
    Feng, F., Wang, X., Li, R.: Cross-modal retrieval with correspondence Autoencoder. In: ACM (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.State Key Lab of Software Development Environment, School of Computer Science and EngineeringBeihang UniversityBeijingChina

Personalised recommendations