Abstract
Nowadays massive amount of images and texts has been emerging on the Internet, arousing the demand of effective cross-modal retrieval. To eliminate the heterogeneity between the modalities of images and texts, the existing subspace learning methods try to learn a common latent subspace under which cross-modal matching can be performed. However, these methods usually require fully paired samples (images with corresponding texts) and also ignore the class label information along with the paired samples. Indeed, the class label information can reduce the semantic gap between different modalities and explicitly guide the subspace learning procedure. In addition, the large quantities of unpaired samples (images or texts) may provide useful side information to enrich the representations from learned subspace. Thus, in this work we propose a novel model for cross-modal retrieval problem. It consists of (1) a semi-supervised coupled dictionary learning step to generate homogeneously sparse representations for different modalities based on both paired and unpaired samples; (2) a coupled feature mapping step to project the sparse representations of different modalities into a common subspace defined by class label information to perform cross-modal matching. We conducted extensive experiments on three benchmark datasets with fully paired setting, and a large-scale real-world web dataset with partially paired setting. The results well demonstrate the effectiveness and reasonableness of the proposed method in performing cross-modal retrieval tasks.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
L. Chen, D. Xu, I.W. Tsang, X. Li, Spectral embedded hashing for scalable image retrieval. IEEE Trans. Cybern. 44(7), 1180–1190 (2014)
F. Shen, C. Shen, Q. Shi, A. Van Den Hengel, Z. Tang, Inductive hashing on manifolds, in IEEE Conference on Computer Vision and Pattern Recognition (2013)
J. Song, Y. Yang, X. Li, Z. Huang, Y. Yang, Robust hashing with local models for approximate similarity search. IEEE Trans. Cybern. 44(7), 1225–1236 (2014)
Y. Yang, Z.-J. Zha, Y. Gao, X. Zhu, T.-S. Chua, Exploiting web images for semantic video indexing via robust sample-specific loss. IEEE Trans. Multimedia 16, 1677–1689 (2014)
N. Srivastava, R. Salakhutdinov, Multimodal learning with deep boltzmann machines. JMLR 15, 2949–2980 (2014)
Y. Yang, Y. Yang, H.T. Shen, Effective transfer tagging from image to video. ACM Trans. Multimedia Comput. Commun. Appl. 9(2), 1–20 (2013)
Y. Yang, Z.-J. Zha, Y. Gao, X. Zhu, T.-S. Chua, Exploiting web images for semantic video indexing via robust sample-specific loss. TMM 17(2), 246–256 (2015)
Y. Zhen, Y. Gao, D. Yeung, H. Zha, X. Li, Spectral multimodal hashing and its application to multimedia retrieval. IEEE Trans. Cybern. 46(1), 27–38 (2016)
V. Ordonez, G. Kulkarni, T.L. Berg, Im2text: describing images using 1 million captioned photographs. NIPS (2011)
J. Verbeek, M. Guillaumin, T. Mensink, C. Schmid, Image annotation with tagprop on the mirflickr set, in ACM MIR, MIR ’10, pp. 537–546 (2010)
Y. Verma, C. Jawahar, Im2text and text2im: associating images and texts for cross-modal retrieval, in BMVC (2014)
L. Ballan, T. Uricchio, L. Seidenari, A. Bimbo, A cross media model for automatic image annotation. ICMR (2014)
M. Guillaumin, T. Mensink, J. Verbeek, C. Schmid, Tagprop: discriminative metric learning in nearest neighbor models for image auto-annotation. CVPR 309–316 (2009)
A. Makadia, V. Pavlovic, S. Kumar, A new baseline for image annotation, in ECCV (2008)
D.R. Hardoon, S.R. Szedmak, J.R. Shawe-taylor, Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16(12), 2639–2664 (2004b)
Y. Gong, Q. Ke, M. Isard, S. Lazebnik, A multi-view embedding space for modeling internet images, tags, and their semantics. IJCV 106, 210–233 (2014)
N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. Lanckriet, R. Levy, N. Vasconcelos, A new approach to cross-modal multimedia retrieval, in ACM MM (2010)
H. Zhang, Y. Zhuang, F. Wu, Cross-modal correlation learning for clustering on image-audio dataset, in Proceedings of the 15th ACM International Conference on Multimedia, pp. 273–276 (2007)
D. Li, N. Dimitrova, M. Li, I.K. Sethi, Multimedia content processing through cross-modal association, in Proceedings of the Eleventh ACM International Conference on Multimedia, pp. 604–611 (2003)
S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, R. Harshman, Indexing by latent semantic analysis. J. Amer. Soc. Inf. Sci. 41(6), 391–407 (1990)
R. Rosipal, N. Krämer, Overview and recent advances in partial least squares, in Proceedings of SLSFS, pp. 34–51 (2006)
A. Sharma, D.W. Jacobs, Bypassing synthesis: pls for face recognition with pose, low-resolution and sketch, in Proceedings of CVPR, pp. 593–600 (2011)
J.B. Tenenbaum, W.T. Freeman, Separating style and content with bilinear models. Neural Comput. 12(6), 1247–1283 (2000)
C. Kang, S. Xiang, S. Liao, C. Xu, C. Pan, Learning consistent feature representation for cross-modal multimedia retrieval. IEEE Trans. Multimedia 17(3), 370–381 (2015)
X. Gao, N. Wang, D. Tao, X. Li, Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans. Circuits Syst. Video Technol. 22(8), 1213–1226 (2012)
Z. Cui, W. Li, D. Xu, S. Shan, X. Chen, X. Li, Flowing on riemannian manifold: domain adaptation by shifting covariance. IEEE Trans. Cybern. 44(12), 2264–2273 (2014)
D. Huang, Y. Wang, Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition, in ICCV, pp. 2496–2503 (2013)
S. Shekhar, V.M. Patel, H.V. Nguyen, R. Chellappa, Coupled projections for adaptation of dictionaries. IEEE Trans. Image Process. 24(10), 2941–2954 (2015)
S. Wang, L. Zhang, Y. Liang, Q. Pan, Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. CVPR 2216–2223 (2012)
J. Wang, S. Kumar, S.-F. Chang, Semi-supervised hashing for scalable image retrieval. CVPR 3424–3431 (2010)
A. Sharma, A. Kumar, H. Daume, D.W. Jacobs, Generalized multiview analysis: a discriminative latent space. CVPR 2160–2167 (2012)
M. Kan, S. Shan, H. Zhang, S. Lao, X. Chen, Multi-view discriminant analysis, in Computer Vision ECCV 2012, vol 7572, pp. 808–821 (2012)
C. Deng, X. Tang, J. Yan, W. Liu, X. Gao, Discriminative dictionary learning with common label alignment for cross-modal retrieval. IEEE Trans. Multimedia 18(2), 208–218 (2016)
X. Xu, A. Shimada, R. Taniguchi, L. He, Coupled dictionary learning and feature mapping for cross-modal retrieval, in IEEE International Conference on Multimedia and Expo, ICME, pp. 1–6 (2015)
D. Hardoon, S. Szedmak, J. Shawe-taylor, Canonical correlation analysis: An overview with application to learning methods. Neural Comput. 16, 2639–2664 (2004a)
H. Hotelling, Relations between two sets of variates. Biometrika (1936)
J.C. Pereira, E. Coviello, G. Doyle, N. Rasiwasia, G.R.G. Lanckriet, R. Levy, N. Vasconcelos, On the role of correlation and abstraction in cross-modal multimedia retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 521–535 (2014)
S. Hwang, K. Grauman, Learning the relative importance of objects from tagged images for retrieval and cross-modal search. IJCV 100, 134–153 (2012)
L. Zhang, Y. Zhao, Z. Zhu, S. Wei, X. Wu, Mining semantically consistent patterns for cross-view data. IEEE Trans. Knowl. Data Eng. 26(11), 2745–2758 (2014)
Y. Chen, L. Wang, W. Wang, Z. Zhang, Continuum regression for cross-modal multimedia retrieval. ICIP (2012)
L. Sun, S. Ji, J. Ye, A least squares formulation for canonical correlation analysis. ICML 1024–1031 (2008)
K. Wang, R. He, W. Wang, L. Wang, T. Tan, Learning coupled feature spaces for cross-modal matching, in ICCV, pp. 2088–2095 (2013)
J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A.Y. Ng, Multimodal deep learning. ICML 689–696 (2011)
N. Srivastava, R. Salakhutdinov, Multimodal learning with deep boltzmann machines, in NIPS, pp. 1–9 (2012)
Y. Peng, X. Zhai, Y. Zhao, X. Huang, Semi-supervised cross-media feature learning with unified patch graph regularization. IEEE Trans. Circuits Syst. Video Technol. 26(3), 583–596 (2016)
M.J. Huiskes, M.S. Lew, The mir flickr retrieval evaluation, in ACM MIR, pp. 39–43 (2008)
D.M. Blei, A.Y. Ng, M.I. Jordan, Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
M. Chen, A. Zheng, K. Weinberger, Fast image tagging. ICML 1274–1282 (2013)
M.J. Huiskes, B. Thomee, M.S. Lew, New trends and ideas in visual concept detection: The mir flickr retrieval evaluation initiative, in ACM MIR, pp. 527–536 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Xu, X. (2017). Cross-Modal Learning with Images, Texts and Their Semantics. In: Lu, H., Li, Y. (eds) Artificial Intelligence and Computer Vision. Studies in Computational Intelligence, vol 672 . Springer, Cham. https://doi.org/10.1007/978-3-319-46245-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-46245-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46244-8
Online ISBN: 978-3-319-46245-5
eBook Packages: EngineeringEngineering (R0)