Multimedia Tools and Applications

, Volume 78, Issue 10, pp 13149–13168 | Cite as

Semi-supervised dual low-rank feature mapping for multi-label image annotation

  • Xiaoying Wang
  • Songhe FengEmail author
  • Congyan Lang


Automatic image annotation as a typical multi-label learning problem, has gained extensive attention in recent years owing to its application in image semantic understanding and relevant disciplines. Nevertheless, existing annotation methods share the same challenge that labels annotated on the training images are usually incomplete and unclean, while the need for adequate training data is costly and unrealistic. Being aware of this, we propose a dual low-rank regularized multi-label learning model under a graph regularized semi-supervised learning framework, which can effectively capture the label correlations in the learned feature space, and enforce the label matrix be self-recovered in label space as well. To be specific, the proposed approach firstly puts forward a label matrix refinement approach, by introducing a label coefficient matrix to build a linear self-recovery model. Then, graph Laplacian regularization is introduced to make use of a large number of unlabeled images by enforcing the local geometric structure on both labeled and unlabeled images. Lastly, we exploit dual trace norm regularization on both feature mapping matrix and self-recovery coefficient matrix to capture the correlations among different labels in both feature space and label space, and control the model complexity as well. Empirical studies on four real-world image datasets demonstrate the effectiveness and efficiency of the proposed framework.


Automatic image annotation Multi-label learning Semi-supervised learning Self-recovery model Graph Laplacian regularization Dual low-rank regularization 



This work is supported in part by National Natural Science Foundation of China (61472028, 61502026, 61673048), the Fundamental Research Funds for the Central Universities (2017JBZ108), Beijing Natural Science Foundation (4162048) and the Joint Research Fund for The Ministry of Education of China and China Mobile (MCM20160206).


  1. 1.
    Bao B, Liu G, Xu C, Yan S (2012) Inductive robust principal component analysis. IEEE Trans Image Process 21(8):3794–3800MathSciNetzbMATHGoogle Scholar
  2. 2.
    Bao B, Zhu G, Shen J, Yan S (2013) Robust image analysis with sparse representation on quantized visual features. IEEE Trans Image Process 22(3):860–871MathSciNetzbMATHGoogle Scholar
  3. 3.
    Bucak SS, Jin R, Jain A (2011) Multi-label learning with incomplete class assignments. In: IEEE Conference on computer vision and pattern recognition, pp 2801–2808Google Scholar
  4. 4.
    Cai D, Zhang C, He X (2010) Unsupervised feature selection for multi-cluster data. In: International conference on knowledge discovery and data mining, pp 333–342Google Scholar
  5. 5.
    Carneiro G, Chan AB, Moreno PJ, Vasconcelos N (2007) Supervised learning of semantic classes for image annotation and retrieval. IEEE Trans Pattern Anal Mach Intell 29(3):394–410Google Scholar
  6. 6.
    Chen M, Zheng A, Weinberger KQ (2013) Fast image tagging. In: International conference on machine learning, pp 1274–1282Google Scholar
  7. 7.
    Chua TS, Tang J, Hong R, Li H, Luo Z, Zheng Y (2009) Nus-wide: a real-world web image database from national university of singapore. In: ACM International conference on image and video retrieval, pp 1–9Google Scholar
  8. 8.
    Fan J, Shen Y, Yang C, Zhou N (2011) Structured max-margin learning for inter-related classifier training and multilabel image annotation. IEEE Trans Image Process 20(3):837–854MathSciNetzbMATHGoogle Scholar
  9. 9.
    Feng S, Feng Z, Jin R (2015) Learning to rank image tags with limited training examples. IEEE Trans Image Process 24(4):1223–1234MathSciNetzbMATHGoogle Scholar
  10. 10.
    Feng S, Lang C (2017) Graph regularized low-rank feature mapping for multi-label learning with application to image annotation. Multidim Syst Sign Process 26(4):1–22MathSciNetGoogle Scholar
  11. 11.
    Feng Z, Jin R, Jain A (2013) Large-scale image annotation by efficient and robust kernel metric learning. In: IEEE International conference on computer vision, pp 1609–1616Google Scholar
  12. 12.
    Goldberg AB, Zhu X, Recht B, Xu J, Nowak R (2010) Transduction with matrix completion: three birds with one stone. In: International conference on neural information processing systems, pp 757–765Google Scholar
  13. 13.
    Guillaumin M, Mensink T, Verbeek J, Schmid C (2009) Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation. In: IEEE International conference on computer vision, pp 309–316Google Scholar
  14. 14.
    Huang S, Zhou Z (2012) Multi-label learning by exploiting label correlations locally. In: Twenty-sixth AAAI conference on artificial intelligence, pp 949–955Google Scholar
  15. 15.
    Hwang SJ, Grauman K (2012) Learning the relative importance of objects from tagged images for retrieval and cross-modal search. Int J Comput Vis 100(2):134–153MathSciNetGoogle Scholar
  16. 16.
    Jeon J, Lavrenko V, Manmatha R (2003) Automatic image annotation and retrieval using cross-media relevance models. In: International ACM SIGIR conference on research and development in informaion retrieval, pp 119–126Google Scholar
  17. 17.
    Ji S, Ye J (2009) An accelerated gradient method for trace norm minimization. In: International conference on machine learning, pp 457–464Google Scholar
  18. 18.
    Jian L, Li J, Shu K, Liu H (2016) Multi-label informed feature selection. In: International joint conference on artificial intelligence, pp 1627–1633Google Scholar
  19. 19.
    Jing L, Yang L, Yu J, Ng MK (2015) Semi-supervised low-rank mapping learning for multi-label classification. In: IEEE International conference on computer vision and pattern recognition, pp 1483–1491Google Scholar
  20. 20.
    Li B, Xiong W, Wu O, Hu W, Maybank S, Yan S (2015) Horror image recognition based on context-aware multi-instance learning. IEEE Trans Image Process 24(12):5193–5205MathSciNetzbMATHGoogle Scholar
  21. 21.
    Li B, Yuan C, Xiong W, Hu W, Peng H, Ding X, Maybank S (2017) Multi-view multi-instance learning based on joint sparse representation and multi-view dictionary learning. IEEE Trans Pattern Anal Mach Intell 39(12):2554–2560Google Scholar
  22. 22.
    Li J, Wang J (2003) Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans Pattern Anal Mach Intell 25(9):1075–1088Google Scholar
  23. 23.
    Li X, Zhao X, Zhang Z, Wu F, Zhuang Y, Wang J, Li X (2015) Joint multilabel classification with community-aware label graph learning. IEEE Trans Image Process 25(1):484–493MathSciNetzbMATHGoogle Scholar
  24. 24.
    Lin Z, Ding G, Hu M, Wang J (2014) Multi-label classification via feature-aware implicit label space encoding. In: International conference on machine learning, pp 325–333Google Scholar
  25. 25.
    Makadia A, Pavlovic V, Kumar S (2010) Baselines for image annotation. Int J Comput Vis 90(1):88–105Google Scholar
  26. 26.
    Monay F, Gaticaperez D (2004) Plsa-based image auto-annotation: constraining the latent space. In: ACM International conference on multimedia, pp 348–351Google Scholar
  27. 27.
    Nesterov Y (1983) A method of solving a convex programming problem with convergence rate \(o(\frac {1}{k^{2}})\). In: Soviet mathematics doklady, pp 372–376Google Scholar
  28. 28.
    Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2016) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832Google Scholar
  29. 29.
    Putthividhy D, Attias HT, Nagarajan SS (2010) Topic regression multi-modal latent dirichlet allocation for image annotation. In: IEEE International conference on computer vision and pattern recognition, pp 3408–3415Google Scholar
  30. 30.
    Sang J, Fang Q, Xu C (2017) Exploiting social-mobile information for location visualization. ACM Trans Intell Syst Technol 8(3):39Google Scholar
  31. 31.
    Sang J, Xu C, Liu J (2012) User-aware image tag refinement via ternary semantic analysis. IEEE Trans Multimedia 14(3):883–895Google Scholar
  32. 32.
    Toh K-C, Yun S (2009) An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems. Pacific J Optim 6(3):615–640MathSciNetzbMATHGoogle Scholar
  33. 33.
    Wang H, Huang H, Ding C (2009) Image annotation using multi-label correlated green’s function. In: IEEE International conference on computer vision, pp 2029–2034Google Scholar
  34. 34.
    Wang X, Zhang L, Li X, Ma W (2008) Annotating images by mining image search results. IEEE Trans Pattern Anal Mach Intell 30(11):1919–1932Google Scholar
  35. 35.
    Wu L, Jin R, Jain A (2013) Tag completion for image retrieval. IEEE Trans Pattern Anal Mach Intell 35(3):716–727Google Scholar
  36. 36.
    Xu L, Wang Z, Shen Z, Wang Y, Chen E (2014) Learning low-rank label correlations for multi-label classification with missing labels. In: IEEE International conference on data mining, pp 1067–1072Google Scholar
  37. 37.
    Xu M, Jin R, Zhou Z (2013) Speedup matrix completion with side information: application to multi-label learning. In: Advances in neural information processing systems, pp 2301–2309Google Scholar
  38. 38.
    Yang Y, Wu F, Nie F, Shen H, Zhuang Y, Hauptmann AG (2012) Web and personal image annotation by mining label correlation with relaxed visual graph embedding. IEEE Trans Image Process 21(3):1339–1351MathSciNetzbMATHGoogle Scholar
  39. 39.
    Yuan Z, Sang J, Xu C, Yan L (2014) A unified framework of latent feature learning in social media. IEEE Trans Multimedia 16(6):1624–1635Google Scholar
  40. 40.
    Zhang M, Zhou Z (2007) Ml-knn: a lazy learning approach to multi-label learning. Pattern Recogn 40(7):2038–2048zbMATHGoogle Scholar
  41. 41.
    Zhang ML (2011) Lift: multi-label learning with label-specific features. In: International joint conference on artificial intelligence, pp 1609–1614Google Scholar
  42. 42.
    Zhao F, Guo Y (2015) Semi-supervised multi-label learning with incomplete labels. In: International joint conference on artificial intelligence, pp 4062–4068Google Scholar
  43. 43.
    Zhao F, Guo Y (2016) Improving top-n recommendation with heterogeneous loss. In: International joint conference on artificial intelligence, pp 2378–2384Google Scholar
  44. 44.
    Zhao F, Xiao M, Guo Y (2016) Predictive collaborative filtering with side information. In: International joint conference on artificial intelligence, pp 2385–2390Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Beijing Key Laboratory of Traffic Data Analysis and MiningBeijing Jiaotong UniversityBeijingChina
  2. 2.School of Computer and Information TechnologyBeijing Jiaotong UniversityBeijingChina

Personalised recommendations