Random Forest for Image Annotation

  • Hao Fu
  • Qian Zhang
  • Guoping Qiu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7577)


In this paper, we present a novel method for image annotation and made three contributions. Firstly, we propose to use the tags contained in the training images as the supervising information to guide the generation of random trees, thus enabling the retrieved nearest neighbor images not only visually alike but also semantically related. Secondly, different from conventional decision tree methods, which fuse the information contained at each leaf node individually, our method treats the random forest as a whole, and introduces the new concepts of semantic nearest neighbors (SNN) and semantic similarity measure (SSM). Thirdly, we annotate an image from the tags of its SNN based on SSM and have developed a novel learning to rank algorithm to systematically assign the optimal tags to the image. The new technique is intrinsically scalable and we will present experimental results to demonstrate that it is competitive to state of the art methods.


Random Forest Image Annotation Semantic Nearest Neighbor 


  1. 1.
    Boiman, O., Shechtman, E., Irani, M.: In defense of Nearest-Neighbor based image classification. In: CVPR (June 2008)Google Scholar
  2. 2.
    Hays, J., Efros, A.A.: Scene completion using millions of photographs. In: SIGGRAPH, vol. 26 (July 2007)Google Scholar
  3. 3.
    Tighe, J., Lazebnik, S.: SuperParsing: Scalable Nonparametric Image Parsing with Superpixels. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 352–365. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  4. 4.
    Guillaumin, M., Mensink, T., Verbeek, J., Schmid, C.: TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation. In: ICCV (September 2009)Google Scholar
  5. 5.
    Makadia, A., Pavlovic, V., Kumar, S.: A New Baseline for Image Annotation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 316–329. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Sivic, J., Zisserman, A.: Video Google: A text retrieval approach to object matching in videos. In: ICCV, vol. 2 (2003)Google Scholar
  7. 7.
    Wang, J., Kumar, S., Chang, S.F.: Semi-Supervised Hashing for Scalable Image Retrieval. In: CVPR (2010)Google Scholar
  8. 8.
    Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.: Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part IV. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Carneiro, G., Vasconcelos, N.: Formulating Semantic Image Annotation as a Supervised Learning Problem. In: CVPR (2005)Google Scholar
  10. 10.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: Towards removing the curse of dimensionality. In: Symposium on Theory of Computing (1998)Google Scholar
  11. 11.
    Kulis, B., Grauman, K.: Kernelized locality-sensitive hashing for scalable image search. In: ICCV (September 2009)Google Scholar
  12. 12.
    Weiss, Y., Torralba, A., Fergus, R.: Spectral Hashing. In: NIPS, vol. (1) (2008)Google Scholar
  13. 13.
    Jain, P., Kulis, B., Grauman, K.: Fast Image Search for Learned Metrics. In: CVPR (June 2008)Google Scholar
  14. 14.
    Jia, Y., Wang, J., Zeng, G., Zha, H., Hua, X.S.: Optimizing kd-trees for scalable visual descriptor indexing. In: CVPR (2010)Google Scholar
  15. 15.
    Kumar, N., Zhang, L., Nayar, S.: What Is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images? In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 364–378. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Muja, M., Lowe, D.G.: Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration. In: VISAPP (2009)Google Scholar
  17. 17.
    Uijlings, J., Smeulders, A., Scha, R.: Real-time Bag of Words, Approximately. In: CIVR (2009)Google Scholar
  18. 18.
    Moosmann, F., Triggs, B., Jurie, F.: Fast discriminative visual codebooks using randomized clustering forests. In: NIPS (2006)Google Scholar
  19. 19.
    Fukui, M., Kato, N., Qi, W.: Multi-Class Labeling Improved by Random Forest for Automatic Image Annotation. In: IAPR Conference on Machine Vision Applications, pp. 202–205 (2011)Google Scholar
  20. 20.
    Fu, H., Qiu, G., He, H.: Feature Combination beyond Basic Arithmetics. In: British Machine Vision Conference (BMVC). BMVA (2011)Google Scholar
  21. 21.
    Bosch, A., Zisserman, A., Munoz, X.: Image Classification using Random Forests and Ferns. In: ICCV (October 2007)Google Scholar
  22. 22.
    Yao, B., Khosla, A., Fei-Fei, L.: Combining Randomization and Discrimination for Fine-Grained Image Categorization. In: CVPR (2011)Google Scholar
  23. 23.
    Yu, G., Yuan, J., Liu, Z.: Unsupervised Random Forest Indexing for Fast Action Search. In: CVPR (2011)Google Scholar
  24. 24.
    Schölkopf, B., Smola, A., Müller, K.R.: Kernel Principal Component Analysis. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 583–588. Springer, Heidelberg (1997)Google Scholar
  25. 25.
    Zhang, K., Tsang, I.W., Kwok, J.T.: Improved Nystrom Low-Rank Approximation and Error Analysis. In: ICML (2008)Google Scholar
  26. 26.
    Criminisi, A., Shotton, J., Konukoglu, E.: Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Foundations and Trends in Computer Graphics and Vision 7(2-3), 81–227 (2012)CrossRefGoogle Scholar
  27. 27.
    Hu, J., Lam, K.M., Qiu, G.: A Hierarchical Algorithm for Image Multi-labeling. In: ICIP (2010)Google Scholar
  28. 28.
    Joachims, T.: Training Linear SVMs in Linear Time. In: ACM KDD (2006)Google Scholar
  29. 29.
    Escalante, H.J., Hernández, C.A., Gonzalez, J.A.: The segmented and annotated IAPR TC-12 benchmark. Computer Vision and Image Understanding (April 2010)Google Scholar
  30. 30.
    Feng, S., Manmatha, R., Lavrenko, V.: Multiple Bernoulli Relevance Models for Image and Video Annotation. In: CVPR (2004)Google Scholar
  31. 31.
    Wang, C., Yan, S., Zhang, L., Zhang, H.J.: Multi-label sparse coding for automatic image annotation. In: CVPR (June 2009)Google Scholar
  32. 32.
    Zhou, N., Cheung, W., Qiu, G., Xue, X.: A Hybrid Probabilistic Model for Unified Collaborative and Content-Based Image Tagging. IEEE TPAMI 33, 1281–1294 (2011)CrossRefGoogle Scholar
  33. 33.
    Liu, D., Yan, S., Rui, Y., Zhang, H.J.: Unified Tag Analysis With Multi-Edge Graph. In: ACM MM (2010)Google Scholar
  34. 34.
    Zhang, S., Huang, J., Huang, Y., Yu, Y., Li, H., Metaxas, D.: Automatic Image Annotation Using Group Sparsity. In: CVPR (2010)Google Scholar
  35. 35.
    Fu, H., Qiu, G.: Fast Semantic Image Retrieval Based on Random Forest. In: ACM MM (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Hao Fu
    • 1
  • Qian Zhang
    • 1
  • Guoping Qiu
    • 1
  1. 1.School of Computer ScienceUniversity of NottinghamNottinghamUK

Personalised recommendations