Interestingness Prediction by Robust Learning to Rank

  • Yanwei Fu
  • Timothy M. Hospedales
  • Tao Xiang
  • Shaogang Gong
  • Yuan Yao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)


The problem of predicting image or video interestingness from their low-level feature representations has received increasing interest. As a highly subjective visual attribute, annotating the interestingness value of training data for learning a prediction model is challenging. To make the annotation less subjective and more reliable, recent studies employ crowdsourcing tools to collect pairwise comparisons – relying on majority voting to prune the annotation outliers/errors. In this paper, we propose a more principled way to identify annotation outliers by formulating the interestingness prediction task as a unified robust learning to rank problem, tackling both the outlier detection and interestingness prediction tasks jointly. Extensive experiments on both image and video interestingness benchmark datasets demonstrate that our new approach significantly outperforms state-of-the-art alternatives.


Majority Vote Outlier Detection Interestingness Prediction Robust Learn Regularisation Path 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Cao, Z., Qin, T., Liu, T.Y., Tsai, M.F., Li, H.: Learning to rank: From pairwise approach to listwise approach. In: ICML (2007)Google Scholar
  2. 2.
    Carvalho, V.R., Elsas, J.L., Cohen, W.W., Carbonell, J.G.: A meta-learning approach for robust rank learning. In: SIGIR 2008 LR4IR - Workshop on Learning to Rank for Information Retrieval (2008)Google Scholar
  3. 3.
    Chapelle, O., Keerthi, S.S.: Efficient algorithms for ranking with svms. Inf. Retr. (2010)Google Scholar
  4. 4.
    Chen, A.: adn R. Pangrazi, P.D.: An examination of situational interest adn its sources. Brit. J. of Edu. Psychology (20001)Google Scholar
  5. 5.
    Chen, K., Wu, C., Chang, Y., Lei, C.: Crowdsourceable QoE evalutation framework for multimedia content. In: ACM MM (2009)Google Scholar
  6. 6.
    Chen, X., Bennett, P.N.: Pairwise ranking aggregation in a crowdsourced setting. In: ACM International Conference on Web Search and Data Mining (2013)Google Scholar
  7. 7.
    Dhar, S., Ordonez, V., Berg, T.L.: High level describable attributes for predicting aesthetics and interestingness. In: CVPR (2011)Google Scholar
  8. 8.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annals of Statistics (2004)Google Scholar
  9. 9.
    Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. JASA (2001)Google Scholar
  10. 10.
    Fan, J., Tang, R., Shi, X.: Partial consistency with sparse incidental parameters. arXiv:1210.6950 (2012)Google Scholar
  11. 11.
    Fu, Y., Hospedales, T.M., Xiang, T., Gong, S.: Attribute learning for understanding unstructured social activity. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 530–543. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Fu, Y., Hospedales, T.M., Xiang, T., Fu, Z., Gong, S.: Transductive multi-view embedding for zero-shot recognition and annotation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part II. LNCS, vol. 8690, pp. 584–599. Springer, Heidelberg (2014)Google Scholar
  13. 13.
    Fu, Y., Hospedales, T.M., Xiang, T., Gong, S.: Learning multi-modal latent attributes. TPAMI (2013)Google Scholar
  14. 14.
    Gannaz, I.: Robust estimation and wavelet thresholding in partial linear models. Stat. Comput. 17, 293–310 (2007)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Gehrlein, W.V.: Condorcet’s paradox. Theory and Decision (1983)Google Scholar
  16. 16.
    Gygli, M., Grabner, H., Riemenschneider, H., Nater, F., Gool, L.V.: The interestingness of images. In: ICCV (2013)Google Scholar
  17. 17.
    Hirani, A.N., Kalyanaraman, K., Watts, S.: Least squares ranking on graphs. arXiv:1011.1716 (2010)Google Scholar
  18. 18.
    Huber, P.J.: Robust Statistics. Wiley, New York (1981)CrossRefzbMATHGoogle Scholar
  19. 19.
    Isola, P., Xiao, J., Torralba, A., Oliva, A.: What makes an image memorable? In: CVPR (2011)Google Scholar
  20. 20.
    Jiang, X., Lim, L.H., Yao, Y., Ye, Y.: Statistical ranking and combinatorial hodge theory. Math. Program. (2011)Google Scholar
  21. 21.
    Jiang, Y.G., YanranWang, F.R., Xue, X., Zheng, Y., Yang, H.: Understanding and predicting interestingness of videos. In: AAAI (2013)Google Scholar
  22. 22.
    Ke, Y., Tang, X., Jing, F.: The design of high-level features for photo quality assessment. In: CVPR (2006)Google Scholar
  23. 23.
    Kittur, A., Chi, E.H., Suh., B.: Crowdsourcing user studies with mechanical turk. In: ACM CHI (2008)Google Scholar
  24. 24.
    Kovashka, A., Parikh, D., Grauman, K.: Whittlesearch: Image search with relative attribute feedback. In: CVPR (2012)Google Scholar
  25. 25.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: ICCV (2009)Google Scholar
  26. 26.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: CVPR (2009)Google Scholar
  27. 27.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI (2013)Google Scholar
  28. 28.
    Liu, F., Niu, Y., Gleicher, M.: Using web photos for measuring video frame interestingness. In: IJCAI (2009)Google Scholar
  29. 29.
    Liu, Y., Gao, B., Liu, T.Y., Zhang, Y., Ma, Z., He, S., Li, H.: Browserank: letting web users vote for page importance. In: ACM SIGIR (2008)Google Scholar
  30. 30.
    Long, C., Hua, G., Kapoor, A.: Active visual recognition with expertise estimation in crowdsourcing. In: ICCV (2013)Google Scholar
  31. 31.
    Maire, M., Yu, S.X., Perona, P.: Object detection and segmentation from joint embedding of parts and pixels. In: ICCV (2011)Google Scholar
  32. 32.
    Meinshausen, N., Bühlmann, P.: High-dimensional graphs and variable selection with the lasso. Ann. Statist. (2006)Google Scholar
  33. 33.
    Murphy, K.P.: Machine learning: a probabilistic perspective. The MIT Press (2012)Google Scholar
  34. 34.
    Oliva, A., Torralba., A.: Modeling the shape of the scene: Aholistic representation of the spatial envelope. IJCV 42 (2001)Google Scholar
  35. 35.
    Parikh, D., Grauman, K.: Relative attributes. In: ICCV (2011)Google Scholar
  36. 36.
    Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. In: Proc. CVPR (2012)Google Scholar
  37. 37.
    Rajkumar, A., Agarwal, S.: A statistical convergence perspective of algorithms for rank aggregation from pairwise data. In: Proceedings of the 31st International Conference on Machine Learning (2014)Google Scholar
  38. 38.
    She, Y., Owen, A.B.: Outlier detection using nonconvex penalized regression. Journal of American Statistical Association (2011)Google Scholar
  39. 39.
    Silvia, P.: Interest - the curious emotion. In: CDPS (2008)Google Scholar
  40. 40.
    Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. In: CVPR Workshops (2008)Google Scholar
  41. 41.
    Sun, Z., Qin, T., Tao, Q., Wang, J.: Robust sparse rank learning for non-smooth ranking measures. In: ACM SIGIR (2009)Google Scholar
  42. 42.
    Vedaldi, A., Zisserman, A.: Efficient additive kernels via explicit feature maps. In: IEEE TPAMI (2011)Google Scholar
  43. 43.
    Wu, O., Hu, W., Gao, J.: Learning to rank under multiple annotators. In: IJCAI (2011)Google Scholar
  44. 44.
    Xu, Q., Huang, Q., Jiang, T., Yan, B., Lin, W., Yao, Y.: Hodgerank on random graphs for subjective video quality assessment. IEEE TMM (2012)Google Scholar
  45. 45.
    Xu, Q., Huang, Q., Yao, Y.: Online crowdsourcing subjective image quality assessment. In: ACM MM (2012)Google Scholar
  46. 46.
    Xu, Q., Xiong, J., Huang, Q., Yao, Y.: Robust evaluation for quality of experience in crowdsourcing. In: ACM MM (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Yanwei Fu
    • 1
  • Timothy M. Hospedales
    • 1
  • Tao Xiang
    • 1
  • Shaogang Gong
    • 1
  • Yuan Yao
    • 2
  1. 1.School of EECSQueen Mary University of LondonUK
  2. 2.School of Mathematical SciencesPeking UniversityChina

Personalised recommendations