Online User Modeling for Interactive Streaming Image Classification

  • Jiagao Hu
  • Zhengxing SunEmail author
  • Bo Li
  • Kewei Yang
  • Dongyang Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10133)


Regarding of the explosive growth of personal images, this paper proposes an online user modeling method for the categorization of the streaming images. In the proposed framework, user interaction is brought in after an automatic classification by the learned classifier, and several strategies have been used for online user modeling. Firstly, to cover diverse personalized taxonomy, we describe images from multiple views. Secondly, to train the classifier gradually, we use an incremental variant of the nearest class mean classifier and update the class means incrementally. Finally, to learn diverse interests of different users, we propose an online learning strategy to learn weights of different feature views. Using the proposed method, user can categorize streaming images flexibly and freely without any pre-labeled images or pre-trained classifiers. And with the classification going on, the efficiency will keep increasing which could ease user’s interaction burden significantly. The experimental results and a user study demonstrated the effectiveness of our approach.


Image classification Online user modeling Streaming images Nearest class mean classifier Online metric learning 



This work is supported by National High Technology Research and Development Program of China (No. 2007AA01Z334), National Natural Science Foundation of China (No. 61321491, 61272219), Innovation Fund of State Key Laboratory for Novel Software Technology (No. ZZKT2013A12, ZZKT2016A11), Program for New Century Excellent Talents in University of China (NCET-04-04605).


  1. 1.
    Bergamo, A., Torresani, L., Fitzgibbon, A.W.: Picodes: learning a compact code for novel-category recognition. In: NIPS, pp. 2088–2096 (2011)Google Scholar
  2. 2.
    Chechik, G., Sharma, V., et al.: Large scale online learning of image similarity through ranking. J. Mach. Learn. Res. 11, 1109–1135 (2010)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Cheng, M., Zhang, Z., Lin, W., Torr, P.H.S.: BING: binarized normed gradients for objectness estimation at 300fps. In: CVPR, pp. 3286–3293. IEEE (2014)Google Scholar
  4. 4.
    Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., Singer, Y.: Online passive-aggressive algorithms. J. Mach. Learn. Res. 7, 551–585 (2006)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106(1), 59–70 (2007)CrossRefGoogle Scholar
  6. 6.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315(5814), 972–976 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Csurka, G., Dance, C.R., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: ECCV Workshop, pp. 1–22 (2004)Google Scholar
  8. 8.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1106–1114 (2012)Google Scholar
  9. 9.
    Kulis, B.: Metric learning: a survey. Found. Trends Mach. Learn. 5(4), 287–364 (2013)CrossRefzbMATHGoogle Scholar
  10. 10.
    Laskov, P., Gehl, C., Krueger, S., Müller, K.R.: Incremental support vector learning: analysis, implementation and applications. J. Mach. Learn. Res. 7, 1909–1936 (2006)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: CVPR, pp. 2169–2178. IEEE (2006)Google Scholar
  12. 12.
    Li, L.J., Fei-Fei, L.: What, where and who? Classifying events by scene and object recognition. In: ICCV, pp. 1–8. IEEE (2007)Google Scholar
  13. 13.
    Lin, Y., Lv, F., Zhu, S., et al.: Large-scale image classification: fast feature extraction and SVM training. In: CVPR 2011, pp. 1689–1696. IEEE (2011)Google Scholar
  14. 14.
    Mensink, T., Verbeek, J.J., Perronnin, F., Csurka, G.: Distance-based image classification: generalizing to new classes at near-zero cost. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2624–2637 (2013)CrossRefGoogle Scholar
  15. 15.
    Ristin, M., Guillaumin, M., Gall, J., Gool, L.J.V.: Incremental learning of random forests for large-scale image classification. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 490–503 (2016)CrossRefGoogle Scholar
  16. 16.
    Rui, Y., Huang, T., Ortega, M., Mehrotra, S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circ. Syst. Video Technol. 8(5), 644–655 (1998)CrossRefGoogle Scholar
  17. 17.
    Sanchez, J., Perronnin, F.: High-dimensional signature compression for large-scale image classification. In: CVPR 2011, pp. 1665–1672. IEEE (2011)Google Scholar
  18. 18.
    Su, Y., Jurie, F.: Learning compact visual attributes for large-scale image classification. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7585, pp. 51–60. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33885-4_6 CrossRefGoogle Scholar
  19. 19.
    Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: CVPR, pp. 1–9 (2015)Google Scholar
  20. 20.
    Torresani, L., Szummer, M., Fitzgibbon, A.: Efficient object category recognition using classemes. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 776–789. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15549-9_56 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jiagao Hu
    • 1
  • Zhengxing Sun
    • 1
    Email author
  • Bo Li
    • 1
  • Kewei Yang
    • 1
  • Dongyang Li
    • 1
  1. 1.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingPeople’s Republic of China

Personalised recommendations