Multimedia Tools and Applications

, Volume 71, Issue 3, pp 1087–1104 | Cite as

Instance based personalized multi-form image browsing and retrieval

  • Esin Guldogan
  • Thomas Olsson
  • Else Lagerstam
  • Moncef Gabbouj


It is important to adapt and personalize image browsing and retrieval systems based on users’ preferences for improved user experience and satisfaction. In this paper, we present a novel instance based personalized multi-form image representation with implicit relevance feedback and adaptive weighting approach for image browsing and retrieval systems. In the proposed system, images are grouped into forms, which represent different information on images such as location, content etc. We conducted user interviews on image browsing, sharing and retrieval systems for understanding image browsing and searching behaviors of users. Based on the insights gained from the user interview study we propose an adaptive weighting method and implicit relevance feedback for multi-form structures that aim to improve the efficiency and accuracy of the system. Statistics of the past actions are considered for modeling the target of the users. Thus, on each iteration weights of the forms are updated adaptively. Moreover, retrieval results are modified according to the users’ preferences on iterations in order to improve personalized user experience. The proposed method has been evaluated and results are illustrated in the paper. It is shown that, satisfactory improvements can be achieved with proposed approaches in the multi-form scheme.


Content-based image indexing and retrieval Image browsing Implicit feedback Personalized and adaptive image image browsing 


  1. 1.
    Benbunan-Fich R, Benbunan A (2007) Understanding user behavior with new mobile applications. J Strateg Inf Syst 16(4):393–412CrossRefGoogle Scholar
  2. 2.
    Bockting S, Ooms M, Hiemstra D, Vet PVD, Huibers T (2008) Evaluating relevance feedback: an image retrieval interface for children. In: Proceedings of the Dutch-Belgian Information Retrieval Workshop, 14–15 Apr, pp 15–20Google Scholar
  3. 3.
    Covey DT (2002) Usage and usability assessment: library practices and concerns. Digital Library Federation, Council on Library and Information Resources reports, JanuaryGoogle Scholar
  4. 4.
    Djordjevic D, Izquierdo E (2007) An object- and user-driven system for semantic-based image annotation and retrieval. IEEE Trans Circuits Syst Video Technol 17(3):313–323CrossRefGoogle Scholar
  5. 5.
    Eakins JP, Briggs P, Burford B (2004) Image retrieval interfaces: a user perspective. In: Proceedings of Third International Conference on Image and Video Retrieval, CIVR 2004, Proceedings of Lecture Notes in Computer Science 3115, Dublin, Ireland, July 21–23, pp 628–637Google Scholar
  6. 6.
    Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. IEEE. CVPR 2004, Workshop on Generative-Model Based VisionGoogle Scholar
  7. 7.
    Gray WD, Altmann EM (2001) Cognitive modeling and human-computer interaction. In: Karwowski W (ed) International encyclopedia of ergonomics and human factors, vol 1, pp 387–391Google Scholar
  8. 8.
    Guldogan E, Gabbouj M (2010) Adaptive image classification based on folksonomy. Proceedings of the International Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS 2010, Italy, 12–14 April, pp 1–4Google Scholar
  9. 9.
    Guldogan E, Lagerstam E, Olsson T, Gabbouj M (2010) Multi-form hierarchical representation of image categories for browsing and retrieval. Proceedings of the SMAP 2010, 5th International Workshop on Semantic Media Adaptation and Personalization, Cyprus, December, pp 64–69Google Scholar
  10. 10.
    Huiskes MJ, Lew MS (2008) The MIR Flickr Retrieval Evaluation. ACM International Conference on Multimedia Information Retrieval (MIR‘08), Vancouver, Canada, pp 39–43Google Scholar
  11. 11.
    Jaimes A (2006) Human factors in automatic image retrieval system design and evaluation. Invited paper, IS&T/SPIE Internet Imaging 2006, San Jose, CA, SPIE 6061, 606103, JanuaryGoogle Scholar
  12. 12.
    Jing F, Li M, Zhang H-J, Zhang B (2004) Relevance feedback in region-based image retrieval. IEEE Trans Circuits Syst Video Technol 14:672CrossRefGoogle Scholar
  13. 13.
    Kelly D, Belkin NJ (2001) Reading time, scrolling and interaction: Exploring implicit sources of user preference for relevance feedback. In: Proceedings of the 24th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR ‘01), USA, pp 408–409Google Scholar
  14. 14.
    Kim YH, Rhee PK. Automatic adaptation method in intelligent image retrieval system. Proceedings of the IEEE Region 10 Conference TENCON 99. South Korea, vol 1, pp 439–442Google Scholar
  15. 15.
    Kosch H, Döller M (2005) Multimedia database systems: where are we now? In: Proceedings of Int. Assoc. of Science and Technology for Development—Databases and Applications (IASTED-DBA), Innsbruck, AustriaGoogle Scholar
  16. 16.
    Kuniavsky M (2003) Observing the user experience: a practitioner’s guide to user research. Published by Morgan Kaufmann 560 p, pp 129–155Google Scholar
  17. 17.
    Laaksonen J, Koskela M, Laakso S, Oja E (2000) PicSOM—content-based image retrieval with self-organizing maps. Pattern Recogn Lett 21:1199–1207CrossRefzbMATHGoogle Scholar
  18. 18.
    Liu Y, Zhang D, Lu G, Ma W-Y (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recognit 40(1):262–282CrossRefzbMATHGoogle Scholar
  19. 19.
    Manavoglu E, Pavlov D, Lee Giles C (2003) Probabilistic user behavior models. IEEE International Conference On Data Mining, pp 203–210Google Scholar
  20. 20.
    Moghaddam B, Tian Q, Lesh N, Shen C, Huang TS (2004) Visualization and user-modeling for browsing personal photo libraries. Int J Comput Vision 56(1–2):109–130CrossRefGoogle Scholar
  21. 21.
    Piras L, Giacinto G (2009) Neighborhood-based feature weighting for relevance feedback in content-based retrieval. Workshop on Image Analysis for Multimedia Interactive Services, London, UK, May 6–8, pp 238–241Google Scholar
  22. 22.
    Rao Y, Mundur P, Yesha Y (2006) Fuzzy SVM ensembles for relevance feedback in image retrieval. Learning 350–359Google Scholar
  23. 23.
    Robertson S (2001) Evaluation in information retrieval. Lecture Notes in Computer Science 1980, USA, pp 81–92Google Scholar
  24. 24.
    Sandhaus P, Boll S (2011) Semantic analysis and retrieval in personal and social photo collections. Multimed Tools Appl 51:5–33CrossRefGoogle Scholar
  25. 25.
    Shen X, Tan B, Zhai C. Context-sensitive information retrieval using implicit feedback. Proceedings of The 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ‘05, 43, USA, pp 43–50Google Scholar
  26. 26.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  27. 27.
    Torres JM, Parkes A (2000) User modeling and adaptivity in visual information retrieval systems. In: Proceedings of the Workshop on Computational Semiotics for New MediaGoogle Scholar
  28. 28.
    Weiss D, Scheuerer J, Wenleder M, Erk A, Gülbahar M, Linnhoff-Popien C (2008) A user profile-based personalization system for digital multimedia content. In: Proceedings of the 3rd International Conference On Digital Interactive Media In Entertainment And Arts, DIMEA ‘08, Athens, Greece, September, vol 349, pp 281–288Google Scholar
  29. 29.
    Zhou XS, Huang TS (2000) CBIR: from low-level features to high-level semantics. In: Proceedings of SPIE Image and Video Communication and Processing, pp 24–28Google Scholar
  30. 30.
    Zhou X, Huang TS (2003) Relevance feedback for image retrieval: a comprehensive review. ACM Multimed Syst 8:536–554CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Esin Guldogan
    • 1
  • Thomas Olsson
    • 2
  • Else Lagerstam
    • 2
  • Moncef Gabbouj
    • 1
  1. 1.Department of Signal ProcessingTampere University of TechnologyTampereFinland
  2. 2.Unit of Human-Centered TechnologyTampere University of TechnologyTampereFinland

Personalised recommendations