Multimedia Tools and Applications

, Volume 77, Issue 23, pp 31115–31138 | Cite as

Extended Bayesian generalization model for understanding user’s intention in semantics based images retrieval

  • Meriem KorichiEmail author
  • Mohamed Lamine Kherfi
  • Mohamed Batouche
  • Khadra Bouanane


Learning concepts from examples presented in user’s query and infer the other items that belong to this query is still a significant challenge for images retrieval systems. Existing models from cognitive science namely Bayesian models of generalization mainly focus on this challenge where they remarkably succeed at explaining how to generalize from few examples in a wide range of domains. However their success largely depends on the validity of examples. They require that each example is a good representative, which is not always the case in the context of images retrieval. In this paper, we will extend the Bayesian models of generalization to identify the appropriate level of generalization for a given query in the context of query by semantic example systems. Our model uses an ontology as the basis of its hypothesis space which allows us to take advantages of its semantic richness and inference capacity. Experimental study using the ImageNet benchmark verifies the efficiency of our model in comparison to the state-of-the-art models of generalization.


Image retrieval User’s intention Bayesian models of generalization Ontology ImageNet 


  1. 1.
    Abbott JT, Austerweil JL, Griffiths TL (2012) Constructing a hypothesis space from the Web for large-scale Bayesian word learning. In Proceedings of the 34th Annual Conference of the Cognitive Science SocietyGoogle Scholar
  2. 2.
    Allani O, Zghal HB, Mellouli N, Akdag H (2016) A knowledge-based image retrieval system integrating semantic and visual features. Procedia Comput Sci 96(September):1428–1436CrossRefGoogle Scholar
  3. 3.
    Austerweil JL, Griffiths TL (2001) Learning hypothesis spaces and dimensions through concept learning. Citeseer, 73–78Google Scholar
  4. 4.
    Austerweil JL, Griffiths TL (2011) Seeking confirmation is rational for deterministic hypotheses. Cogn Sci 35(3):499–526CrossRefGoogle Scholar
  5. 5.
    Barnard K, Forsyth D (2001) Learning the semantics of words and pictures. In ICCV, volume 2, pages 408–415, VancouverGoogle Scholar
  6. 6.
    Ben-Haim N, Babenko B, Belongie S (2006) Improving Web- Based Image Search via Content Based Clustering, Proc. Int’lWorkshop Semantic Learning Applications in MultimediaGoogle Scholar
  7. 7.
    Blei D, Jordan M (2003) Modeling annotated data. In Proc. ACM SIGIRGoogle Scholar
  8. 8.
    Carneiro G, Chan A, Moreno P, Vasconcelos N (2007) Supervised learning of semantic classes for image annotation and retrieval. IEEE PAMI 29(3):394–410CrossRefGoogle Scholar
  9. 9.
    Celik C, Bilge HS (2017) Content based image retrieval with sparse representations and local feature descriptors : a comparative study. Pattern Recogn 68:1–13CrossRefGoogle Scholar
  10. 10.
    Datta D, Varma S, Chowdary CR, Singh SK (2017) Multimodal retrieval using mutual information based textual query reformulation. Expert Syst Appl 68:81–92CrossRefGoogle Scholar
  11. 11.
    Deepa C (2017) SABC-SBC: a hybrid ontology based image and webpage retrieval for datasets. Autom Control Comput Sci 51(2):108–113MathSciNetCrossRefGoogle Scholar
  12. 12.
    Deng JDJ, Dong WDW, Socher R, Li L.-J. L. L.-J, Li K. L. K, and Fei-Fei L. F.-F. L (2009) ImageNet: A large-scale hierarchical image database, 2009 IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2–9Google Scholar
  13. 13.
    Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) DeCAF: a deep convolutional activation feature for generic visual recognition. Icml 32:647–655Google Scholar
  14. 14.
    Faloutsos C, Barber R, Flickner M, Hafner J, Niblack W, Petkovic D, Equitz W (1994) Efficient and effective querying by image content. J Intell Inf Syst 3(3–4):231–262CrossRefGoogle Scholar
  15. 15.
    Filali J, Zghal HB, Martinet J (2016) Towards visual vocabulary and ontology-based image retrieval system. Proc 8th Int Conf Agents Artif Intell 2(Icaart):560–565Google Scholar
  16. 16.
    Gao L, Guo Z, Zhang H et al (2017) Video captioning with attention-based lstm and semantic consistency. IEEE Transactions on Multimedia 19(9):2045–2055CrossRefGoogle Scholar
  17. 17.
    Gao L, Song J, Liu X et al (2017) Learning in high-dimensional multimedia data: the state of the art. Multimedia Systems 23(3):303–313CrossRefGoogle Scholar
  18. 18.
    Ghahramani Z, Heller KA (2006) Bayesian Sets. Adv Neural Inf Proces Syst 18:435–442Google Scholar
  19. 19.
    Hannan MA, Arebey M, Begum RA, Basri H, Al Mamun MA (2016) Content-based image retrieval system for solid waste bin level detection and performance evaluation. Waste Manag 50:10–19CrossRefGoogle Scholar
  20. 20.
    Hartvedt C (2010) Using context to understand user intentions in image retrieval. 2nd Int Conf Adv Multimedia, MMEDIA 2010:130–133CrossRefGoogle Scholar
  21. 21.
    Heller KA (2008) Efficient Bayesian Methods for Clustering. Ph.D. thesis, University College London, Gatsby Computational Neuroscience UnitGoogle Scholar
  22. 22.
    Heller KA, Ghahramani Z (2006) A simple Bayesian framework for content-based image retrieval. IEEE Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  23. 23.
    Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-r, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. Signal Processing Magazine, IEEE 29(6):82–97CrossRefGoogle Scholar
  24. 24.
    Hsu WH, Kennedy LS, Chang S-F (2006) Video Search Reranking via Information Bottleneck Principle, Proc. 14th Ann. ACM Int’l Conf. MultimediaGoogle Scholar
  25. 25.
    Jia Y, Abbott J, Austerweil JL, Griffiths TL, Darrell T (2013) Visual Concept Learning: Combining Machine Vision and Bayesian Generalization on Concept Hierarchies, Adv. Neural Inf. Process. Syst. 27 (NIPS 2013), vol. 1, no. 1, pp. 1–9Google Scholar
  26. 26.
    Jing Y, Baluja S (2008) Pagerank for Product Image Search, Proc. Int’l Conf. World Wide WebGoogle Scholar
  27. 27.
    Kherfi ML (2008). Review of Human-Computer Interaction Issues in Image Retrieval, Advances in Human Computer Interaction, Shane Pinder (Ed.), InTech, DOI: Available from: Google Scholar
  28. 28.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106–1114Google Scholar
  29. 29.
    Krizhevsky A, Sutskever I, Hinton, Geoffrey E. (2012) Imagenet classification with deep convolutional neural networks. In : Advances in neural information processing systems. p. 1097–1105Google Scholar
  30. 30.
    Liaqat M, Khan S, Majid M (2017). Image retrieval based on fuzzy ontology. Multimedia Tools and ApplicationsGoogle Scholar
  31. 31.
    Miller GA (1995) WordNet: a lexical database for English. Commun ACM 38(11):39–41CrossRefGoogle Scholar
  32. 32.
    Natsev AP, Haubold A, Tešić J, Xie L, Yan R (2007) Semantic concept-based query expansion and re-ranking for multimedia retrieval. Proceedings of the 15th International Conference on MultimediaGoogle Scholar
  33. 33.
    Nematzadeh A, Grant E, Stevenson SA (2015) Computational Cognitive Model of Novel Word Generalization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 17–21, (September), 1795–1804Google Scholar
  34. 34.
    Niblack W et al. (1993) The QBIC project: Querying images by content using color, texture, and shape. In Storage and Retrieval for Image and Video Databases, pages 173–181, SPIEGoogle Scholar
  35. 35.
    Park G, Baek Y, Lee H (2003) Majority Based Ranking Approach in Web Image Retrieval, Proc. Second Int’l Conf. Image and Video RetrievalGoogle Scholar
  36. 36.
    Rasiwasia N, Vasconcelos N. (2008) A study of query by semantic example. 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR WorkshopsGoogle Scholar
  37. 37.
    Rasiwasia N, Moreno PJ, Vasconcelos N (2007) Bridging the gap: query by semantic example. IEEE Trans Multimed 9(5):923–938CrossRefGoogle Scholar
  38. 38.
    SHEPARD RN et al (1987) Toward a universal law of generalization for psychological science. Science 237(4820):1317–1323MathSciNetCrossRefGoogle Scholar
  39. 39.
    Silva R, Heller KA, Ghahramani Z (2007) Analogical reasoning with relational bayesian sets. International Conference on AI and StatisticsGoogle Scholar
  40. 40.
    Song J, He T, Gao L et al. (2017) Deep region hashing for efficient large-scale instance search from images. arXiv preprint arXiv:1701.07901Google Scholar
  41. 41.
    Song J, Gao L, Liu L et al (2018) Quantization-based hashing: a general framework for scalable image and video retrieval. Pattern Recogn 75:175–187CrossRefGoogle Scholar
  42. 42.
    Tang X, Liu K, Cui J, Wen F, Wang X (2012) IntentSearch:capturing user intention for one-click internet image search. IEEE Trans Pattern Anal Mach Intell 34(7):1342–1353CrossRefGoogle Scholar
  43. 43.
    Tenenbaum JB, Griffiths TL (2001) Generalization, similarity, and Bayesian inference, behavioral and brain sciences. pp. 629–630Google Scholar
  44. 44.
    Tenenbaum JB, Xu F (2000) Word learning as Bayesian inference, Proc. TwentySecond Annu. Conf. Cogn. Sci. SocGoogle Scholar
  45. 45.
    The BBC Wildlife Ontology. Available from:
  46. 46.
    Tolias G, Jégou H (2014) Visual query expansion with or without geometry: refining local descriptors by feature aggregation. Pattern Recogn 47(10):3466–3476CrossRefGoogle Scholar
  47. 47.
    Torralba A, Murphy K, Freeman W, Rubin M (2003) Context Based Vision System for Place and Object Recognition, Proc. Int’l Conf. Computer VisionGoogle Scholar
  48. 48.
    Wan, Ji, Wang, Dayong, Hoi, Steven, Chu Hong, et al. (2014) Deep learning for content-based image retrieval: A comprehensive study. In : Proceedings of the 22nd ACM international conference on Multimedia. ACM. p. 157–166Google Scholar
  49. 49.
    Wang X, Gao L, Wang P et al (2018) Two-stream 3-D convNet fusion for action recognition in videos with arbitrary size and length. IEEE Transactions on Multimedia 20(3):634–644CrossRefGoogle Scholar
  50. 50.
    Wang J, Zhang T, Sebe N et al (2018) A survey on learning to hash. IEEE Trans Pattern Anal Mach Intell 40(4):769–790CrossRefGoogle Scholar
  51. 51.
    Yu D, Seltzer ML, Li J, Huang J-T, and Seide F (2013) Feature learning in deep neural networks - a study on speech recognition tasks. CoRR, abs/1301.3605Google Scholar
  52. 52.
    Zeiler MD, Fergus R (2013) Visualizing and understanding convolutional networks. CoRR, abs/1311.2901Google Scholar
  53. 53.
    Zha Z-J, Yang L, Mei T, Wang M, & Wang Z. (2009). Visual query suggestion. Proceedings of the Seventeen ACM International Conference on Multimedia - MM ‘09, 6(3), 15Google Scholar
  54. 54.
    Zhao, Fang, Huang, Yongzhen, Wang, Liang, et al. (2015) Deep semantic ranking based hashing for multi-label image retrieval. In : Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, p. 1556–1564Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of Constantine 2 - Abdelhamid MehriConstantineAlgeria
  2. 2.LAMIA LaboratoryUniversity du Québec à Trois-RivièresTrois-RivièresCanada
  3. 3.LINATI LaboratoryUniversity of Ouargla - Kasdi MerbahOuarglaAlgeria
  4. 4.Department of MathematicsUniversity of Ouargla - Kasdi MerbahOuarglaAlgeria

Personalised recommendations