Towards Semantic Image Retrieval Using Multimodal Fusion with Association Rules Mining

  • Raniah A. Alghamdi
  • Mounira Taileb
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8521)


This paper proposes a semantic retrieving method for an image retrieval system that employs the fusion of the textual and visual information of the image dataset which is a recent trend in image retrieval researches. It combines two different data mining techniques to retrieve semantically related images: clustering and association rule mining algorithm. At the offline phase of the method, the association rules are discovered between the text semantic clusters and the visual clusters to use it later in the online phase. To evaluate the proposed system, the experiment was conducted on more than 54,500 images of ImageCLEF 2011 Wikipedia collection. The proposed retrieval system was compared to an online system called MMRetrieval and to the proposed system but without using association rules. The obtained results show that our proposed method achieved the best precision and mean average precision.


Image Retrieval Multimodal Fusion Association Rules Mining Clustering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Liu, Y., Zhanga, D., Lua, G., Ma, W.-Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recognition 40(1), 262–282 (2007)CrossRefzbMATHGoogle Scholar
  2. 2.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys (CSUR) 40(2), 1–60 (2008)CrossRefGoogle Scholar
  3. 3.
    Wu, S., McClean, S.: Performance prediction of data fusion for information retrieval. Management Information Processing 42(4), 899–915 (2006)CrossRefGoogle Scholar
  4. 4.
    Müller, H., Clough, P., Deselaers, T., Caputo, B.: ImageCLEF. The Springer International Series on Information Retrieval, vol. 32, pp. 95–114. Springer (2010)Google Scholar
  5. 5.
    Ferecatu, M., Sahbi, H.: TELECOM ParisTech at ImageClefphoto 2008: Bi–modal text and image retrieval with diversity enhancement. In: Working Notes of CLEF 2008, Aarhus, Denmark (2008)Google Scholar
  6. 6.
    Deselaers, T., Weyand, T., Ney, H.: Image retrieval and annotation using maximum entropy. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 725–734. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  7. 7.
    Gass, T., Weyand, T., Deselaers, T., Ney, H.: FIRE in ImageCLEF 2007: Support vector machines and logistic models to fuse image descriptors for photo retrieval. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 492–499. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  8. 8.
    Bellman, R.: Adaptive Control Process: A Guided Tour. Princeton University Press (1961)Google Scholar
  9. 9.
    Atrey, P.K., Hossain, M.A., Saddik, A.E., Kankanhall, M.S.: Multimodal fusion for multimedia analysis: A survey. Multimedia Syst. 16(3), 1432–1882 (2010)Google Scholar
  10. 10.
    Lau, C., Tjondronegoro, D., Zhang, J., Geva, S., Liu, Y.: Fusing visual and textual retrieval techniques to efectively search large collections of wikipedia images. International Journal of Business Intelligence and Data Mining, 345–357 (2007)Google Scholar
  11. 11.
    Frigui, H., Caudill, J., Ben Abdallah, A.: Fusion of multi-modal features for effiecient content-based image retrieval. In: IEEE World Congress on Computational Intelligence, pp. 1992–1998 (2008)Google Scholar
  12. 12.
    Bartolini, I., Ciaccia, P.: Scenique: a multimodal image retrieval interface. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 476–477. ACM, Italy (2008)CrossRefGoogle Scholar
  13. 13.
    Zhou, X., Depeursinge, A., Müller, H.: Information fusion for combining visual and textual image retrieval. In: Proceedings of the 20th International Conference on Recognizing Patterns in Signals, Speech, Images, and Videos, pp. 1590–1593 (2010)Google Scholar
  14. 14.
    Tong, H., He, J.R., Li, M.J., Zhang, C.S., Ma, W.Y.: Graph-based multi-modality learning. In: Proceeding of the ACM Int. Conf. on Multimedia, pp. 862–871 (2005)Google Scholar
  15. 15.
    Escalante, H.J., Hernandez, C., Sucar, E., Montes, M.: Late fusion of heterogeneous methods for multimedia image retrieval. In: Proceeding of MIR, pp. 172–179. ACM, Vancouver (2008)Google Scholar
  16. 16.
    Zagoris, K., Arampatzis, A., Chatzichristofis, S.A.: a multimodal search engine. In: Proceedings of the Third International Conference on SImilarity Search and Applications, Turkey (2010)Google Scholar
  17. 17.
    Ah-Pine, J., Bressan, M., Clinchant, S., Csurka, G., Hoppenot, Y., Renders, J.: Crossing textual and visual content in different application scenarios. Multimedia Tools and Applications 42(1), 31–56 (2009)Google Scholar
  18. 18.
    Wei, S., Zhao, Y., Zhu, Z., Liu, N.: Multimodal Fusion for Video Search Reranking. IEEE Transactions on Knowledge and Data Engineering 22(8), 1191–1199 (2010)CrossRefGoogle Scholar
  19. 19.
    He, R., Xiong, N., Yang, L., Park, J.: Using multi-modal semantic association rules to fuse keywords and visual features automatically for web image retrieval. In: International Conference on Information Fusion (2011)Google Scholar
  20. 20.
    Kouomou-Choupo, A., Berti-Equille, L., Morin, A.: Multimedia indexing and retrieval with features association rules mining. In: IEEE International Conference on Multimedia and Expo (ICME 2004), pp. 1299–1302 (2004)Google Scholar
  21. 21.
    Agrawal, R., Imielinski, T., Swami, A.: Mining association rules between sets of items in large databases. In: Proceedings of ACM SIGMOD, pp. 207–216 (1993)Google Scholar
  22. 22.
    Manjunath, B.S., Salembier, P., Sikora, T.: Introduction to MPEG-7: multimedia content description interface. Wiley (2002)Google Scholar
  23. 23.
    Taileb, M., Lamrous, S., Touati, S.: Non Overlapping Hierarchical Index Structure. International Journal of Computer Science 3(1), 29–35 (2008)Google Scholar
  24. 24.
    Wu, S.: Data Fusion in Information Retrieval. ALO, vol. 13. Springer, Heidelberg (2012)zbMATHGoogle Scholar
  25. 25.
    Tsikrika, T., Popescu, A., Kludas, J.: Overview of the Wikipedia Image Retrieval Task at ImageCLEF 2011. In: Working Notes of CLEF 2011, Amsterdam, The Netherlands (2011)Google Scholar
  26. 26.
    Bastan, M., Cam, H., Gudukbay, U., Ulusoy, O.: BilVideo-7: An MPEG-7 Compatible Video Indexing and Retrieval System. IEEE MultiMedia 17(3), 62–73 (2010)CrossRefGoogle Scholar
  27. 27.
    Müller, H., Clough, P., Deselaers, T., Caputo, B.: ImageCLEF. The Springer International Series on Information Retrieval, vol. 32, pp. 81–92. Springer (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Raniah A. Alghamdi
    • 1
  • Mounira Taileb
    • 1
  1. 1.Faculty of Computing and Information TechnologyKing Abdulaziz UniversityJeddahSaudi Arabia

Personalised recommendations