Semantic content based image retrieval using object-process diagrams

  • Dov Dori
  • Hagit Hel-Or
Invited Talks
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1451)


The increase in accessability to on-line visual data has promoted the interest in browsing and retrieval of images from Image Databases. Current approaches assume either a text based key-word oriented approach or a visual feature based approach, The keyword approach usually provides neither layout information nor object relevence and significance in the scene. The visual feature based approach relies on low level features such as color, texture and orientation as image descriptors. These are non-intuitive and unnatural for human observers. This paper presents a new approach to image retrieval in which image content, based on the “visual scene”, is the basis for both retrieval and user interface. We propose to model image content using Object-Process Diagrams. Our hierarchical approach incorporates both the low-level image features and textual key sentencess as descriptors of the image. These descriptions involve the objects in the scene and their inter- and intra-relationships. This allows for abstract, high-level representation of the layout of the scene, as well as a distinction between the dominant core of the scene and its background. Querying is is performed by representing the sought image with an Object-Process Diagram and finding the images in the database whose Object-Process Diagrams best match the query.


Visual Feature Image Retrieval Query Image Visual Scene Color Histogram 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    J. Bigun, S.K. Bhattacharjee, and S. Michel. Orientation radiograms for image retrieval: an alternative to segmentation. In Proceedings of the 13th International Conference on Pattern Recognition, volume 3, pages 346–350, 1996.Google Scholar
  2. 2.
    A. Del Bimbo and P. Pala. Visual image retrieval by elastic matching of user sketches. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2):121–132, 1997.CrossRefGoogle Scholar
  3. 3.
    A. Finkelstein C.E. Jacobs and D.H. Salesin. Fast multiresolution image querying. In Computer Graphics Proceedings-SIGGRAPH 95, pages p. 277–286, 1995.Google Scholar
  4. 4.
    D. Dori. Arc segmentation in the machine drawing understanding environment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(11):1057–1068, 1995.CrossRefGoogle Scholar
  5. 5.
    D. Dori.Representing pattern recognition-embedded systems through object-process diagrams: the case of the machine drawing understanding system. Pattern Recognition Letters, 16(4):377–384, 1995.CrossRefGoogle Scholar
  6. 6.
    D. Dori. Unifying system structure and behaviour through object-process analysis. Journal of Logic and Computation, 5(2):227–249, 1995.Google Scholar
  7. 7.
    D. Dori. Analysis and representation o£ the image understanding environment using the object-process methodology. Journal of Object Oriented Programming, September, 1996.Google Scholar
  8. 8.
    D. Dori. Expressing structural relations among dimension-set components using the object-process methodology. Report on Object Analysis and Design, 2(6):20–24, 1996.Google Scholar
  9. 9.
    D. Dori. Object-process analysis of computer integrated manufacturing documentation and inspection. International Journal of Computer Integrated Manufacturing, 9(5):339–353, 1996.CrossRefGoogle Scholar
  10. 10.
    D. Dori. Unifying system structure and behaviour through object-process analysis. Journal of Object-Oriented Analysis, July–August:66–73, 1996.Google Scholar
  11. 11.
    D. Dori and M. Goodman. From object-process analysis to object-process design. Annals of Software Engineering, 2, 1996.Google Scholar
  12. 12.
    D. Dori and M. Goodman. On bridging the analysis-design and structure-behavior grand canyons with object paradignns. Report on Object Analysis and Design, 2(5):25–35, 1996.Google Scholar
  13. 13.
    D. Dori and M. Weiss. A scheme for 3d object reconstruction from dimensioned orthographic views. Engineering Applications of Artificial Intelligence, 9(1):53–64, 1996.CrossRefGoogle Scholar
  14. 14.
    D.A. Forsyth et. al. Finding pictures of objects in large collections of images. In Proceedings of the International Workshop on Object Representation in Computer Vision II, ECCCV-96, pages 335–360, 1996.Google Scholar
  15. 15.
    J. Hafner et. al. Efficient color histogram indexing for quadratic form distance functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(7):729–735. 1995.CrossRefGoogle Scholar
  16. 16.
    T. Kato et. al. A sketch retrieval method for full color image database-query by visual example. In Proceedings. 11th IAPR International Conference on Pattern Recognition, pages 530–533, 1992.Google Scholar
  17. 17.
    S. Even. Graph algorithms. Computer Science Press, Potomac, Md, 1979.Google Scholar
  18. 18.
    C. Faloutsos. Access methods for text. ACM Computing Survey, 1:49–74, 1985.CrossRefGoogle Scholar
  19. 19.
    G.L. Gimel'farb and A.K. Jain. 9. Pattern Recognition, 29:1461–1483, 1996.CrossRefGoogle Scholar
  20. 20.
    A.K. Jain and A. Vailaya. Image retrieval using color and shape. Pattern Recognition, 29(9):1233–1244, 1996.CrossRefGoogle Scholar
  21. 21.
    A. Kankanhalli, J.Z. Hong, and Y.L. Chien. Using texture for image retrieval. In Proceedings of The Third International Conference on Automation, Robotics and Computer Vision, volume 3, pages 935–939, 1994.Google Scholar
  22. 22.
    H.C. Lin, L.L. Wang, and S.N. Yang. Color image retrieval based on hidden markov models. IEEE Transactions on Image Processing, 6(2):332–339, 1997.CrossRefGoogle Scholar
  23. 23.
    W.Y. Ma and B.S. Manjunath. Texture-based pattern retrieval from image databases. Multimedia Tools and Applications, 2(1):35–51, 1996.Google Scholar
  24. 24.
    M. De Marsicoi, L. Cinque, and s. Levialdi. Indexing pictorial documents by their content: a survey of current techniques. Image and Vision Computing, 15(2):119–141, 1997.CrossRefGoogle Scholar
  25. 25.
    D. Meyersdorf and D. Dori. The r&d universe and its feedback cycles: an object-process analysis. R&D Management, To Appear.Google Scholar
  26. 26.
    W. Niblack, R. Barber, W. Equitz,M. Flickner,E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin. The QBIC project: Querying images by content, using color, texture, and shape. In SPIE Conference on Storage nad Retrieval for Image and Video Databases, volume 1908, pages 173–187, 1993.Google Scholar
  27. 27.
    A. Ono, M. Amano, M. Hakaridani, T. Satou, and M. Sakauchi. A flexible content-based image retrieval system with combined scene description keyword. In Proceedings of the International Conference on Multimedia Computing and Systems, pages 201–208, 1996.Google Scholar
  28. 28.
    G. Pass and R. Zabih. Histogram refinement for content-based image retrieval. In Proceeding. Third IEEE Workshop on Applications of Computer Vision, pages 96–102, 1996.Google Scholar
  29. 29.
    A. Pentland, R.W. Picard, and S. Sclaroff. Photobook: content-based manipulation of image databases. Int. Journal of Computer Vision, 18(3):233–254, 1996.CrossRefGoogle Scholar
  30. 30.
    R.W. Picard. A society of models for video and image libraries. IBM Systems Journal, 35(3-4):292–312, 1996.Google Scholar
  31. 31.
    Z. Qing-Long, C. Shi-Kuo, and S.S-T. Yan. Iconic indexing and maintenance of spatial relationships in image databases. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2916, pages 385 3116, 1996.Google Scholar
  32. 32.
    E. Reinim, G. Sheikholeslanii, and A. Zhang. Block-oriented image decomposition and retrieval in image database systems. In Proceedings. International Workshop on Multi-Media Database Management Systems, pages 85–92, 1996.Google Scholar
  33. 33.
    M. Shneier and M. Abdel-Mottaleb. Exploiting the jpeg compression scheme for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence; 18(8):849–853, 1996.CrossRefGoogle Scholar
  34. 34.
    H.G. Stark. On image retrieval with wavelets. International Journal of Imaging Systems and Technology, 7(3):200–210, 1996.CrossRefGoogle Scholar
  35. 35.
    M.J. Swain and D.H. Ballard. Color indexing. Int. Journal of Computer Vision, 7(1):11–32, 1991.CrossRefGoogle Scholar
  36. 36.
    D.L. Swets and J.J. Weng. Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):831–836, 1996.CrossRefGoogle Scholar
  37. 37.
    C.W. Tong and C.C. Chang, Application of geometric hashing to iconic database retrieval. Pattern Recognition Letters, 15(9):871–876, 1994.CrossRefGoogle Scholar
  38. 38.
    A. Vailaya, Z. Yu, and A.K. Jain. A hierarchical system for efficient image retrieval. In Proceedings of the 13th International Conference on Pattern Recognition, pages 356–360, 1996.Google Scholar
  39. 39.
    X. Wan and C.-C.J. Kuo. Image retrieval with multiresolution color space quantization. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2898, pages 148-1-59, 1996Google Scholar
  40. 40.
    X. Wan and C.J. Kuo. Image retrieval based on jpeg compressed data. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2916, pages 104–115, 1996.Google Scholar
  41. 41.
    G. Yihong, C.H. Chuan, and Z. Guo. Image indexing and retrieval based on color histograms. Multimedia Tools and Applications, 2(2):133–156, 1996.Google Scholar
  42. 42.
    J. You, H. Shan, and H.A. Cohen. An efficient parallel texture classification for image retrieval. In Proceedings. Advances in Parallel and Distributed Computing, pages 18–25, 1997.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  • Dov Dori
    • 1
  • Hagit Hel-Or
    • 2
  1. 1.Faculty of Industrial Engineering and ManagementTechnion - Israel Institute of TechnologyHaifaIsrael
  2. 2.Department of Mathematics and Computer ScienceBar-Ilan UniversityRamat-GanIsrael

Personalised recommendations