Semantic content based image retrieval using object-process diagrams
Abstract
The increase in accessability to on-line visual data has promoted the interest in browsing and retrieval of images from Image Databases. Current approaches assume either a text based key-word oriented approach or a visual feature based approach, The keyword approach usually provides neither layout information nor object relevence and significance in the scene. The visual feature based approach relies on low level features such as color, texture and orientation as image descriptors. These are non-intuitive and unnatural for human observers. This paper presents a new approach to image retrieval in which image content, based on the “visual scene”, is the basis for both retrieval and user interface. We propose to model image content using Object-Process Diagrams. Our hierarchical approach incorporates both the low-level image features and textual key sentencess as descriptors of the image. These descriptions involve the objects in the scene and their inter- and intra-relationships. This allows for abstract, high-level representation of the layout of the scene, as well as a distinction between the dominant core of the scene and its background. Querying is is performed by representing the sought image with an Object-Process Diagram and finding the images in the database whose Object-Process Diagrams best match the query.
Keywords
Visual Feature Image Retrieval Query Image Visual Scene Color HistogramReferences
- 1.J. Bigun, S.K. Bhattacharjee, and S. Michel. Orientation radiograms for image retrieval: an alternative to segmentation. In Proceedings of the 13th International Conference on Pattern Recognition, volume 3, pages 346–350, 1996.Google Scholar
- 2.A. Del Bimbo and P. Pala. Visual image retrieval by elastic matching of user sketches. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2):121–132, 1997.CrossRefGoogle Scholar
- 3.A. Finkelstein C.E. Jacobs and D.H. Salesin. Fast multiresolution image querying. In Computer Graphics Proceedings-SIGGRAPH 95, pages p. 277–286, 1995.Google Scholar
- 4.D. Dori. Arc segmentation in the machine drawing understanding environment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(11):1057–1068, 1995.CrossRefGoogle Scholar
- 5.D. Dori.Representing pattern recognition-embedded systems through object-process diagrams: the case of the machine drawing understanding system. Pattern Recognition Letters, 16(4):377–384, 1995.CrossRefGoogle Scholar
- 6.D. Dori. Unifying system structure and behaviour through object-process analysis. Journal of Logic and Computation, 5(2):227–249, 1995.Google Scholar
- 7.D. Dori. Analysis and representation o£ the image understanding environment using the object-process methodology. Journal of Object Oriented Programming, September, 1996.Google Scholar
- 8.D. Dori. Expressing structural relations among dimension-set components using the object-process methodology. Report on Object Analysis and Design, 2(6):20–24, 1996.Google Scholar
- 9.D. Dori. Object-process analysis of computer integrated manufacturing documentation and inspection. International Journal of Computer Integrated Manufacturing, 9(5):339–353, 1996.CrossRefGoogle Scholar
- 10.D. Dori. Unifying system structure and behaviour through object-process analysis. Journal of Object-Oriented Analysis, July–August:66–73, 1996.Google Scholar
- 11.D. Dori and M. Goodman. From object-process analysis to object-process design. Annals of Software Engineering, 2, 1996.Google Scholar
- 12.D. Dori and M. Goodman. On bridging the analysis-design and structure-behavior grand canyons with object paradignns. Report on Object Analysis and Design, 2(5):25–35, 1996.Google Scholar
- 13.D. Dori and M. Weiss. A scheme for 3d object reconstruction from dimensioned orthographic views. Engineering Applications of Artificial Intelligence, 9(1):53–64, 1996.CrossRefGoogle Scholar
- 14.D.A. Forsyth et. al. Finding pictures of objects in large collections of images. In Proceedings of the International Workshop on Object Representation in Computer Vision II, ECCCV-96, pages 335–360, 1996.Google Scholar
- 15.J. Hafner et. al. Efficient color histogram indexing for quadratic form distance functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(7):729–735. 1995.CrossRefGoogle Scholar
- 16.T. Kato et. al. A sketch retrieval method for full color image database-query by visual example. In Proceedings. 11th IAPR International Conference on Pattern Recognition, pages 530–533, 1992.Google Scholar
- 17.S. Even. Graph algorithms. Computer Science Press, Potomac, Md, 1979.Google Scholar
- 18.C. Faloutsos. Access methods for text. ACM Computing Survey, 1:49–74, 1985.CrossRefGoogle Scholar
- 19.G.L. Gimel'farb and A.K. Jain. 9. Pattern Recognition, 29:1461–1483, 1996.CrossRefGoogle Scholar
- 20.A.K. Jain and A. Vailaya. Image retrieval using color and shape. Pattern Recognition, 29(9):1233–1244, 1996.CrossRefGoogle Scholar
- 21.A. Kankanhalli, J.Z. Hong, and Y.L. Chien. Using texture for image retrieval. In Proceedings of The Third International Conference on Automation, Robotics and Computer Vision, volume 3, pages 935–939, 1994.Google Scholar
- 22.H.C. Lin, L.L. Wang, and S.N. Yang. Color image retrieval based on hidden markov models. IEEE Transactions on Image Processing, 6(2):332–339, 1997.CrossRefGoogle Scholar
- 23.W.Y. Ma and B.S. Manjunath. Texture-based pattern retrieval from image databases. Multimedia Tools and Applications, 2(1):35–51, 1996.Google Scholar
- 24.M. De Marsicoi, L. Cinque, and s. Levialdi. Indexing pictorial documents by their content: a survey of current techniques. Image and Vision Computing, 15(2):119–141, 1997.CrossRefGoogle Scholar
- 25.D. Meyersdorf and D. Dori. The r&d universe and its feedback cycles: an object-process analysis. R&D Management, To Appear.Google Scholar
- 26.W. Niblack, R. Barber, W. Equitz,M. Flickner,E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin. The QBIC project: Querying images by content, using color, texture, and shape. In SPIE Conference on Storage nad Retrieval for Image and Video Databases, volume 1908, pages 173–187, 1993.Google Scholar
- 27.A. Ono, M. Amano, M. Hakaridani, T. Satou, and M. Sakauchi. A flexible content-based image retrieval system with combined scene description keyword. In Proceedings of the International Conference on Multimedia Computing and Systems, pages 201–208, 1996.Google Scholar
- 28.G. Pass and R. Zabih. Histogram refinement for content-based image retrieval. In Proceeding. Third IEEE Workshop on Applications of Computer Vision, pages 96–102, 1996.Google Scholar
- 29.A. Pentland, R.W. Picard, and S. Sclaroff. Photobook: content-based manipulation of image databases. Int. Journal of Computer Vision, 18(3):233–254, 1996.CrossRefGoogle Scholar
- 30.R.W. Picard. A society of models for video and image libraries. IBM Systems Journal, 35(3-4):292–312, 1996.Google Scholar
- 31.Z. Qing-Long, C. Shi-Kuo, and S.S-T. Yan. Iconic indexing and maintenance of spatial relationships in image databases. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2916, pages 385 3116, 1996.Google Scholar
- 32.E. Reinim, G. Sheikholeslanii, and A. Zhang. Block-oriented image decomposition and retrieval in image database systems. In Proceedings. International Workshop on Multi-Media Database Management Systems, pages 85–92, 1996.Google Scholar
- 33.M. Shneier and M. Abdel-Mottaleb. Exploiting the jpeg compression scheme for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence; 18(8):849–853, 1996.CrossRefGoogle Scholar
- 34.H.G. Stark. On image retrieval with wavelets. International Journal of Imaging Systems and Technology, 7(3):200–210, 1996.CrossRefGoogle Scholar
- 35.M.J. Swain and D.H. Ballard. Color indexing. Int. Journal of Computer Vision, 7(1):11–32, 1991.CrossRefGoogle Scholar
- 36.D.L. Swets and J.J. Weng. Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):831–836, 1996.CrossRefGoogle Scholar
- 37.C.W. Tong and C.C. Chang, Application of geometric hashing to iconic database retrieval. Pattern Recognition Letters, 15(9):871–876, 1994.CrossRefGoogle Scholar
- 38.A. Vailaya, Z. Yu, and A.K. Jain. A hierarchical system for efficient image retrieval. In Proceedings of the 13th International Conference on Pattern Recognition, pages 356–360, 1996.Google Scholar
- 39.X. Wan and C.-C.J. Kuo. Image retrieval with multiresolution color space quantization. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2898, pages 148-1-59, 1996Google Scholar
- 40.X. Wan and C.J. Kuo. Image retrieval based on jpeg compressed data. In Proceedings of the SPIE — The International Society for Optical Engineering, volume 2916, pages 104–115, 1996.Google Scholar
- 41.G. Yihong, C.H. Chuan, and Z. Guo. Image indexing and retrieval based on color histograms. Multimedia Tools and Applications, 2(2):133–156, 1996.Google Scholar
- 42.J. You, H. Shan, and H.A. Cohen. An efficient parallel texture classification for image retrieval. In Proceedings. Advances in Parallel and Distributed Computing, pages 18–25, 1997.Google Scholar