A Visual Multimedia Query Language for Temporal Analysis of Video Data

  • Stacie Hibino
  • Elke A. Rundensteiner


The storage of various media in multimedia databases poses new challenges to query techniques — challenges that exceed the expressive power of traditional text-based query languages. New query interfaces should take advantage of characteristics inherent in multimedia data, such as the dynamic temporal nature of video, the visual and spatial characteristics of images, the pitch of audio, etc. The focus of our research is to exploit the temporal continuity and combined spatio-temporal characteristics of video data for the purpose of video analysis. We do so by integrating a visual query paradigm with a dynamic visual presentation of results into a user-friendly interactive visualization environment. In this chapter, we present our overall approach for identifying trends in video data via querying for relationships between video annotations. Our approach allows users to analyze the video in terms of temporal relationships between events (e.g., events of type A frequently follow events of type B). We present a temporal visual query language (TVQL) for specifying relative temporal queries between sets of annotations. This query language builds on the notion of dynamic query filters and significantly extends them. It is tailored for temporal analysis — allowing users to pose queries, as well as to browse the data in a temporally continuous manner, thereby aiding them in the discovery of temporal trends. The TVQL is augmented with complementary temporal diagrams, which provide intuitive visual feedback for quickly and qualitatively verifying the temporal query specified. This chapter includes the complete specification of our TVQL, a transformation function for generating corresponding temporal diagrams, and the process for mapping TVQL queries to system queries.


Video Data Query Language Video Segment Video Object Media Object 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Allen, J.F. (1983). Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11), 832–843.MATHCrossRefGoogle Scholar
  2. [2]
    Ahlberg, C., Williamson, C., & Shneiderman, B. (1992). Dynamic Queries for Information Exploration: An Implementation and Evaluation. CHI’92 Conference Proceedings, 619–626: ACM Press.Google Scholar
  3. [3]
    Ahlberg, C., & Shneiderman, B. (1994). Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfleld Displays. CHF’94 Conference Proceedings, 313–317: ACM Press.Google Scholar
  4. [4]
    Chakravarthy, A.S. (1994). Toward Semantic Retrieval of Pictures and Video. AAAI’94 Workshop on Indexing and Reuse in Multimedia Systems, 12–18.Google Scholar
  5. [5]
    Chu, W.W., Ieong, I.T., Taira, R.K., & Breant, C.M. (1992). A Temporal Evolutionary Object-Oriented Data Model and Its Query Language for Medical Image Management. Proceedings of the 18th VLDB Conference, 53–64: Very Large Data Base Endowment.Google Scholar
  6. [6]
    Chua, T.-S., Lim, S.-K., & Pung, H.-K. (1994). Content-Based Retrieval of Segmented Images. ACM Multimedia’94 Proceedings: ACM Press.Google Scholar
  7. [7]
    Davis, M. (1994). Knowledge Representation for Video. Proceedings of the Twelfth National Conference on Artificial Intelligence, 120–127: AAAI Press.Google Scholar
  8. [8]
    Freksa, C. (1992). Temporal reasoning based on semi-intervals. Artificial Intelligence, 54 (1992), 199–227.MathSciNetCrossRefGoogle Scholar
  9. [9]
    Fishkin, K. and Stone, M.C. (1995). Enhanced Dynamic Queries via Movable Filters. CHI’95 Conference Proceedings, 415–420. ACM Press.Google Scholar
  10. [10]
    Gevers, T. and Smeulders, A.W.M. (1992). Indexing of Images by Pictorial Information. Visual Database Systems, II (E. Knuth and L.M. Wegner, Eds.), North Holland: Amsterdam, 93–100.Google Scholar
  11. [11]
    Goldstein, J. & Roth, S. (1994). Using Aggregation and Dynamic Queries for Exploring Large Data Sets. CEI’ 94 Conference Proceedings, 23–29. ACM Press.Google Scholar
  12. [12]
    Hampapur, A., Weymouth, T., & Jain, R. (1994). Digital Video Segmentation. ACM Multimedia’94 Proceedings, 357–364: ACM Press.Google Scholar
  13. [13]
    Harrison, B.L., Owen, R., & Baecker, R.M. (1994). Timelines: An Interactive System for the Collection of Visualization of Temporal Data. Proceedings of Graphics Interface’ 94. Canadian Information Processing Society.Google Scholar
  14. [14]
    Hibino, S. & Rundensteiner, E. (1995a). A Visual Query Language for Identifying Temporal Trends in Video Data. To appear in Proceedings of the First International Workship on Multimedia Database Management Systems.Google Scholar
  15. [15]
    Hibino, S. & Rundensteiner, E. (1995b). Interactive Visualizations for Exploration and Spatio-Temporal Analysis of Video Data. To appear in IJCAI’95 Workshop on Intelligent Multimedia Information Retrieval.Google Scholar
  16. [16]
    Hibino, S. & Rundensteiner, E. (Dec 1994). A Graphical Query Language for Identifying Temporal Trends in Video Data. University of Michigan, EECS Technical Report CSE-TR-225-94.Google Scholar
  17. [17]
    Lenat, D. & Guha, R.V. (1994). Strongly Semantic Information Retrieval. AAAI’94 Workshop on Indexing and Reuse in Multimedia Systems, 58–68.Google Scholar
  18. [18]
    Little, T.D.C. & Ghafoor, A. (1993). Interval-Based Conceptual Models for Time-Dependent Multimedia Data. IEEE Transactions on Knowledge and Data Engineering, 5(4), 551–563.CrossRefGoogle Scholar
  19. [19]
    Mackay, W. E. (1989). EVA: An experimental video annotator for symbolic analysis of video data. SIGCHI Bulletin, 21(2), 68–71.CrossRefGoogle Scholar
  20. [20]
    Nagasaka, A. and Tanaka, A. (1992). Automatic Video Indexing and Full-Video Search for Object Appearances. Visual Database Systems, II (E. Knuth and L.M. Wegner, Eds.), 113–127. Elsevier Science Publishers.Google Scholar
  21. [21]
    Oomoto, E. & Tanaka, K. (1993). OVID: Design and Implementation of a Video-Object Database System. IEEE Transactions on Knowledge and Data Engineering, 5(4), 629–643.CrossRefGoogle Scholar
  22. [22]
    Roschelle, J., Pea, R., & Trigg, R. (1990). VIDEONOTER: A tool for exploratory analysis (Research Rep. No. IRL90-0021). Palo Alto, CA: Institute for Research on Learning.Google Scholar
  23. [23]
    Santucci, G. & Sottile, P.A. (1993). Query by Diagram: a Visual Environment for Querying Databases. Software — Practice and Experience, 23(3), 317–340.CrossRefGoogle Scholar
  24. [24]
    Snodgrass, R. (1987). The Temporal Query Language TQuel. ACM Transactions on Database Systems, 12(2), 247–298.CrossRefGoogle Scholar
  25. [25]
    Snodgrass, R. (1992). Temporal Databases. Theories and Methods of Spatio-Temporal Reasoning in Geographic Space (A.U. Frank, I. Campari, and U. Formentini, Eds.), Springer-Verlag: New York, 22–64.Google Scholar
  26. [26]
    Ueda, H., Miyatake, T., Sumino, S., & Nagasaka, A. (1993). Automatic Structure Visualization for Video Editing. InterCEI’93 Proceedings, (pp. 137–141): ACM Press.Google Scholar
  27. [27]
    Weber, K. & Poon, A. (1994). Marquee: A Tool for Real-Time Video Logging. CHI’94 Conference Proceedings, 58–64: ACM Press.Google Scholar
  28. [28]
    Whang, K., Malhotra, A., Sockut, G., Burns, L., & Coi, K-S. (1992). Two-Dimensional Specification of Universal Quantification in a Graphical Database Query Language, IEEE Transactions on Software Engineering, 18(3), 216–224.CrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Stacie Hibino
    • 1
  • Elke A. Rundensteiner
    • 1
  1. 1.Electrical Engr. and Computer Sc. Dept. Software Systems Research LaboratoryUniversity of MichiganAnn ArborUSA

Personalised recommendations