Abstract
Content-based video retrieval systems (CBVR) are creating new search and browse capabilities using metadata describing significant features of the data. An often overlooked aspect of human interpretation of multimedia data is the affective dimension. Incorporating affective information into multimedia metadata can potentially enable search using this alternative interpretation of multimedia content. Recent work has described methods to automatically assign affective labels to multimedia data using various approaches. However, the subjective and imprecise nature of affective labels makes it difficult to bridge the semantic gap between system-detected labels and user expression of information requirements in multimedia retrieval. We present a novel affect-based video retrieval system incorporating an open-vocabulary query stage based on WordNet enabling search using an unrestricted query vocabulary. The system performs automatic annotation of video data with labels of well defined affective terms. In retrieval annotated documents are ranked using the standard Okapi retrieval model based on open-vocabulary text queries. We present experimental results examining the behaviour of the system for retrieval of a collection of automatically annotated feature films of different genres. Our results indicate that affective annotation can potentially provide useful augmentation to more traditional objective content description in multimedia retrieval.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Eakins, J.P.: Automatic image content retrieval - are we getting anywhere? In: Proceedings of the 3rd International Conference on Electronic Library and Visual Information Research, pp. 123–135. De Montfort University, Milton Keynes (1996)
Hanjalic, A., Qun Xu, L.: Affective Video Content Representation and Modeling. IEEE Transactions on Multimedia 7(1), 143–154 (2005)
Lee, H., Smeaton, A.F., McCann, P., Murphy, N., O’Connor, N., Marlow, S.: FÃschlár on a PDA: A Handheld User Interface to a Video Indexing, Browsing, and Playback System. In: ERCIM Workshop User Interfaces for All, Florence, Italy (2000)
Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1349–1380 (2000)
Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based Multimedia Information Retrieval: State of the Art and Challenges. ACM Transactions on Multimedia Computing, Communications and Applications 2(1), 1–19 (2006)
Hauptmann, A.G.: Lessons for the future from a decade of informedia video analysis research. In: Leow, W.-K., Lew, M., Chua, T.-S., Ma, W.-Y., Chaisorn, L., Bakker, E.M. (eds.) CIVR 2005. LNCS, vol. 3568, pp. 1–10. Springer, Heidelberg (2005)
Hauptmann, A.G., Christel, M.G.: Successful Approaches in the TREC Video Retrieval Evaluations. In: Proceedings of the Twelfth ACM International Conference on Multimedia 2004, New York, NY, USA, pp. 668–675 (2004)
Zhang, T., Jay Kuo, C.-C.: Content-Based Audio Classification and Retrieval for Audiovisual Data Parsing. Kluwer Academic Publishers, Dordrecht (2001)
Zhai, Y., Shah, M., Rasheed, Z.: A Framework for Semantic Classification of Scenes using Finite State Machines. In: Proceedings of the Conference for Image and Video Retrieval, Dublin, Ireland, pp. 279–288 (2004)
Browne, P., Smeaton, A.F.: Video Information Retrieval Using Objects and Ostensive Relevance Feedback. In: ACM Symposium on Applied Computing, Nicosia, Cyprus, pp. 1084–1090 (2004)
Sadlier, D., O’Connor, N.: Event Detection based on Generic Characteristics of Field Sports. In: IEEE International Conference on Multimedia and Expo. (ICME 2005), Amsterdam, The Netherlands, pp. 759–762 (2005)
Jones, G.J.F., Chan, C.H.: Affect-Based Indexing for Multimedia Data. In: Maybury, M.T. (ed.) Multimedia Information Extraction. IEEE Computer Society Press, Los Alamitos (2011)
Russell, J.A., Mehrabian, A.: Evidence for a Three-Factor Theory of Emotions. Journal of Research in Personality 11, 273–294 (1977)
de Kok, I.: A Model for Valence Using a Color Component in Affective Video Content Analysis. In: Proceedings of the Fourth Twente Student Conference on IT, Faculty of Electrical Engineering, Mathematics and Computer Science. University of Twente, The Netherlands (2006)
Salway, A., Graham, M.: Extracting Information about Emotions in Films. In: Proceedings of the Eleventh ACM International Conference on Multimedia 2003, Berkeley, CA, USA, pp. 299–302 (2003)
Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press Inc., Palo Alto (1978)
Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., Schröder, M.: ‘FEELTRACE’: An Instrument for Recording Perceived Emotion in Real Time. In: Proceedings of the ISCA Workshop on Speech and Emotion: A Conceptual Framework for Research, Belfast, U.K, pp. 19–24 (2000)
Harman, D. K.: The Fifth Text Retrieval Conference (TREC-5). National Institute of Standards and Technology, Gaithersburg(1997)
Robertson, S.E., Spärck Jones, K.: Simple, proven approaches to text retrieval, Technical Report, TR356, Cambridge University Computer Laboratory (1997)
Pedersen, T., Patwardhan, S., Michelizzi, J.: WordNet:Similarity - Measuring the Relatedness of Concepts. In: Proceedings of the 19th National Conference on Artificial Intelligence (AAAI 2004), San Jose, CA, USA (2004)
Leacock, C., Chodorow, M.: Combining local context and WordNet similarity for word sense identification. In: WordNet: An Electronic Lexical Database, pp. 265–283. MIT Press, Cambridge (1998)
Resnik, P.: Using information content to evaluate semantic similarity. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, pp. 488–453 (1995)
Hirst, G., St-Onge, D.: Lexical chains as representations of context for the detection and correction of malapropisms. In: Wordnet: An Electronic Lexical Database, pp. 305–332. MIT Press, Cambridge (1998)
Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: MIR 2006: Proceedings of the Eighth ACM International Workshop on Multimedia Information Retrieval, Santa Barbara, CA, USA, pp. 321–330 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chan, C.H., Jones, G.J.F. (2011). An Affect-Based Video Retrieval System with Open Vocabulary Querying. In: Detyniecki, M., Knees, P., Nürnberger, A., Schedl, M., Stober, S. (eds) Adaptive Multimedia Retrieval. Context, Exploration, and Fusion. AMR 2010. Lecture Notes in Computer Science, vol 6817. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27169-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-642-27169-4_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-27168-7
Online ISBN: 978-3-642-27169-4
eBook Packages: Computer ScienceComputer Science (R0)