Abstract
This paper reports on a user-centered evaluation of visual video summaries. We evaluated four types of summaries (fastforward, user-controlled fastforward, scene clips and storyboard) with a set of existing performance and satisfaction measures. We further conducted a repertory grid elicitation with our participants gathering evaluation constructs related to both video summary content and controls. Results showed a lack of correlation between performance and satisfaction measures. User-supplied evaluation constructs were shown to span both the performance and satisfaction dimensions of the video summary evaluation space. Most constructs achieved moderate to good inter-rater agreement in a consequent survey.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Christel, M.C.: Evaluation and user studies with respect to video summarization and browsing. In: Multimedia Content Analysis, Management and Retrieval, pp. 196–210 (2006)
Taskiran, C.M., Pizlo, Z., Amir, A., Ponceleon, D., Delp, E.J.: Automated video program summarization using speech transcripts. IEEE T. Multimedia 8, 775–791 (2006)
Marchionini, G.: Human performance measures for video retrieval. In: 8th ACM International Workshop on Multimedia Information Retrieval, pp. 307–312 (2006)
Truong, B.T., Venkatesh, S.: Video abstraction: A systematic review and classification. ACM Transactions on Multimedia Computing, Communications, and Applications 3 (2007)
Marchionini, G., Song, Y., Farrell, R.: Multimedia surrogates for video gisting: Towards combining spoken words and imagery. Inform. Process. Manag. 45, 615–630 (2009)
Over, P., Smeaton, A.F., Kelly, P.: The trecvid 2007 BBC rushes summarization evaluation pilot. In: TRECVID Workshop on Video Summarization, pp. 1–15 (2007)
Wildemuth, B.M., Marchionini, G., Yang, M., Geisler, G., Wilkens, T., Hughes, A., Gruss, R.: How Fast Is Too Fast? Evaluating Fast Forward Surrogates for Digital Video. In: 3rd ACM/IEEE-CS JCDL, pp. 221–230 (2003)
Corchs, S., Ciocca, G., Schettini, R.: Video summarization using a neurodynamical model of visual attention. In: IEEE 6th Workshop on Multimedia Signal Processing, pp. 71–74 (2004)
Ma, Y.-F., Lu, L., Zhang, H.-J., Li, M.: A user attention model for video summarization. In: ACM Multimedia, ACMMM (2002)
Guironnet, M., Pellerin, D., Guyader, N., Ladret, P.: Video Summarization Based on Camera Motion and a Subjective Evaluation Method. J. Image Video Processing (2007)
Marchionini, G., Wildemuth, B.M., Geisler, G.: The Open Video Digital Library: A Möbius strip of research and practice. J. Am. Soc. Inf. Sci. Technol. 57, 1629–1643 (2006)
Li, Y., Narayanan, S., Kuo, C.-C.J.: Movie Content Analysis, Indexing and Skimming via Multimodal Information. In: Rosenfeld, A., Doermann, D., Dementhon, D. (eds.) Video Mining. Kluwer Academic Publishers, Dordrecht (2003)
Tan, F.B., Hunter, M.G.: The repertory grid technique: A method for the study of cognition in information systems. MIS Quarterly 26, 39–57 (2002)
Oppenheim, C., Stenson, J., Wilson, R.M.S.: Studies on Information as an Asset II: Repertory Grid. J. Inform. Sci. 29, 419–432 (2003)
Dillon, A., McKnight, C.: Towards a classification of text types: a repertory grid approach. Int. J. Man-Mach. Stud. 33, 623–636 (1990)
Crudge, S.E., Johnson, F.C.: Using the repertory grid and laddering technique to determine the user’s evaluative model of search engines. J. Doc. 63, 259–280 (2004)
Yang, M., Wildemuth, B.M., Marchionini, G., Wilkens, T., Geisler, G., Hughes, A., Gruss, R., Webster, C.: Measures of User Performance in Video Retrieval Research. UNC (SILS) Technical Report TR-2003-02 (2003)
Yang, M., Marchionini, G.: Deciphering visual gist and its implications for video retrieval and interface design. In: ACM CHI, pp. 1877–1880 (2005)
Sundaram, H., Chang, S.-F.: Condensing computable scenes using visual complexity and film syntax analysis. In: IEEE ICME (2001)
Cohen, J.: Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70, 213–220 (1968)
Balatsoukas, P., Morris, A., O’Brien, A.: An evaluation framework of user interaction with metadata surrogates. J. Inform. Sci. 35, 321–339 (2009)
Goodrum, A.: Multidimensional scaling of video surrogates. J. Am. Soc. Inf. Sci. Technol. 52, 174–182 (2001)
Wildemuth, B.M., Russell, T., Ward, T.J., Marchionini, G., Oh, S.: The Influence of Context and Interactivity on Video Browsing. UNC (SILS) Technical Report TR 2006-01 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Westman, S. (2010). Evaluation Constructs for Visual Video Summaries. In: Lalmas, M., Jose, J., Rauber, A., Sebastiani, F., Frommholz, I. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2010. Lecture Notes in Computer Science, vol 6273. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15464-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-15464-5_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15463-8
Online ISBN: 978-3-642-15464-5
eBook Packages: Computer ScienceComputer Science (R0)