Assessing the Usability of Video Browsing and Summarization Techniques

Part of the Signals and Communication Technology book series (SCT)


Since 1994, the Informedia group at Carnegie Mellon University has been developing and evaluating surrogates, summary interfaces, and visualizations for accessing digital video collections containing thousands of documents, millions of shots, and terabytes of data. This chapter samples Informedia user studies that have taken place through the years, reporting on how these studies provide a user pull complementing the technology push as automated video processing advances. Specifically, this chapter examines indicative video summaries— i.e., the assessment of video surrogates meant to help users better judge the relevance of the source program for their task at hand. The merits of discount usability techniques for iterative improvement and evaluation are presented, as well as the structure of formal empirical investigations with end users that have ecological validity while addressing the human computer interaction metrics of efficiency, effectiveness, and satisfaction. Lessons learned from such studies are reported with respect to video summarization and browsing, ranging from the simplest portrayal of a single thumbnail to represent video stories, to collections of thumbnails in storyboards, to playable video skims, and to video collages with multiple synchronized information perspectives. Advances in interactive video retrieval are charted through the annual National Institiute for Standards Technology (NIST) TRECVID evaluation forum, concluding with discussion on difficulties in evaluating video summarization and browsing interfaces.


Mean Average Precision Information Visualization News Video Video Retrieval Video Summarization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahlberg C, Shneiderman B (1994) Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfield Displays. Proc. ACM CHI, 313–317Google Scholar
  2. 2.
    Boreczky J, Girgensohn A, Golovchinsky G, Uchihashi S (2000) An Interactive Comic Book Presentation for Exploring Video. Proc. ACM CHI, 185–192Google Scholar
  3. 3.
    Christel MG (2002) Accessing News Libraries through Dynamic Information Extraction, Summarization, and Visualization. In Börner K, Chen C (eds.) Visual Interfaces to Digital Libraries LNCS 2539. Springer-Verlag, BerlinGoogle Scholar
  4. 4.
    Christel MG (2006) Evaluation and User Studies with Respect to Video Summarization and Browsing. Proc. SPIE Conf. Multimedia Content Analysis, Management and Retrieval 6073:196–210Google Scholar
  5. 5.
    Christel MG (2007) Examining User Interactions with Video Retrieval Systems. Proc. SPIE Vol. 6506 Conf. Multimedia Content Analysis: Algorithms and Systems 6506-06Google Scholar
  6. 6.
    Christel MG (2007) Establishing the Utility of Non-Text Search for News Video Retrieval with Real World Users. Proc. ACM Multimedia, 706–717Google Scholar
  7. 7.
    Christel MG, Conescu R (2005) Addressing the Challenge of Visual Information Access from Digital Image and Video Libraries. Proc. Joint Conference on Digital Libraries, 69–78Google Scholar
  8. 8.
    Christel MG, Conescu R (2006) Mining Novice User Activity with TRECVID Interactive Retrieval Tasks. Proc. Conf. Image and Video Retrieval (LNCS 4071), 21–30Google Scholar
  9. 9.
    Christel MG, Hauptmann AG, et al. (2002) Collages as Dynamic Summaries for News Video. Proc. ACM Multimedia, 561–569Google Scholar
  10. 10.
    Christel MG, Martin D (1998) Information Visualization within a Digital Video Library. Journal of Intelligent Information Systems 11:235–257CrossRefGoogle Scholar
  11. 11.
    Christel MG, Moraveji N (2004) Finding the Right Shots: Assessing Usability and Performance of a Digital Video Library Interface. Proc. ACM Multimedia, 732–739Google Scholar
  12. 12.
    Christel MG, Smith MA, Taylor CR, Winkler DB (1998) Evolving Video Skims into Useful Multimedia Abstractions. Proc. ACM CHI, 171–178Google Scholar
  13. 13.
    Christel MG, Winkler D, Taylor, CR (1997) Improving Access to a Digital Video Library. In Howard S, Hammond J, Lindgaard G (eds.) Human-Computer Interaction: INTERACT97. Chapman and Hall, LondonGoogle Scholar
  14. 14.
    Christel MG, Warmack AS (2001) The Effect of Text in Storyboards for Video Navigation. Proc. IEEE Conf. Acoustics, Speech, and Signal Processing (ICASSP), III:1409–1412Google Scholar
  15. 15.
    Cox RV, et al. (1998) Applications of Multimedia Processing to Communications. Proc. of IEEE 86:754–824Google Scholar
  16. 16.
    Davis M, King S, Good N, Sarvas R (2004) From Context to Content: Leveraging Context to Infer Media Metadata. Proc. ACM Multimedia, 188–195Google Scholar
  17. 17.
    Ding W, Marchionini G, Soergel D (1999) Multimodal Surrogates for Video Browsing. Proc. ACM Digital Libraries, 85–93Google Scholar
  18. 18.
    Dumas JF, Redish JC (1999) A Practical Guide to Usability Testing, rev. ed. Intellect Books, Exeter, UKGoogle Scholar
  19. 19.
    Ekin A, Tekalp AM, Mehrotra, R (2003) Automatic Soccer Video Analysis and Summarization. IEEE Trans. on Image Processing 12:796–807CrossRefGoogle Scholar
  20. 20.
    Enser PGB, Sandom CJ (2002) Retrieval of Archival Moving Imagery - CBIR Outside the Frame? Proc. Conf. Image and Video Retrieval, 206–214Google Scholar
  21. 21.
    Ferman AM, Tekalp AM (2003) Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences. IEEE Trans. Multimedia 5:244–256CrossRefGoogle Scholar
  22. 22.
    Foraker Design (2007) Usability in Website and Software Design. Accessed 15 November 2007
  23. 23.
    Frøkjær E, Hertzum M, Hornbæk K (2000) Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proc. ACM CHI, 345–352Google Scholar
  24. 24.
    Girgensohn A, Adcock J, el al. (2005) Interactive Search in Large Video Collections. CHI '05 Extended Abstracts on Human Factors in Computing Systems, 1395–1398Google Scholar
  25. 25.
    Hart PE, Piersol K, Hull, JJ (2005) Refocusing Multimedia Research on Short Clips. IEEE MultiMedia, 12:8–13CrossRefGoogle Scholar
  26. 26.
    Hauptmann AG (2005) Lessons for the Future from a Decade of Informedia Video Analysis Research. Proc. Conf. Image and Video Retrieval (LNCS 3568), 1–10Google Scholar
  27. 27.
    Hauptmann AG, Christel MG (2004) Successful Approaches in the TREC Video Retrieval Evaluations. Proc. ACM Multimedia, 668–675Google Scholar
  28. 28.
    Hauptmann AG, Christel MG, et al. (2007) Clever Clustering vs. Simple Speed-up for Summarizing Rushes. Proc. ACM Workshop TRECVID Video Summarization, 20–24Google Scholar
  29. 29.
    Hearst MA (1999) User Interfaces and Visualization. In Baeza-Yates R, Ribeiro-Neto, B (eds.), Modern Information Retrieval, Addison Wesley/ACM Press, New YorkGoogle Scholar
  30. 30.
    Hearst, MA, et al. (2002) Finding the flow in web site search. Comm. ACM, 45:42–49CrossRefGoogle Scholar
  31. 31.
    Heesch D, Rüger S (2004) Three interfaces for content-based access to image collections. Proc. Conf. Image and Video Retrieval (LNCS 3115), 491–499Google Scholar
  32. 32.
    Hollink L, Nguyen GP, et al. (2005) Assessing User Behaviour in News Video Retrieval. IEE Proc. Vision, Image, & Signal Processing 152:911–918Google Scholar
  33. 33.
    Kobsa A (2001) An Empirical Comparison of Three Commercial Information Visualization Systems. Proc. IEEE InfoVis, 123–130Google Scholar
  34. 34.
    Komlodi A, Marchionini G (1998) Key Frame Preview Techniques for Video Browsing. Proc. ACM Digital Libraries, 118–125Google Scholar
  35. 35.
    Lee H, Smeaton AF (2002) Designing the User Interface for the Físchlár Digital Video Library, Journal of Digital Information 2,
  36. 36.
    Lienhart R, Pfeiffer S, Effelsberg W (1997) Video Abstracting. Comm. ACM 40:54–62Google Scholar
  37. 37.
    Naphade M, Smith JR, et al. (2006) Large-Scale Concept Ontology for Multimedia. IEEE MultiMedia 13:86–91CrossRefGoogle Scholar
  38. 38.
    Nielsen J (1992) Evaluating the Thinking Aloud Technique for Use by Computer Scientists. In Hartson HR, Hix D (eds.) Advances in Human-Computer Interaction, Vol. 3. Ablex, Norwood, NJGoogle Scholar
  39. 39.
    Nielsen J (1994) Heuristic Evaluation. In Nielsen, J., and Mack, R.L. (eds.), Usability Inspection Methods. John Wiley and Sons, New York, NYCrossRefGoogle Scholar
  40. 40.
    Nielsen J, Clemmensen T, Yssing C (2002) Getting Access to What Goes on in People’s Heads? Reflections on the Think-Aloud Technique. Proc. ACM Nordic CHI, 101–110Google Scholar
  41. 41.
    Nielsen J, Molich R (1990) Heuristic Evaluation of User Interfaces. Proc. ACM CHI, 249–256Google Scholar
  42. 42.
    NIST TREC Video Retrieval Evaluation Home Page (2007) Accessed 15 November 2007
  43. 43.
    Over P, Smeaton AF, Kelly P (2007) The TRECVID 2007 BBC Rushes Summarization Evaluation Pilot. Proc. ACM Workshop on TRECVID Video Summarization, 1–15Google Scholar
  44. 44.
    Plaisant C (2004) The Challenge of Information Visualization Evaluation. Proc. ACM Advanced Visual Interfaces, 109–116Google Scholar
  45. 45.
    Ponceleon D, Srinivasan S, et al. (1998) Key to Effective Video Retrieval: Effective Cataloging and Browsing. Proc. ACM Multimedia, 99–107Google Scholar
  46. 46.
    Rowe LA, Jain R (2005) ACM SIGMM retreat report on future directions in multimedia research. ACM Trans. on Multimedia Computing, Communications, & Applications 1:3–13CrossRefGoogle Scholar
  47. 47.
    Shatford S (1986) Analyzing the Subject of a Picture: A Theoretical Approach. Cataloguing and Classification Quarterly 6:39–62CrossRefGoogle Scholar
  48. 48.
    Shneiderman B, Plaisant C (2006) Strategies for Evaluating Information Visualization Tools: Multi-dimensional In-depth Long-term Case Studies. Proc. ACM AVI Workshop on BEyond time and errors: novel evaluation methods for info. visualization, 1–7Google Scholar
  49. 49.
    Snoek C, Worring M, Koelma D, Smeulders A (2007) A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval. IEEE Trans. Multimedia 9:280–292CrossRefGoogle Scholar
  50. 50.
    Taskiran CM, Pizlo Z, Amir A, Ponceleon D, Delp EJ (2006) Automated Video Program Summarization Using Speech Transcripts. IEEE Trans. on Multimedia 8:775–791CrossRefGoogle Scholar
  51. 51.
    Truong BT, Venkatesh S (2006) Video Abstraction: A Systematic Review and Classification. ACM Trans. Multimedia Computing, Comm., & Applications 3:1–37Google Scholar
  52. 52.
    Uchihashi S, Foote J, et al. (1999) Video Manga: Generating Semantically Meaningful Video Summaries. Proc. ACM Multimedia, 383–392Google Scholar
  53. 53.
    Wactlar H, Christel MG, et al. (1999) Lessons Learned from the Creation and Deployment of a Terabyte Digital Video Library. IEEE Computer 32:66–73CrossRefGoogle Scholar
  54. 54.
    Worring M, Snoek C, et al. (2006) Mediamill: Advanced Browsing in News Video Archives. Proc. Conf. Image and Video Retrieval (LNCS 4071), 533–536Google Scholar
  55. 55.
    Xing L, Yu H., Huang Q, Ye Q, Divakaran A (2006) Subjective Evaluation Criterion for Selecting Affective Features and Modeling Highlights. Proc. SPIE Conf. Multimedia Content Analysis, Management and Retrieval 6073:188–195Google Scholar
  56. 56.
    Yang M, Wildemuth B, Marchionini G (2004) The Relative Effectiveness of Concept-based Versus Content-based Video Retrieval. Proc. ACM Multimedia, 368–371Google Scholar
  57. 57.
    Yeo B-L, Yeung, MM (1997) Retrieving and Visualizing Video. Comm. ACM 40:43–52CrossRefGoogle Scholar
  58. 58.
    Zhang H-J, et al. (1995) Video Parsing and Browsing Using Compressed Data. Multimedia Tools and Applications 1:89–111CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Senior Systems Scientist, Computer Science DeptCarnegie Mellon UniversityPittsburgh

Personalised recommendations