Multimedia Tools and Applications

, Volume 77, Issue 7, pp 9093–9110 | Cite as

Movie summarization using bullet screen comments

  • Shan Sun
  • Feng Wang
  • Liang He


Automatic movie summarization helps users to skim a movie in an efficient way. However, it is challenging because it requires the computer to automatically understand the movie content and the users’ opinions. Most previous works rely on the movie data itself without considering the opinions of the audience. In this paper, a novel approach for automatic movie summarization is presented by exploring a new type of user-generated data, i.e. bullet screen comments, which allow the audience to comment on the movie in a real-time manner. The number of the comments on a movie segment shows the exciting degree of the audience, while the content of the comments includes the concepts (e.g. the characters and the scenes) that interest the audience. In our approach, given a movie, bullet screen comments are utilized to select candidate highlight segments which are the most commented. Then the candidates are scored based on the number and the content of the bullet screen comments. Visual diversity is also considered in the scoring process. Finally, a subset of candidates which achieves the highest score is selected to compose a summary. Our experiments carried out on movies of different genres have shown the effectiveness of our proposed approach.


Movie summarization Multimedia content understanding Bullet screen comments User-generated data 



The work described in this paper was supported by the National Natural Science Foundation of China (No. 61375016) and the Science and Technology Commission of Shanghai Municipality (No. 16511102702).


  1. 1.
    Agnihotri L, Devera KV, McGee T, Dimitrova N (2001) Summarization of video programs based on closed captions. Storage Retriev Media Databases. doi: 10.1117/12.410973
  2. 2.
  3. 3.
  4. 4.
    Darabi K, Ghinea G (2016) User-centred personalised video abstraction approach adopting SIFT features. Multimed Tools Appl. doi: 10.1007/s11042-015-3210-4
  5. 5.
    Dimoulas CA, Symeonidis AL (2015) Syncing shared multimedia through audiovisual bimodal segmentation. IEEE MultiMed 22(3):26–42. doi: 10.1109/MMUL.2015.33 CrossRefGoogle Scholar
  6. 6.
    Dogra DP, Ahmed A, Bhaskar H (2015) Smart video summarization using mealy machine-based trajectory modelling for surveillance applications. Multimed Tools Appl 75(11):6373–6401. doi: 10.1007/s11042-015-2576-7 CrossRefGoogle Scholar
  7. 7.
    Doman K, Tomita T, Ide I, Deguchi D, Murase H (2014) Event detection based on twitter enthusiasm degree for generating a sports highlight video. In: Proceedings of the 18th ACM international conference on multimedia. doi: 10.1145/2647868.2654973
  8. 8.
    Ejaz N, Mehmood I, Baik SW (2013) Efficient visual attention based framework for extracting key frames from videos. Signal Process Image Commun 28(1):34–44. doi: 10.1016/j.image.2012.10.002 CrossRefGoogle Scholar
  9. 9.
    Ejaz N, Mehmood I, Baik SW (2014) Feature aggregation based visual attention model for video summarization. Comput Electric Eng 40(3):993–1005. doi: 10.1016/j.compeleceng.2013.10.005 CrossRefGoogle Scholar
  10. 10.
    Evangelopoulos G, Zlatintsi A, Potamianos A, Maragos P, Rapantzikos K, Skoumas G, Avrithis Y (2013) Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Trans Multimed 15(7):1553–1568. doi: 10.1109/TMM.2013.2267205 CrossRefGoogle Scholar
  11. 11.
    Ferreira L, Silva Cruz LA, Assuncao P (2015) A generic framework for optimal 2D/3D key-frame extraction driven by aggregated saliency maps. Signal Process Image Commun 39:98–110. doi: 10.1016/j.image.2015.09.005 CrossRefGoogle Scholar
  12. 12.
    Furini M, Ghini V (2006) An audio-video summarization scheme based on audio and video analysis. In: Proceedings of IEEE consumer communications & networking. doi: 10.1109/CCNC.2006.1593230
  13. 13.
    Hannon J, McCarthy K, Lynch J, Smyth B (2011) Personalized and automatic social summarization of events in video. In: Proceedings of the 16th international conference on intelligent user interfaces. doi: 10.1145/1943403.1943459
  14. 14.
    Li Y, Lee SH, Yeh CH, Kuo CC (2006) Techniques for movie content analysis and skimming: tutorial and overview on video abstraction techniques. IEEE Signal Process Mag 23(2):79–89. doi: 10.1109/MSP.2006.1621451 CrossRefGoogle Scholar
  15. 15.
    Lin W, Sun MT, Li H, Chen Z, Li W, Zhou B (2012) Macroblock classification for video applications involving motions. IEEE Trans Broadcast 58 (1):34–46. doi: 10.1109/TBC.2011.2170611 CrossRefGoogle Scholar
  16. 16.
    Lin W, Zhang Y, Lu J, Zhou B, Wang J, Zhou Y (2015) Summarizing surveillance videos with local-patch-learning-based abnormality detection, blob sequence optimization, and type-based synopsis. Neurocomputing 155:84–98. doi: 10.1016/j.neucom.2014.12.044 CrossRefGoogle Scholar
  17. 17.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the 7th IEEE international conference on computer vision. doi: 10.1109/ICCV.1999.790410
  18. 18.
    Lu Z, Grauman K (2013) Story-driven summarization for egocentric video. In: Proceedings of IEEE conference on computer vision and pattern recognition. doi: 10.1109/CVPR.2013.350
  19. 19.
    Ma YF, Hua XS, Lu L, Zhang HJ (2005) A generic framework of user attention model and its application in video summarization. IEEE Trans Multimed 7(5):907–919. doi: 10.1109/TMM.2005.854410 CrossRefGoogle Scholar
  20. 20.
    Mei S, Guan G, Wang Z, He M, Hua X, Feng D (2014) L 2,0 constrained sparse dictionary selection for video summarization. In: International conference on multimedia & Expo. doi: 10.1109/ICME.2014.6890179
  21. 21.
    Mei S, Guan G, Wang Z, Wan S, He M, Feng D (2015) Video summarization via minimum sparse reconstruction. Pattern Recogn 48(2):522–533. doi: 10.1016/j.patcog.2014.08.002 CrossRefGoogle Scholar
  22. 22.
  23. 23.
    Over P, Smeaton AF, Kelly P (2007) The TRECVID 2007 BBC rushes summarization evaluation pilot. In: Proceedings of the TRECVID workshop on video summarization. doi: 10.1145/1290031.1290032
  24. 24.
    Panagiotakis C, Doulamis A, Tziritas G (2009) Equivalent key frames selection based on iso-content principles. IEEE Trans Circ Syst Vid Technol 19(3):447–451. doi: 10.1109/TCSVT.2009.2013517 CrossRefGoogle Scholar
  25. 25.
    Peng J, Lin Q (2007) Keyframe-based video summary using visual attention clues. IEEE Multimed 17(2):64–73. doi: 10.1109/MMUL.2009.65 Google Scholar
  26. 26.
    Pickering MJ, Wong L, Rger SM (2003) ANSES: summarization of news video. Image Vid Retriev. doi: 10.1007/3-540-45113-7_42
  27. 27.
    Pritch Y, Rav-Acha A, Peleg S (2008) Nonchronological video synopsis and indexing. IEEE Trans Pattern Anal Mach Intell 30(11):1971–1984. doi: 10.1109/TPAMI.2008.29 CrossRefGoogle Scholar
  28. 28.
    Rapantzikos K, Evangelopoulos G, Maragos P, Avrithis YS (2007) An audio-visual saliency model for movie summarization. In: IEEE 9th Workshop on multimedia signal processing. doi: 10.1109/MMSP.2007.4412882
  29. 29.
    Sang J, Xu C (2010) Character-based movie summarization. In: Proceedings of the 18th ACM international conference on multimedia. doi: 10.1145/1873951.1874096
  30. 30.
    Shamma D, Kennedy L, Churchill E (2010) Summarizing media through short-messaging services. In: ACM Conference on computer supported cooperative work.Google Scholar
  31. 31.
    Song Y, Vallmitjana J, Stent A, Jaimes A (2015) TVSum: summarizing web videos using titles. In: Proceedings of the IEEE conference on computer vision and pttern recognition. doi: 10.1109/CVPR.2015.7299154
  32. 32.
    Taskiran CM, Pizlo Z, Amir A, Ponceleon D, Delp EJ (2006) Automated video program summarization using speech transcripts. IEEE Trans Multimed 8 (4):775–791. doi: 10.1109/TMM.2006.876282 CrossRefGoogle Scholar
  33. 33.
    Tian Z, Xue J, Lan X, Li C, Zheng N (2014) Object segmentation and key-pose based summarization for motion video. Multimed Tools Appl 72(2):1773–1802. doi: 10.1007/s11042-013-1488-7 CrossRefGoogle Scholar
  34. 34.
    Uchihashi S, Foote J, Girgenson A, Boreczky J (1999) Video manga: generating semantically meaningful video summaries. In: Proceedings of the 7th ACM international conference on multimedia. doi: 10.1145/319463.319654
  35. 35.
    Wang M, Hong R, Li G, Zha Z, Yan S, Chua T (2012) Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimed 14(4):975–985. doi: 10.1109/TMM.2012.2185041 CrossRefGoogle Scholar
  36. 36.
    Xiang X, Kankanhalli MS (2011) Affect-based adaptive presentation of home videos. In: Proceedings of the 19th ACM international conference on multimedia. doi: 10.1145/2072298.2072370
  37. 37.
    Yeung MM, Yeo BL (1997) Video visualization for compact presentation and fast browsing of pictorial content. IEEE Trans Circ Syst Vid Technol 7(5):771–785. doi: 10.1109/76.633496 CrossRefGoogle Scholar
  38. 38.
    Zhou H, Hermans T, Karandikar A, Rehg J (2010) Movie genre classification via scene categorization. In: Proceedings of ACM international conference on multimedia. doi: 10.1145/1873951.1874068

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Shanghai Key Laboratory of Multidimensional Information Processing, Dept. of Computer Science and TechnologyEast China Normal UniversityShanghaiChina

Personalised recommendations