Advertisement

Multimedia Tools and Applications

, Volume 76, Issue 2, pp 2353–2378 | Cite as

User-centred personalised video abstraction approach adopting SIFT features

  • Kaveh Darabi
  • Gheorghita Ghinea
Article

Abstract

The rapid growth of digital video content in recent years has imposed the need for the development of technologies with the capability to produce condensed but semantically rich versions of original input video. Consequently, the topic of Video Summarisation is becoming increasingly popular in the multimedia community and numerous video abstraction approaches have been proposed. Creating personalised video summaries remains a challenge, though. Accordingly, in this paper we propose a methodology for generating user-tailored video abstracts. First, video frames are scored by a group of video experts (operators) according to audio, visual and textual content of the video. Later, SIFT visual features are adopted in our proposed approach to identify the video scenes’ semantic categories. Fusing this retrieved data with pre-built users’ profiles will provide a metric to update the previously averaged saliency scores assigned by video experts to each frame in accordance to users’ priorities. In the next stage, the initial averaged scores of the frames are updated based on the end-users’ generated profiles. Eventually, the highest scored video frames alongside the auditory and textual content are inserted into final digest Experimental results showed the effectiveness of this method in delivering superior outcomes comparing to our previously recommended algorithm and the three other automatic summarisation techniques.

Keywords

Video summarization SIFT Personalization Saliency score Relevancy level 

References

  1. 1.
    Almeida J, Leite LJ, Torres RS (2012) VISON: VIdeo Summarization for ONline applications. Pattern Recogn Lett 33(4):397–409CrossRefGoogle Scholar
  2. 2.
    Beom M, Williem L, Park I (2013) Spatiotemporal saliency-based video summarization on a smartphone. J Broadcast Eng 18(2):185–195CrossRefGoogle Scholar
  3. 3.
    Bhatt RB, Krishnamoorthy P, Kumar S (2009) Efficient general genre video abstraction scheme for embedded devices using pure audio cues, Proc. 7th International Conference on ICT and Knowledge Engineering, Bangkok, pp 63–67Google Scholar
  4. 4.
    Carvajal J, McCool C, Sanderson C (2014) Summarisation of short-term and long-term videos using texture and colour, Proc. IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, Colorado, pp 769–775Google Scholar
  5. 5.
    Chen F, De Vleeschouwer C, Cavallaro A (2014) Resource allocation for personalized video summarization. IEEE Trans Multimed 16(2):455–469CrossRefGoogle Scholar
  6. 6.
    Darabi K, Ghinea G (2014) Video summarization by group scoring, Proc. IEEE International Conference on Multimedia Computing and Systems, Marrakech, pp 112–116Google Scholar
  7. 7.
    Darabi K, Ghinea G (2014) Personalized video summarization based on group scoring, Proc. China Summit & International Conference on Signal and Information Processing, Xi’an, pp 310–314Google Scholar
  8. 8.
    Darabi K, Ghinea G (2014) Personalized video summarization by highest quality frames, Proc. IEEE International Conference on Multimedia and Expo Workshops, Chengdu, pp 1–6Google Scholar
  9. 9.
    Datta R, Li J, Wang J (2005) Content-based image retrieval—Approaches and trends of the new age, Proc. ACM Multimedia Workshop Multimedia Information Retrieval, Singapore, pp 253–262Google Scholar
  10. 10.
    Evangelopoulos G, Zlatintsi A, Potamianos A, Maragos P, Rapantzikos K, Skoumas G, Avrithis Y (2013) Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Trans Multimed 15(7):1553–1568CrossRefGoogle Scholar
  11. 11.
    Fukumura S, Nakano T, Harumoto K, Shimojo S, Nishio S (2003) Realization of personalized presentation for digital contents based on browsing history. IEEE Pac Rim Conf Commun Comput Signal Process 2:605–608Google Scholar
  12. 12.
    Ghinea G, Kannan R, Swaminathan S, Kannaiyan S (2014) A novel user-centered design for personalized video summarization, Proc. IEEE International Conference on Multimedia and Expo Workshops, Chengdu, pp 1–6Google Scholar
  13. 13.
    Guo Y, Zhu Y, Liu F, Song C, Zhou H (2012) Multi-view video summarisation. IEEE Multimedia 12(7):717–729Google Scholar
  14. 14.
    Han B, Hamm J, Sim J (2011) Personalised video summarization with human in the loop, Proc. IEEE Workshop on Applications of Computer Vision, Hawaii, pp 51–57Google Scholar
  15. 15.
    Han J, Li K, Shao L, Hua X, He S, Guo L, Han J, Liu T (2014) Video abstraction based on FMRI-driven visual attention model. Inf Sci 281:781–796CrossRefGoogle Scholar
  16. 16.
    Harel J, Koch C, Perona P (2007) Graph-based visual saliency. Adv Neural Inf Process Syst 19:545–552Google Scholar
  17. 17.
    Hari R, Roopesh CP, Wilscy M (2013) Human face based approach for video summarization, Proc. IEEE Recent Advances in Intelligent Computational Systems, Trivandrum, pp 245–250Google Scholar
  18. 18.
    Hays J, Efros AA (2007) Scene completion using millions of photographs. ACM Trans Graph 26(3):4CrossRefGoogle Scholar
  19. 19.
    Hopfgartner F, Jose JM, Yu Z, Lugmayr A, Chorianopoulos K, Mei T (2010) Semantic user profiling techniques for personalised multimedia recommendation. Multimedia Systems 16(5):255–274CrossRefGoogle Scholar
  20. 20.
    http://www.avcutty.de/english/. Accessed 25 Mar 2014
  21. 21.
    http://www.image-net.org. Accessed 28 Apr 2014
  22. 22.
    Jaimes A, Echigo T, Teraguchi M, Satoh F (2002) Learning personalized video highlights from detailed MPEG-7 metadata, Proc. International Conference on Image Processing, New York, 1, pp I-133–I-136Google Scholar
  23. 23.
    Khosla A, Hamid R, Lin CJ, Sundaresan N (2013) Large-scale video summarization using web-image priors, Proc. IEEE Conference on Computer Vision and Pattern Recognition, Portland, pp 2698–2705Google Scholar
  24. 24.
    Li X, Chen L, Zhang L, Lin F, Ma W (2006) Image annotation by large-scale content-based image retrieval, Proc. ACM International Conference on Multimedia, Santa Barbara, pp 607–610Google Scholar
  25. 25.
    Li J, Wang J (2003) Automatic linguistic indexing of pictures by statistical modelling approach. IEEE Trans Pattern Anal Mach Intell 25(9):1075–1088CrossRefGoogle Scholar
  26. 26.
    Liu Y, Liu H, Liu Y, Sun F (2014a) User-generated-video summarization using sparse modelling, Proc. International Joint Conference on Neural Networks, Beijing, pp 3909–3915Google Scholar
  27. 27.
    Liu Y, Liu H, Liu Y, Sun F (2014b) Outlier-attenuating summarization for user-generated-video, Proc. IEEE International Conference on Multimedia and Expo, Chengdu, IEEE, pp 1–6Google Scholar
  28. 28.
    Lu S, King I, Lyu MR. (2004) Video summarisation by video structure analysis and graph optimization, Proc. IEEE International Conference on Multimedia and Expo, Taipei, pp 1959–1962Google Scholar
  29. 29.
    Lu S, Wang Z, Mei T, Guan G, Feng D (2014) A bag-of-importance model with locality-constrained coding based feature learning for video summarization. IEEE Trans Multimed 16(6):1497–1509CrossRefGoogle Scholar
  30. 30.
    Mahmoud KM, Ghanem NM, Ismail MA (2013) VGRAPH: an effective approach for generating static video summaries, Proc. IEEE International Conference on Computer Vision Workshops, Sydney, pp 811–818Google Scholar
  31. 31.
    Mei S, Guan G, Wang Z, He M, Hua X, Dagan FD (2014) L2,0 constrained sparse dictionary selection for video summarization, Proc. IEEE International Conference on Multimedia and Expo, Chengdu, pp 1–6Google Scholar
  32. 32.
    Mobasher B, Cooley R, Siravstava J (2000) Automatic personalization based on Web usage mining. Commun ACM 43(8):142–151CrossRefGoogle Scholar
  33. 33.
    Money AG, Agius H (2007) Video summarisation: a conceptual framework and survey of the state of the art. J Vis Commun Image Represent 19(2):121–143CrossRefGoogle Scholar
  34. 34.
    Money AG, Agius H (2009) Analysing user physiological responses for affective video summarisation. Displays 30(2):59–70CrossRefGoogle Scholar
  35. 35.
    Ngo C, Ma Y, Zhang HJ (2005) Video summarisation and scene detection by graph modelling. IEEE Trans Circ Syst Video Technol 15(2):296–305CrossRefGoogle Scholar
  36. 36.
    Park HS, Cho SB (2011) A personalized summarization of video life-logs from an indoor multi-camera system using a fuzzy rule-based system with domain knowledge. Inf Syst 36(8):1124–1134CrossRefGoogle Scholar
  37. 37.
    Potapov D, Douze M, Harchaoui Z, Schmid C (2014) Category-specific video summarization, Proc. European Conference on Computer Vision, Zurich, pp 540–555Google Scholar
  38. 38.
    Takahashi Y, Nitta N, Babaguchi N (2005a) Video Summarisation for large sports video archives, Proc. IEEE International Conference on Multimedia and Expo, Amsterdam, pp 1170–1173Google Scholar
  39. 39.
    Takahashi Y, Nitta N, Babaguchi N (2005) Automatic video summarization of sports videos using metadata. Adv Multimed Inf Process 3332:272–280Google Scholar
  40. 40.
    Tropp JA, Gilbert AC, Strauss MJ (2006) Algorithms for simultaneous sparse approximation, part I: greedy pursuit. Signal Process 86(3):572–588CrossRefMATHGoogle Scholar
  41. 41.
    Wu S, Thawonmas R, Chen K (2011) Video summarization via crowdsourcing, Proc. CHI ‘11 Extended Abstracts on Human Factors in Computing Systems, Vancouver, pp 1531–1536Google Scholar
  42. 42.
    Xu C, Zhang YF, Zhu G, Rui Y, Lu H, Huang Q (2008) Using webcast text for semantic event detection in broadcast sports video. IEEE Trans Multimed 10(7):1342–1355CrossRefGoogle Scholar
  43. 43.
    You J, Hannuksela M, Gabbouj M (2009) Semantic audio-visual analysis for video summarisation, Proc. IEEE Region 8 EUROCON 2009 Conference, pp 1358–1363Google Scholar
  44. 44.
    Zhao S, Yao H, Sun X, Jiang X, Xu P (2013) Flexible presentation of videos based on affective content analysis. Adv Multimed Model 7732:368–379CrossRefGoogle Scholar
  45. 45.
    Zimmerman DW (1997) A note on interpretation of the paired-samples t test. J Educ Behav Stat 22(3):349–360Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceBrunel UniversityLondonUK

Personalised recommendations