OPFSumm: on the video summarization using Optimum-Path Forest

  • Guilherme B. Martins
  • Danillo R. Pereira
  • Jurandy G. Almeida
  • Victor Hugo C. de Albuquerque
  • João Paulo Papa
Article
  • 19 Downloads

Abstract

Video summarization attempts at encoding a given video into a compact representation for a better storage and retrieval purposes. This work copes with the problem of static video summarization using the unsupervised Optimum-Path Forest (OPF). We sampled the encoded video sequence into frames and extracted features based on color information or spectral properties. After that, meaningless frames are removed, and OPF models the problem of video summarization as a clustering process. Possible redundant keyframes are filtered, and at last the video summary is created based on non-redundant keyframes. We presented a more in-depth study that also considers temporal information to obtain better video representations. The experiments over three public datasets were analyzed through F-measure evaluation metric and showed the robustness of OPF for automatic video summarization: 0.19 for SumMe dataset, 0.728 concerning Open Video dataset, and 0.451 regarding YouTube dataset..

Keywords

Video summarization Optimum-path forest OPFSumm Multimedia tools 

Notes

Acknowledgments

The authors acknowledge “Coordination for the Improvement of Higher Education Personnel”, “São Paulo Research Foundation” grants 2013/07375-0, 2014/16250-9, 2014/12236-1, 2016/06441-7, and 2016/19403-6, and “National Council for Scientific and Technological Development” grants 306166/2014-3, 423228/2016-1, 304315/2017-6, and 307066/2017-7).

References

  1. 1.
    Almeida J, Leite NJ, Torres RS (2012) VISON: VIdeo Summarization for ONline applications. Pattern Recogn Lett 33(4):397–409CrossRefGoogle Scholar
  2. 2.
    Almeida J, Leite NJ, Torres RS (2013) Online video summarization on compressed domain. J Vis Commun Image Represent 24(6):729–738CrossRefGoogle Scholar
  3. 3.
    Amorim WP, Falcão AX, Papa JP, Carvalho MH (2016) Improving semi-supervised learning through optimum connectivity. Pattern Recogn 60:72–85CrossRefGoogle Scholar
  4. 4.
    Avila SEF, Lopes APB, Luz A Jr, Araújo AA (2011) VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recogn Lett 32(1):56–68CrossRefGoogle Scholar
  5. 5.
    Castelo-Fernández C, Calderón-Ruiz G (2015) Progress in pattern recognition, image analysis, computer vision, and applications. In: Pardo A, Kittler J (eds) 20th Iberoamerican congress on pattern recognition, lecture notes in computer science, vol 9423, chap. Automatic video summarization using the optimum-path forest unsupervised classifier. Springer International Publishing, New York, pp 760–767Google Scholar
  6. 6.
    Choi YS, Kim KJ (2004) In: Laganá A, Gavrilova ML, Kumar V, Mun YS, Tan CJK, Gervasi O (eds) International conference on computational science and its applications. Lecture notes in computer science, vol 3043, chap. Video summarization using fuzzy one-class support vector machine. Springer, Berlin, pp 49–56Google Scholar
  7. 7.
    DeMenthon D, Kobla V, Doermann DS (1998) Video summarization by curve simplification. In: ACM international conference multimedia (MM’08), pp 211–218Google Scholar
  8. 8.
    Furini M, Geraci F, Montangero M, Pellegrini M (2010) STIMO: STIll and MOving video storyboard for the web scenario. Multimed Tools Appl 46(1):47–69CrossRefGoogle Scholar
  9. 9.
    Gong B, Chao WL, Grauman K, Sha F (2014) Diverse sequential subset selection for supervised video summarization. In: Proceedings of the 27th international conference on neural information processing systems, NIPS’14. MIT Press, Cambridge, pp 2069–2077. http://dl.acm.org/citation.cfm?id=2969033.2969058
  10. 10.
    Gygli M, Grabner H, Riemenschneider H, Van Gool L (2014) Creating summaries from user videos. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) 13th European conference on computer vision. Springer International Publishing, pp 505–520Google Scholar
  11. 11.
    Gygli M, Grabner H, Van Gool L (2015) Video summarization by learning submodular mixtures of objectives. In: CVPRGoogle Scholar
  12. 12.
    Huang J, Kumar R, Mitra M, Zhu WJ, Zabih R (1997) Image indexing using color correlograms. In: IEEE international conference computer vision and pattern recognition (CVPR’97), pp 762–768Google Scholar
  13. 13.
    Jacob H, Pádua FLC, Lacerda A, Pereira ACM (2017) A video summarization approach based on the emulation of bottom-up mechanisms of visual attention. J Intell Inf Syst 1–19.  https://doi.org/10.1007/s10844-016-0441-4
  14. 14.
    Jacobs CE, Finkelstein A, Salesin D (1995) Fast multiresolution image querying. In: International conference on computer graphics and interactive techniques (SIGGRAPH’95), pp 277–286Google Scholar
  15. 15.
    Mahasseni B, Lam M, Todorovic S (2017) Unsupervised video summarization with adversarial lstm networks. In: The IEEE conference on computer vision and pattern recognition (CVPR). IEEEGoogle Scholar
  16. 16.
    Martins GB, Afonso LCS, Osaku D, Almeida JG, Papa JP (2014) Progress in pattern recognition, image analysis, computer vision, and applications. In: Bayro-Corrochano E, Hancock E (eds) Lecture Notes in Computer Science, vol. 8827, chap. Static Video Summarization through Optimum-Path Forest Clustering, pp 893–900. Springer International Publishing. 19th Iberoamerican congress on pattern recognitionGoogle Scholar
  17. 17.
    Martins GB, Papa JP, Almeida J (2016) Temporal-and spatial-driven video summarization using optimum-path forest. In: 29th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pp 335–339Google Scholar
  18. 18.
    Minetto R, Spina TV, Falcão AX, Leite NJ, Papa JP, Stolfi J (2012) IFTrace: video segmentation of deformable objects using the image foresting transform. Comput Vis Image Underst 116(2):274–291CrossRefGoogle Scholar
  19. 19.
    Money AG, Agius HW (2008) Video summarization: a conceptual framework and survey of the state of the art. J Vis Commun Image Represent 19(2):121–143CrossRefGoogle Scholar
  20. 20.
    Mundur P, Rao Y, Yesha Y (2006) Keyframe-based video summarization using Delaunay clustering. Int J Digit Libr 6(2):219–232CrossRefGoogle Scholar
  21. 21.
    Panda R, Kuanar SK, Chowdhury AS (2014) Scalable video summarization using skeleton graph and random walk. In: Proceedings of the 2014 22nd international conference on pattern recognition, ICPR ’14. IEEE Computer Society, Washington, DC, pp 3481–3486Google Scholar
  22. 22.
    Papa JP, Falcão AX (2008) A new variant of the optimum-path forest classifier. In: Bebis G, Boyle R, Parvin B, Koracin D, Remagnino P, Porikli F, Peters J, Klosowski J, Arns L, Chun Y, Rhyne TM, Monroe L (eds) Advances in visual computing, lecture notes in computer science, vol 5358. Springer, Berlin, pp 935–944Google Scholar
  23. 23.
    Papa JP, Falcão AX (2009) A learning algorithm for the optimum-path forest classifier. In: Torsello A, Escolano F, Brun L (eds) Graph-based representations in pattern recognition, lecture notes in computer science, vol 5534. Springer, Berlin, pp 195–204Google Scholar
  24. 24.
    Papa JP, Falcão AX, Suzuki CTN (2009) Supervised pattern classification based on optimum-path forest. Int J Imaging Syst Technol 19(2):120–131CrossRefGoogle Scholar
  25. 25.
    Papa JP, Falcão AX, Albuquerque VHC, Tavares JMRS (2012) Efficient supervised optimum-path forest classification for large datasets. Pattern Recogn 45 (1):512–520CrossRefGoogle Scholar
  26. 26.
    Papa JP, Scheirer W, Cox DD (2016) Fine-tuning deep belief networks using harmony search. Appl Soft Comput 46:875–885CrossRefGoogle Scholar
  27. 27.
    Papa JP, Fernandes SEN, Falcão AXs (2017) Optimum-path forest based on k-connectivity: theory and applications. Pattern Recogn Lett 87:117–126CrossRefGoogle Scholar
  28. 28.
    Papadopoulos DP, Chatzichristofis AA, Papamarkos N (2011) 5th international conference on computer vision/computer graphics collaboration techniques. chap. Video summarization using a self-growing and self-organized neural gas network. Springer, Berlin, pp 216–226Google Scholar
  29. 29.
    Pass G, Zabih R, Miller J (1996) Comparing images using color coherence vectors. In: ACM international conference on multimedia, pp 65–73Google Scholar
  30. 30.
    Pisani RJ, Nakamura RYM, Riedel PS, Zimback CRL, Falcão AX, Papa JP (2014) Toward satellite-based land cover classification through optimum-path forest. IEEE Trans Geosci Remote Sens 52(10):6075–6085CrossRefGoogle Scholar
  31. 31.
    Rocha LM, Cappabianco FAM, Falcão AX (2009) Data clustering as an optimum-path forest problem with applications in image analysis. Int J Imaging Syst Technol 19(2):50–68CrossRefGoogle Scholar
  32. 32.
    Rosa GH, Costa KAP, Passos LA Jr, Papa JP, Falcão AX, Tavares JMRS (2014) On the training of artificial neural networks with radial basis function using optimum-path forest clustering. In: 22nd international conference on pattern recognition, pp 1472–1477Google Scholar
  33. 33.
    Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905CrossRefGoogle Scholar
  34. 34.
    Souza GB, Santos DFS, Pires RG, Marana AN, Papa JP (2017) Deep boltzmann machines for robust fingerprint spoofing attack detection. In: Proceedings of the international joint conference on neural networks, pp 1863–1870Google Scholar
  35. 35.
    Stehling RO, Nascimento MA, Falcão AX (2002) A compact and efficient image retrieval approach based on border/interior pixel classification. In: ACM international conference on information and knowledge management (CIKM’02), pp 102–109Google Scholar
  36. 36.
    Sun K, Zhu J, Lei Z, Hou X, Zhang Q, Duan J, Qiu G (2017) Learning deep semantic attributes for user video summarization. In: The IEEE international conference on multimedia and expo (ICME). IEEEGoogle Scholar
  37. 37.
    Swain MJ, Ballard BH (1991) Color indexing. Int J Comput Vis 7(1):11–32CrossRefGoogle Scholar
  38. 38.
    Zhang D, Lu G (2002) Shape-based image retrieval using generic fourier descriptor. Signal Process Image Commun 17(10):825–848CrossRefGoogle Scholar
  39. 39.
    Zhang L, Xia Y, Mao K, Ma H, Shan Z (2015) An effective video summarization framework toward handheld devices. IEEE Trans Ind Electron 62 (2):1309–1316CrossRefGoogle Scholar
  40. 40.
    Zhang K, Chao WL, Sha F, Grauman K (2016) Summary transfer: exemplar-based subset selection for video summarization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1059–1067Google Scholar
  41. 41.
    Zhang K, Chao WL, Sha F, Grauman K (2016) Video summarization with long short-term memory. Springer International Publishing, Berlin, pp 766–782Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of ComputingSão Paulo State UniversityBauruBrazil
  2. 2.Institute of Science and TechnologyFederal University of São PauloSão PauloBrazil
  3. 3.Graduate Program in Applied InformaticsUniversity of FortalezaFortalezaBrazil

Personalised recommendations