Advertisement

Semantic Context Detection Using Audio Event Fusion

  • Wei-Ta ChuEmail author
  • Wen-Huang Cheng
  • Ja-Ling Wu
Open Access
Research Article
Part of the following topical collections:
  1. Information Mining from Multimedia Databases

Abstract

Semantic-level content analysis is a crucial issue in achieving efficient content retrieval and management. We propose a hierarchical approach that models audio events over a time series in order to accomplish semantic context detection. Two levels of modeling, audio event and semantic context modeling, are devised to bridge the gap between physical audio features and semantic concepts. In this work, hidden Markov models (HMMs) are used to model four representative audio events, that is, gunshot, explosion, engine, and car braking, in action movies. At the semantic context level, generative (ergodic hidden Markov model) and discriminative (support vector machine (SVM)) approaches are investigated to fuse the characteristics and correlations among audio events, which provide cues for detecting gunplay and car-chasing scenes. The experimental results demonstrate the effectiveness of the proposed approaches and provide a preliminary framework for information mining by using audio characteristics.

Keywords

Support Vector Machine Content Analysis Markov Model Hide Markov Model Quantum Information 

References

  1. 1.
    Lienhart RW: Comparison of automatic shot boundary detection algorithms. Storage and Retrieval for Image and Video Databases VII, January 1999, San Jose, Calif, USA, Proceedings of SPIE 3656: 290–301.CrossRefGoogle Scholar
  2. 2.
    Hanjalic A: Shot-boundary detection: unraveled and resolved? IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(2):90–105. 10.1109/76.988656CrossRefGoogle Scholar
  3. 3.
    Chang S-F, Vetro A: Video adaptation: concepts, technologies, and open issues. Proceedings of the IEEE 2005, 93(1):148–158.CrossRefGoogle Scholar
  4. 4.
    Lu L, Zhang H-J, Jiang H: Content analysis for audio classification and segmentation. IEEE Transactions Speech Audio Processing 2002, 10(7):504–516. 10.1109/TSA.2002.804546CrossRefGoogle Scholar
  5. 5.
    Zhang T, Jay Kuo C-C: Hierarchical system for content-based audio classification and retrieval. Multimedia Storage and Archiving Systems III, November 1998, Boston, Mass, USA, Proceedings of SPIE 3527: 398–409.CrossRefGoogle Scholar
  6. 6.
    Tzanetakis G, Cook P: Musical genre classification of audio signals. IEEE Transactions Speech Audio Processing 2002, 10(5):293–302. 10.1109/TSA.2002.800560CrossRefGoogle Scholar
  7. 7.
    Lu L, Zhang H-J: Automated extraction of music snippets. Proc. 11th ACM International Conference on Multimedia, November 2003, Berkeley, Calif, USA 140–147.Google Scholar
  8. 8.
    Fischer S, Lienhart R, Effelsberg W: Automatic recognition of film genres. Proc. 3rd ACM International Conference on Multimedia, November 1995, San Francisco, Calif, USA 295–304.Google Scholar
  9. 9.
    Liu Z, Huang J, Wang Y: Classification of TV programs based on audio information using hidden Markov model. Proc. IEEE 2nd Workshop on Multimedia Signal Processing (MMSP '98), December 1998, Redonda Beach, Calif, USA 27–31.Google Scholar
  10. 10.
    Wang Y, Liu Z, Huang J-C: Multimedia content analysis-using both audio and visual clues. IEEE Signal Processing Magazine 2000, 17(6):12–36. 10.1109/79.888862CrossRefGoogle Scholar
  11. 11.
    Zettl H: Sight Sound Motion: Applied Media Aesthetics. Wadsworth, Belmont, Calif, USA; 1999.Google Scholar
  12. 12.
    Dorai C, Venkatesh S: Media Computing: Computational Media Aesthetics. Kluwer Academic, Boston, Mass, USA; 2002.CrossRefGoogle Scholar
  13. 13.
    Moncrieff S, Venkatesh S, Dorai C: Horror film genre typing and scene labeling via audio analysis. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 193–196.Google Scholar
  14. 14.
    Cai R, Lu L, Zhang H-J, Cai L-H: Highlight sound effects detection in audio stream. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 3: 37–40.Google Scholar
  15. 15.
    Naphade MR, Kristjansson T, Frey B, Huang TS: Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems. Proc. International Conference on Image Processing (ICIP '98), October 1998, Chicago, Ill, USA 3: 536–540.CrossRefGoogle Scholar
  16. 16.
    Naphade MR, Huang TS: Extracting semantics from audio-visual content: the final frontier in multimedia retrieval. IEEE Transactions on Neural Networks 2002, 13(4):793–810. 10.1109/TNN.2002.1021881CrossRefGoogle Scholar
  17. 17.
    Smith JR, Naphade M, Natsev A: Multimedia semantic indexing using model vectors. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 445–448.Google Scholar
  18. 18.
    Hyvärinen A, Karhunen J, Oja E: Independent Component Analysis. John Wiley & Sons, New York, NY, USA; 2001.CrossRefGoogle Scholar
  19. 19.
    Rabiner LR: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 1989, 77(2):257–286. 10.1109/5.18626CrossRefGoogle Scholar
  20. 20.
    Duda RO, Hart PE, Stork DG: Pattern Classification. John Wiley & Sons, New York, NY, USA; 2001.zbMATHGoogle Scholar
  21. 21.
    Li SZ: Content-based audio classification and retrieval using the nearest feature line method. IEEE Transactions Speech Audio Processing 2000, 8(5):619–625. 10.1109/89.861383CrossRefGoogle Scholar
  22. 22.
    Bow S-T: Pattern Recognition and Image Preprocessing. Marcel Dekker, New York, NY, USA; 2002.CrossRefGoogle Scholar
  23. 23.
    Sound Ideas: Sound Effects Library https://doi.org/www.sound-ideas.com/
  24. 24.
    Zilca RD: Text-independent speaker verification using covariance modeling. IEEE Signal Processing Letters 2001, 8(4):97–99. 10.1109/97.911465CrossRefGoogle Scholar
  25. 25.
    Vapnik VN: Statistical Learning Theory. John Wiley & Sons, New York, NY, USA; 1998.zbMATHGoogle Scholar
  26. 26.
    Platt JC, Cristianini N, Shawe-Taylor J: Large margin DAGs for multiclass classification. In Advances in Neural Information Processing Systems. Volume 12. MIT Press, Cambridge, Mass, USA; 2000:547–553.Google Scholar
  27. 27.
    Hsu C-W, Lin C-J: A comparison of methods for multiclass support vector machines. IEEE Transactions Neural Networks 2002, 13(2):415–425. 10.1109/72.991427CrossRefGoogle Scholar
  28. 28.
    Wang J, Xu C, Chng E, Tian Q: Sports highlight detection from keyword sequences using HMM. Proc. IEEE International Conference on Multimedia and Expo (ICME '04), June 2004, Taipei, Taiwan 1: 599–602.Google Scholar
  29. 29.
    Naphade MR, Garg A, Huang TS: Audio-visual event detection using duration dependent input output Markov models. Proc. IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '01), December 2001, Kauai, Hawaii, USA 39–43.CrossRefGoogle Scholar
  30. 30.

Copyright information

© Chu et al. 2006

Authors and Affiliations

  1. 1.Department of Computer Science and Information EngineeringNational Taiwan UniversityTaipeiTaiwan
  2. 2.Graduate Institute of Networking and MultimediaNational Taiwan UniversityTaipeiTaiwan

Personalised recommendations