Advertisement

Relational recurrent neural networks for polyphonic sound event detection

  • Junbo Ma
  • Ruili WangEmail author
  • Wanting JiEmail author
  • Hao Zheng
  • En Zhu
  • Jianping Yin
Article
  • 19 Downloads

Abstract

A smart environment is one of the application scenarios of the Internet of Things (IoT). In order to provide a ubiquitous smart environment for humans, a variety of technologies are developed. In a smart environment system, sound event detection is one of the fundamental technologies, which can automatically sense sound changes in the environment and detect sound events that cause changes. In this paper, we propose the use of Relational Recurrent Neural Network (RRNN) for polyphonic sound event detection, called RRNN-SED, which utilized the strength of RRNN in long-term temporal context extraction and relational reasoning across a polyphonic sound signal. Different from previous sound event detection methods, which rely heavily on convolutional neural networks or recurrent neural networks, the proposed RRNN-SED method can solve long-lasting and overlapping problems in polyphonic sound event detection. Specifically, since the historical information memorized inside RRNNs is capable of interacting with each other across a polyphonic sound signal, the proposed RRNN-SED method is effective and efficient in extracting temporal context information and reasoning the unique relational characteristic of the target sound events. Experimental results on two public datasets show that the proposed method achieved better sound event detection results in terms of segment-based F-score and segment-based error rate.

Keywords

Internet of Things smart environment deep neural networks recurrent neural networks sound event detection 

Notes

Acknowledgments

This work is partially supported by the National Key R&D Program of China (2018YFB1003203), the Natural Science Foundation of Zhejiang Province (No. LY18F010008), the National Science Foundation of China (No. 61672528, 61773392), and the Marsden Fund of New Zealand.

References

  1. 1.
    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M et al (2016) "Tensorflow: a system for large-scale machine learning." In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 16. 265-283Google Scholar
  2. 2.
    Sharath A, Virtanen T (2017) "A report on sound event detection with different binaural features." arXiv preprint arXiv:1710.02997Google Scholar
  3. 3.
    Adavanne S, G Parascandolo, P Pertilä, T Heittola, T Virtanen (2016) “Sound event detection in multichannel audio using spatial and harmonic features,” IEEE Detection and Classification of Acoustic Scenes and Events workshopGoogle Scholar
  4. 4.
    Adavanne S, G Parascandolo, P Pertilä, T Heittola, T Virtanen (2017a) "Sound event detection in multichannel audio using spatial and harmonic features." arXiv preprint arXiv:1706.02293Google Scholar
  5. 5.
    Adavanne S, P Pertilä, T Virtanen (2017b) "Sound event detection using spatial features and convolutional recurrent neural network." In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pp. 771-775. IEEEGoogle Scholar
  6. 6.
    Cakır E, T Virtanen (2018) "End-to-End polyphonic sound event detection using convolutional recurrent neural networks with learned time-frequency representation input.". In Neural Networks (IJCNN), 2018 International Joint Conference on, pp. 1-7. IEEEGoogle Scholar
  7. 7.
    Cakir E, T Heittola, H Huttunen, T Virtanen (2015) "Polyphonic sound event detection using multi label deep neural networks." In Neural Networks (IJCNN), 2015 International Joint Conference on, pp. 1-7. IEEEGoogle Scholar
  8. 8.
    Chen Y, Y Zhang, Z Duan (2017) "DCASE2017: sound event detection using convolutional neural networks." DCASE2017 Challenge, Tech. RepGoogle Scholar
  9. 9.
    Dang A, TH Vu, J-C Wang (2017a) "A survey of deep learning for polyphonic sound event detection." In Orange Technologies (ICOT), 2017 International Conference on, pp. 75-78. IEEEGoogle Scholar
  10. 10.
    Dang A, TH Vu, J-C Wang (2017b) "Deep learning for DCASE2017 challenge." Detection and Classification of Acoustic Scenes and Events (DCASE 2017) Proceedings 2017Google Scholar
  11. 11.
    Heittola T, Mesaros A, Eronen A, Virtanen T (2013) "Context-dependent sound event detection" EURASIP J Audio, Speech, Music Proc 2013(1):1CrossRefGoogle Scholar
  12. 12.
    Ioffe S, C Szegedy (2015) "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167Google Scholar
  13. 13.
    Jeong Il-Y, S Lee, Y Han, and K Lee (2017) "Audio event detection using multiple-input convolutional neural network." Detection and Classification of Acoustic Scenes and Events (DCASE)Google Scholar
  14. 14.
    Ji W, R Wang, J Ma (2018) "Dictionary-based active learning method for sound event classification." Multimedia tools and applicationsGoogle Scholar
  15. 15.
    Kingma DP, J Ba (2014) "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980Google Scholar
  16. 16.
    Kroos C, M Plumbley (2017) "Neuroevolution for sound event detection in real life audio: A pilot study." Detection and Classification of Acoustic Scenes and Events (DCASE 2017) Proceedings 2017Google Scholar
  17. 17.
    Lai Y-H, C-H Wang, S-Y Hou, B-Y Chen, Y Tsao, Y-W Liu (2016) "DCASE report for task 3: Sound event detection in real life audio." IEEE AASP Challenge: Detection and Classification of Acoustic Scenes and EventsGoogle Scholar
  18. 18.
    Li P, Chen Z, Yang LT, Zhang Q, Jamal Deen M (2018) "Deep convolutional computation model for feature learning on big data in Internet of Things." IEEE Trans Ind Inform 14(2):790–798CrossRefGoogle Scholar
  19. 19.
    Srivastava, N, Hinton, G, Krizhevsky, A, Sutskever, I & Salakhutdinov, R (2014) "Dropout: a simple way to prevent neural networks from overfitting." J Machine Learning Res 15, pp. 1929–1958Google Scholar
  20. 20.
    Mahdavinejad, M Saeid, M Rezvan, M Barekatain, P Adibi, P Barnaghi, and AP Sheth (2017) "Machine learning for Internet of Things data analysis: A survey." Digital Communications and NetworksGoogle Scholar
  21. 21.
    Mesaros A, T Heittola, A Eronen, T Virtanen (2010) "Acoustic event detection in real life recordings." In Signal Processing Conference, 2010 18th European, pp. 1267-1271. IEEEGoogle Scholar
  22. 22.
    Mesaros A, T Heittola, O Dikmen, T Virtanen (2015) "Sound event detection in real life recordings using coupled matrix factorization of spectral representations and class activity annotations." In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 151-155. IEEEGoogle Scholar
  23. 23.
    Mesaros A, T Heittola, T Virtanen (2016a) "TUT database for acoustic scene classification and sound event detection." In Signal Processing Conference (EUSIPCO), 2016 24th European, pp. 1128-1132. IEEEGoogle Scholar
  24. 24.
    Mesaros A, Heittola T, Virtanen T (2016b) "Metrics for polyphonic sound event detection." Appl Sci 6(6):162CrossRefGoogle Scholar
  25. 25.
    Mohammadi M, Al-Fuqaha A, Sorour S, Guizani M (2018) "Deep learning for IoT big data and streaming analytics: A survey." IEEE Commun Surv TutorGoogle Scholar
  26. 26.
    Morrison D, R Wang, LC De Silva (2005a) "Spoken affect classification using neural networks." In Granular Computing, 2005 IEEE International Conference on, vol. 2, pp. 583-586. IEEEGoogle Scholar
  27. 27.
    Morrison D, R Wang, LC De Silva, WL Xu (2005b) "Real-time spoken affect classification and its application in call-centres." In Information Technology and Applications, 2005. ICITA 2005. Third International Conference on, vol. 1, pp. 483-487. IEEEGoogle Scholar
  28. 28.
    Ozer I, Ozer Z, Findik O (2018) "Noise robust sound event classification with convolutional neural network." Neurocomputing 272:505–512CrossRefGoogle Scholar
  29. 29.
    Parascandolo G, H Huttunen, T Virtanen (2016) "Recurrent neural networks for polyphonic sound event detection in real life recordings." In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pp. 6440-6444. IEEEGoogle Scholar
  30. 30.
    Parascandolo G, Heittola T, Huttunen H, Virtanen T (2017) "Convolutional recurrent neural networks for polyphonic sound event detection." IEEE/ACM Trans Audio, Speech, Lang Proc 25(6):1291–1303CrossRefGoogle Scholar
  31. 31.
    Phan H, M Krawczyk-Becker, T Gerkmann, A Mertins (2017) "DNN and CNN with weighted and multi-task loss functions for audio event detection." arXiv preprint arXiv:1708.03211Google Scholar
  32. 32.
    Poliner GE, Ellis DPW (2006) "A discriminative model for polyphonic piano transcription." EURASIP J Adv Sign Proc 2007(1):048317CrossRefGoogle Scholar
  33. 33.
    Santoro A, R Faulkner, D Raposo, J Rae, M Chrzanowski, T Weber, D Wierstra, O Vinyals, R Pascanu, T Lillicrap (2018) "Relational recurrent neural networks." arXiv preprint arXiv:1806.01822Google Scholar
  34. 34.
    Schmidhuber J (2015) "Deep learning in neural networks: An overview." Neural Netw 61:85–117CrossRefGoogle Scholar
  35. 35.
    Sharath A, A Politis, T Virtanen (2018) "Multichannel sound event detection using 3D convolutional neural networks for learning inter-channel features." arXiv preprint arXiv:1801.09522Google Scholar
  36. 36.
    Stojkoska, Risteska BL, Trivodaliev KV (2017) "A review of Internet of Things for smart home: Challenges and solutions." J Clean Prod 140:1454–1464CrossRefGoogle Scholar
  37. 37.
    Vaswani A, N Shazeer, N Parmar, J Uszkoreit, L Jones, AN Gomez, Ł Kaiser, I Polosukhin (2017) "Attention is all you need." In Advances in Neural Information Processing Systems, pp. 6000-6010Google Scholar
  38. 38.
    Vu TH, Wang J-C (2016) "Acoustic scene and event recognition using recurrent neural networks." Detection and Classification of Acoustic Scenes and Events 2016Google Scholar
  39. 39.
    Wang R, Ji W, Liu M, Wang X, Weng J, Deng S, Gao S, Yuan C (2018) "Review on mining data from multiple data sources." Pattern Recognition LettersGoogle Scholar
  40. 40.
    Yang J, He S, Lin Y, Lv Z (2017) "Multimedia cloud transmission and storage system based on Internet of Things." Multimed Tools Appl 76(17):17735–17750CrossRefGoogle Scholar
  41. 41.
    Zhang H, McLoughlin IV, Song Y (2016) "Robust Sound Event Detection in Continuous Audio Environments." In Interspeech, pp. 2977-2981Google Scholar
  42. 42.
    Zhou J (2017) "Sound event detection in multichannel audio LSTM network." DCASE2017 Challenge, Tech. RepGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Massey UniversityAucklandNew Zealand
  2. 2.School of ComputerNational University of Defense TechnologyChangshaChina
  3. 3.Zhejiang Gongshang UniversityHangzhouChina
  4. 4.College of information engineeringNanjing Xiaozhuang UniversityNanjingChina
  5. 5.School of ComputerNational University of Defense TechnologyChangshaChina
  6. 6.Dongguan University of TechnologyDongguanChina

Personalised recommendations