Single Channel Music Sound Separation Based on Spectrogram Decomposition and Note Classification

  • Wenwu Wang
  • Hafiz Mustafa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6684)


Separating multiple music sources from a single channel mixture is a challenging problem. We present a new approach to this problem based on non-negative matrix factorization (NMF) and note classification, assuming that the instruments used to play the sound signals are known a priori. The spectrogram of the mixture signal is first decomposed into building components (musical notes) using an NMF algorithm. The Mel frequency cepstrum coefficients (MFCCs) of both the decomposed components and the signals in the training dataset are extracted. The mean squared errors (MSEs) between the MFCC feature space of the decomposed music component and those of the training signals are used as the similarity measures for the decomposed music notes. The notes are then labelled to the corresponding type of instruments by the K nearest neighbors (K-NN) classification algorithm based on the MSEs. Finally, the source signals are reconstructed from the classified notes and the weighting matrices obtained from the NMF algorithm. Simulations are provided to show the performance of the proposed system.


Non-negative matrix factorization single-channel sound separation Mel frequency cepstrum coefficients instrument classification K nearest neighbors unsupervised learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdallah, S.A., Plumbley, M.D.: Polyphonic Transcription by Non-Negative Sparse Coding of Power Spectra. In: International Conference on Music Information Retrieval, Barcelona, Spain (October 2004)Google Scholar
  2. 2.
    Barry, D., Lawlor, B., Coyle, E.: Real-time Sound Source Separation: Azimuth Discrimination and Re-synthesis, AES (2004)Google Scholar
  3. 3.
    Brown, G.J., Cooke, M.P.: Perceptual Grouping of Musical Sounds: A Computational Model. J. New Music Res. 23, 107–132 (1994)CrossRefGoogle Scholar
  4. 4.
    Casey, M.A., Westner, W.: Separation of Mixed Audio Sources by Independent Subspace Analysis. In: Proc. Int. Comput. Music Conf. (2000)Google Scholar
  5. 5.
    Devijver, P.A., Kittler, J.: Pattern Recognition - A Statistical Approach. Prentice Hall International, Englewood Cliffs (1982)zbMATHGoogle Scholar
  6. 6.
    Every, M.R., Szymanski, J.E.: Separation of Synchronous Pitched Notes by Spectral Filtering of Harmonics. IEEE Trans. Audio Speech Lang. Process. 14, 1845–1856 (2006)CrossRefGoogle Scholar
  7. 7.
    Fevotte, C., Bertin, N., Durrieu, J.-L.: Nonnegative Matrix Factorization With the Itakura-Saito Divergence. With Application to Music Analysis. Neural Computation 21, 793–830 (2009)CrossRefzbMATHGoogle Scholar
  8. 8.
    FitzGerald, D., Cranitch, M., Coyle, E.: Extended Nonnegative Tensor Factorisation Models for Musical Sound Source Separation, Article ID 872425, 15 pages (2008)Google Scholar
  9. 9.
    Fukunage, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic Press Inc., London (1990)Google Scholar
  10. 10.
    Gutierrez-Osuna, R.: Lecture 12: K Nearest Neighbor Classifier, (accessed January 17, 2010)
  11. 11.
    Hoyer, P.: Non-Negative Sparse Coding. In: IEEE Workshop on Networks for Signal Processing XII, Martigny, Switzerland (2002)Google Scholar
  12. 12.
    Lee, D.D., Seung, H.S.: Learning the Parts of Objects by Non-Negative Matrix Factorization. Nature 401, 788–791 (1999)CrossRefzbMATHGoogle Scholar
  13. 13.
    Lee, D.D., Seung, H.S.: Algorithms for Non-negative Matrix Factorization. In: Neural Information Processing Systems, Denver (2001)Google Scholar
  14. 14.
    Li, Y., Woodruff, J., Wang, D.L.: Monaural Musical Sound Separation Based on Pitch and Common Amplitude Modulation. IEEE Transactions on Audio, Speech, and Language Processing 17, 1361–1371 (2009)CrossRefGoogle Scholar
  15. 15.
    Mellinger, D.K.: Event Formation and Separation in Musical Sound. PhD dissertation, Dept. of Comput. Sci., Standford Univ., Standford, CA (1991)Google Scholar
  16. 16.
    Opolko, F., Wapnick, J.: McGill University master samples, McGill Univ., Montreal, QC, Canada, Tech. Rep. (1987)Google Scholar
  17. 17.
    Pedersen, M.S., Wang, D.L., Larsen, J., Kjems, U.: Two-Microphone Separation of Speech Mixtures. IEEE Trans. on Neural Networks 19, 475–492 (2008)CrossRefGoogle Scholar
  18. 18.
    Rickard, S., Balan, R., Rosca, J.: Real-time Time-Frequency based Blind Source Separation. In: 3rd International Conference on Independent Component Analysis and Blind Source Separation, San Diego, CA (December 2001)Google Scholar
  19. 19.
    Smaragdis, P., Brown, J.C.: Non-negative Matrix Factorization for Polyphonic Music Transcription. In: Proc. IEEE Int. Workshop Application on Signal Process. Audio Acoust., pp. 177–180 (2003)Google Scholar
  20. 20.
    Smaragdis, P.: Non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic inputs. In: Puntonet, C.G., Prieto, A.G. (eds.) ICA 2004. LNCS, vol. 3195, pp. 494–499. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  21. 21.
    The University of Iowa Musical Instrument Samples Database,
  22. 22.
    Virtanen, T.: Sound Source Separation Using Sparse Coding with Temporal Continuity Objective. In: International Computer Music Conference, Singapore (2003)Google Scholar
  23. 23.
    Virtanen, T.: Separation of Sound Sources by Convolutive Sparse Coding. In: Proceedings of ISCA Tutorial and Research Workshop on Statistical and Perceptual Audio Processing, Jeju, Korea (2004)Google Scholar
  24. 24.
    Virtanen, T.: Sound Source Separation in Monaural Music Signals. PhD dissertation, Tampere Univ. of Technol., Tampere, Finland (2006)Google Scholar
  25. 25.
    Virtanen, T.: Monaural Sound Source Separation by Non-Negative Matrix Factorization with Temporal Continuity and Sparseness Criteria. IEEE Transactions on Audio, Speech, and Language Processing 15, 1066–1073 (2007)CrossRefGoogle Scholar
  26. 26.
    Wang, D.L., Brown, G.J.: Computational Auditory Scene Analysis: Principles, Algorithms, and Applications. Wiley/IEEE Press (2006)Google Scholar
  27. 27.
    Wang, B., Plumbley, M.D.: Investigating Single-Channel Audio Source Separation Methods based on Non-negative Matrix Factorization. In: Nandi, Zhu (eds.) Proceedings of the ICA Research Network International Workshop, pp. 17–20 (2006)Google Scholar
  28. 28.
    Wang, B., Plumbley, M.D.: Single Channel Audio Separation by Non-negative Matrix Factorization. In: Digital Music Research Network One-day Workshop (DMRN+1), London (2006)Google Scholar
  29. 29.
    Wang, W., Luo, Y., Chambers, J.A., Sanei, S.: Note Onset Detection via Non-negative Factorization of Magnitude Spectrum. EURASIP Journal on Advances in Signal Processing, Article ID 231367, 15 pages (June 2008); doi:10.1155/2008/231367Google Scholar
  30. 30.
    Wang, W., Cichocki, A., Chambers, J.A.: A Multiplicative Algorithm for Convolutive Non-negative Matrix Factorization Based on Squared Euclidean Distance. IEEE Transactions on Signal Processing 57, 2858–2864 (2009)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Webb, A.: Statistical Pattern Recognition, 2nd edn. Wiley, New York (2005)zbMATHGoogle Scholar
  32. 32.
    Woodruff, J., Pardo, B.: Using Pitch, Amplitude Modulation and Spatial Cues for Separation of Harmonic Instruments from Stereo Music Recordings. EURASIP J. Adv. Signal Process. (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Wenwu Wang
    • 1
  • Hafiz Mustafa
    • 1
  1. 1.Centre for Vision, Speech and Signal Processing (CVSSP)University of SurreyUK

Personalised recommendations