iMusic: a session-sensitive clustered classical music recommender system using contextual representation learning

Abstract

Music has emerged to be of paramount importance to humanity and is not only considered as a source of entertainment but also an agent that causes social and psychological influences. A large number of existing digital music libraries have improved awareness among people through the music recommendation approach. However, several critical technical challenges still require attention and must be well–addressed to provide a reliable recommendation to music listeners. In Indian Classical Music, Raga is composed of coalescing diverse musical note structures. The history of classical music unveils that every raga possesses some distinct sessions. With a vast collection of classical music files in online music libraries, locating and listening to classical music is no more a difficult task. However, searching and listening to the audio of one’s preference may not be simple as it must instinctively satisfy the listener’s preference in a precise session. In this paper, a system termed as iMusic has been proposed to classify, analyze, and recommend the session–sensitive performance of Indian classical music by analyzing the musical note structures followed by features matching. Available note-patterns in the raga performance have been illustrated using a deep neural network and a set of machine learning algorithms where raga samples have been represented as inputs in the projected network and are classified based on the performing sessions. A context-aware k-means clustering algorithm has also been illustrated, entitled as a data filtering algorithm. The proficiency of the filtering algorithm has been established in two ways. Primarily, as a data-sensitive analysis, and second as empirical studies on synthetically obtained real classical music dataset (https://github.com/SamarjitRoy89/iMusic.git) to put on the hybrid music recommendation. In this work, a case-study of session-sensitive Indian music recommender system has been demonstrated using key-strategies viz. listener modelling, representation learning, and music profiling. Eventually, several evaluation metrics have been discussed to characterize the effectiveness of proposed representation learning-based playing session-sensitive music recommendation strategies. The proposed iMusic system renders data classification accuracy of ~ 88%. Such a framework could provide a useful basis regarding studies on hybrid music recommendation systems based on the usefulness of end-users.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. 1.

    Allik A, Thalmann F, Sandler M (2018) Musiclynx: exploring music through artist similarity graphs. In: Companion of the The Web Conference 2018 on The Web Conference 2018, pp 167-170. https://doi.org/10.1145/3184558.3186970

  2. 2.

    Al-Qarni BH, Almogren A, Hassan MM (2018) An efficient networking protocol for internet of things to handle multimedia big data. Multimed Tools Appl 78:1–18. https://doi.org/10.1007/s11042-018-6883-7

    Article  Google Scholar 

  3. 3.

    Alsouda Y, Pllana S, Kurti A (2019) IoT-based urban noise identification using machine learning: performance of SVM, KNN, bagging, and random Forest. In: International Conference on Omni-Layer Intelligent Systems, ACM pp. 62–67. https://doi.org/10.1145/3312614.3312631

  4. 4.

    Andjelkovic I, Parra D, O’Donovan J (2018) Moodplay: interactive music recommendation based on artists’ mood similarity. International Journal of Human-Computer Studies 121:142–159. https://doi.org/10.1016/j.ijhcs.2018.04.004

    Article  Google Scholar 

  5. 5.

    Chakrabarty S, Roy S, De D (2015) A Foremost Survey on State-of-The-Art Computational Music Research. Recent Trends in Computations and Mathematical Analysis in Engineering and Sciences-2015 “CRCMAS 2015”, 16 (2015). Available at < https://ecitydoc.com/download/proceeding-of-recent-trends-in-computations-and-mathematical_pdf >

  6. 6.

    Chakrabarty S, Roy S, De D (2017) Time-slot based intelligent music recommender in Indian music. Intelligent Analysis of Multimedia Information, IGI Global, In, pp 319–351. https://doi.org/10.4018/978-1-5225-0498-6.ch012

    Google Scholar 

  7. 7.

    Chang SK, Deufemia V, Polese G, Vacca M (2007) A normalization framework for multimedia databases. IEEE Trans Knowl Data Eng 19(12):1666–1679. https://doi.org/10.1109/TKDE.2007.190651

    Article  Google Scholar 

  8. 8.

    Fan T (2018) Research and implementation of user clustering based on MapReduce in multimedia big data. Multimed Tools Appl 77(8):10017–10031. https://doi.org/10.1007/s11042-017-4825-4

    Article  Google Scholar 

  9. 9.

    Gulati S, Serrà J, Ganguli KK, Sentürk S, Serra X (2016) Time-delayed melody surfaces for raga recognition. In: 17th International Society for Music Information Retrieval Conference (ISMIR), pp. 751–757, New York, USA (2016). Available at <http://hdl.handle.net/10230/33117>

  10. 10.

    Gulati S, Serra J, Ishwar V, Sentürk S, Serra X (2016) Phrase-based rāga recognition using vector space modeling. In: IEEE conference on acoustics. Speech and Signal Processing (ICASSP), IEEE, pp 66–70. https://doi.org/10.1109/ICASSP.2016.7471638

    Google Scholar 

  11. 11.

    Jeong J, Kim Y, Ahn CW (2017) A multi-objective evolutionary approach to automatic melody generation. Expert Syst Appl 90:50–61. https://doi.org/10.1016/j.eswa.2017.08.014

    Article  Google Scholar 

  12. 12.

    Kluver D, Ekstrand MD, Konstan JA (2018) Rating-based collaborative filtering: algorithms and evaluation. In: Social information access. Springer, Cham, pp 344–390. https://doi.org/10.1007/978-3-319-90092-6_10

    Google Scholar 

  13. 13.

    Kodati S, Vivekanandam R, Ravi G (2019) Comparative analysis of clustering algorithms with heart disease datasets using data mining Weka tool. In: Soft computing and signal processing. Springer, Singapore, pp 111–117. https://doi.org/10.1007/978-981-13-3600-3_11

    Google Scholar 

  14. 14.

    Kovačević A, Milosavljević B, Konjović Z, Vidaković M (2010) Adaptive content-based music retrieval system. Multimed Tools Appl 47(3):525–544. https://doi.org/10.1007/s11042-009-0336-2

    Article  Google Scholar 

  15. 15.

    Lacey L (2015) How the internet of things could impact music composition, production and performance. https://ask.audio/articles/how-the-internet-of-things-could-impact-music-composition-production-performance. Accessed 26 February, 2019.

  16. 16.

    Lee K, Lee YS, Nam Y (2019) A novel approach of making better recommendations by revealing hidden desires and information curation for users of internet of things. Multimed Tools Appl 78(3):3183–3201. https://doi.org/10.1007/s11042-018-6084-4

    Article  Google Scholar 

  17. 17.

    Lu CC, Tseng VS (2009) A novel method for personalized music recommendation. Expert Syst Appl 36(6):10035–10044. https://doi.org/10.1016/j.eswa.2009.01.074

    Article  Google Scholar 

  18. 18.

    Ma X, Lei X, Zhao G, Qian X (2018) Rating prediction by exploring user’s preference and sentiment. Multimed Tools Appl 77(6):6425–6444. https://doi.org/10.1007/s11042-017-4550-z

    Article  Google Scholar 

  19. 19.

    MuChen J, Ying P, Zou M (2018) Improving music recommendation by incorporating social influence. Multimed Tools Appl 78:1–21. https://doi.org/10.1007/s11042-018-5745-7

    Article  Google Scholar 

  20. 20.

    Namin SR (2017) Getting Ya music recommendation groove on with Google cloud platform! https://shinesolutions.com/2017/12/15/getting-ya-music-recommendation-groove-on-with-google-cloud-platform/ (2017). Accessed 9 November, 2018

  21. 21.

    Rao Z, Feng C (2018) Sparse representation classification-based automatic chord recognition for Noisy music. Journal of Information Hiding and Multimedia Signal Processing 9(2):400–409

    MathSciNet  Google Scholar 

  22. 22.

    Raschka S, Patterson J, Nolet C (2020) Machine learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence. Information 11(4):193. https://doi.org/10.3390/info11040193

    Article  Google Scholar 

  23. 23.

    Reddy MS, Adilakshmi T, Akhila M, User Based Collaborative Filtering For Music Recommendation System. International Journal of Innovative Research and Development 2. <http://www.ijird.com/index.php/ijird/article/view/42306>

  24. 24.

    Roy S, Chakrabarty S, De D (2017) Time-Based Raga Recommendation and Information Retrieval of Musical Patterns in Indian Classical Music Using Neural Networks. IAES International Journal of Artificial Intelligence (IJ-AI) 6(1):33–48. https://doi.org/10.11591/ij-ai.v6.i1.pp33-48

    Article  Google Scholar 

  25. 25.

    Roy S, Sarkar D, Hati S, De D (2018) Internet of music things: an edge computing paradigm for opportunistic crowdsensing. J Supercomput 74(11):6069–6101. https://doi.org/10.1007/s11227-018-2511-6

    Article  Google Scholar 

  26. 26.

    Roy S, Sarkar D, De D (2020) Entropy-aware ambient IoT analytics on humanized music information fusion. J Ambient Intell Humaniz Comput 11(1):151–171. https://doi.org/10.1007/s12652-019-01261-x

    Article  Google Scholar 

  27. 27.

    Rui T, Cui P, Zhu W (2016) Joint user-interest and social-influence emotion prediction for individuals. Neurocomputing 230:66–76. https://doi.org/10.1016/j.neucom.2016.11.054

    Article  Google Scholar 

  28. 28.

    Salas J (2016) Generating music from literature using topic extraction and sentiment analysis. IEEE Potentials 37(1):15–18. https://doi.org/10.1109/MPOT.2016.2550015

    MathSciNet  Article  Google Scholar 

  29. 29.

    Sánchez-Moreno D, González ABG, Vicente MDM, Batista VFL, García MNM (2016) A collaborative filtering method for music recommendation using playing coefficients for artists and users. Expert Syst Appl 66:234–244. https://doi.org/10.1016/j.eswa.2016.09.019

    Article  Google Scholar 

  30. 30.

    Schedl M, Bauer C (2018) An analysis of global and regional mainstreaminess for personalized music recommender systems. Journal of Mobile Multimedia 14(1):95–112. https://doi.org/10.13052/jmm1550-4646.1415

    Article  Google Scholar 

  31. 31.

    Schedl M, Zamani H, Chen CW, Deldjoo Y, Elahi M (2018) Current challenges and visions in music recommender systems research. International Journal of Multimedia Information Retrieval 7(2):95–116. https://doi.org/10.1007/s13735-018-0154-2

    Article  Google Scholar 

  32. 32.

    Selvi C, Sivasankar E (2018) A novel adaptive genetic neural network (AGNN) model for recommender systems using modified k-means clustering approach. Multimedia Tools and Applications 1-28. https://doi.org/10.1007/s11042-018-6790-y

  33. 33.

    Sevillano X, Alías F (2014) A one-shot domain-independent robust multimedia clustering methodology based on hybrid multimodal fusion. Multimed Tools Appl 73(3):1507–1543. https://doi.org/10.1007/s11042-013-1655-x

    Article  Google Scholar 

  34. 34.

    Shakirova E (2017) Collaborative filtering for music recommender system. In: Young Researchers in Electrical and Electronic Engineering (EIConRus), 2017 IEEE Conference of Russian, pp. 548-550, IEEE (2017). https://doi.org/10.1109/EIConRus.2017.7910613

  35. 35.

    Stober S, Nürnberger A (2013) Adaptive music retrieval–a state of the art. Multimed Tools Appl 65(3):467–494. https://doi.org/10.1007/s11042-012-1042-z

    Article  Google Scholar 

  36. 36.

    Su JH, Chang WY, Tseng VS (2017) Integrated Mining of Social and Collaborative Information for music recommendation. Data Science and Pattern Recognition 1(1):13–30

    Google Scholar 

  37. 37.

    Sunitha M, Adilakshmi T (2018) Music recommendation system with user-based and item-based collaborative filtering technique. In: Networking communication and data knowledge engineering. Springer, Singapore, pp 267–278. https://doi.org/10.1007/978-981-10-4585-1_22

    Google Scholar 

  38. 38.

    Tsai CW, Liao MY, Yang CS, Chiang MC (2013) Classification algorithms for interactive multimedia services: a review. Multimed Tools Appl 67(1):137–165. https://doi.org/10.1007/s11042-011-0957-0

    Article  Google Scholar 

  39. 39.

    Turchet L, Fischione C, Essl G, Keller D, Barthet M (2018) Internet of musical things: vision and challenges. IEEE Access 6:61994–62017. https://doi.org/10.1109/ACCESS.2018.2872625

    Article  Google Scholar 

  40. 40.

    Uhlich S, Porcu M, Giron F, Enenkl M, Kemp T, Takahashi N, Mitsufuji Y (2017) Improving music source separation based on deep neural networks through data augmentation and network blending. In: 2017 IEEE international conference on acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 261–265. https://doi.org/10.1109/ICASSP.2017.7952158

  41. 41.

    Wang D, Deng S, Xu G (2018) Sequence-based context-aware music recommendation. Information Retrieval Journal 21(2–3):230–252. https://doi.org/10.1007/s10791-017-9317-7

    Article  Google Scholar 

  42. 42.

    Wang Q, Su F, Wang Y (2020) Hierarchical attentive deep neural networks for semantic music annotation through multiple music representations. International Journal of Multimedia Information Retrieval 9(1):3–16. https://doi.org/10.1007/s13735-019-00186-7

    Article  Google Scholar 

  43. 43.

    Yang J, He S, Lin Y, Lv Z (2017) Multimedia cloud transmission and storage system based on internet of things. Multimed Tools Appl 76(17):17735–17750. https://doi.org/10.1007/s11042-015-2967-9

    Article  Google Scholar 

  44. 44.

    Zangerle E, Chen CM, Tsai MF, Yang YH (2018) Leveraging affective Hashtags for ranking music recommendations. IEEE Trans Affect Comput:1. https://doi.org/10.1109/TAFFC.2018.2846596

  45. 45.

    Zhang Y, Chen M, Huang D, Wu D, Li Y (2017) iDoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Futur Gener Comput Syst 66:30–35. https://doi.org/10.1016/j.future.2015.12.001

    Article  Google Scholar 

  46. 46.

    Zheng E, Kondo GY, Zilora S, Yu Q (2018) Tag-aware dynamic music recommendation. Expert Syst Appl 106:244–251. https://doi.org/10.1016/j.eswa.2018.04.014

    Article  Google Scholar 

  47. 47.

    Zheng HT, Chen JY, Liang N, Sangaiah AK, Jiang Y, Zhao CZ (2019) A deep temporal neural music recommendation model utilizing music and user metadata. Appl Sci 9(4):703. https://doi.org/10.3390/app9040703

    Article  Google Scholar 

Download references

Acknowledgments

Authors are grateful to the University Grant Commission (UGC), Govt. of India, for sanctioning a research fellowship under which this contribution has been completed. Authors are also grateful to the Department of Science and Technology (DST) for sanctioning projects and the TEQIP-III, MAKAUT, WB.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Debashis De.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Roy, S., Biswas, M. & De, D. iMusic: a session-sensitive clustered classical music recommender system using contextual representation learning. Multimed Tools Appl (2020). https://doi.org/10.1007/s11042-020-09126-8

Download citation

Keywords

  • Classical music
  • Computational musicology
  • Deep neural network
  • Representation learning
  • Clustering
  • Hybrid collaborative filtering
  • Music recommendation
  • Internet of music things