Query-by-Dancing: A Dance Music Retrieval System Based on Body-Motion Similarity
This paper presents Query-by-Dancing, a dance music retrieval system that enables a user to retrieve music using dance motions. When dancers search for music to play when dancing, they sometimes find it by referring to online dance videos in which the dancers use motions similar to their own dance. However, previous music retrieval systems could not support retrieval specialized for dancing because they do not accept dance motions as a query. Therefore, we developed our Query-by-Dancing system, which uses a video of a dancer (user) as the input query to search a database of dance videos. The query video is recorded using an ordinary RGB camera that does not obtain depth information, like a smartphone camera. The poses and motions in the query are then analyzed and used to retrieve dance videos with similar poses and motions. The system then enables the user to browse the music attached to the videos it retrieves so that the user can find a piece that is appropriate for their dancing. An interesting problem here is that a simple search for the most similar videos based on dance motions sometimes includes results that do not match the intended dance genre. We solved this by using a novel measure similar to tf-idf to weight the importance of dance motions when retrieving videos. We conducted comparative experiments with 4 dance genres and confirmed that the system gained an average of 3 or more evaluation points for 3 dance genres (waack, pop, break) and that our proposed method was able to deal with different dance genres.
KeywordsDance Music Video Retrieval system Body-motion
This work was supported in part by JST ACCEL Grant Number JPMJAC1602, Japan.
- 1.Cao, Z., Simon, T., Wei, S., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: The 2017 IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
- 3.Chen, J., Chen, A.: Query by rhythm: an approach for song retrieval in music databases. In: Proceedings of the 8th International Workshop on Research Issues in Data Engineering: Continuous-Media Databases and Applications, pp. 139–146 (1998)Google Scholar
- 4.Ghias, A., Logan, J., Chamberlin, D., Smith, B.C.: Query by humming - musical information retrieval in an audio database. In: Proceedings of ACM Multimedia 1995, pp. 231–236 (1995)Google Scholar
- 5.Jang, J.-S.R., Lee, H.-R., Yeh, C.-H.: Query by tapping: a new paradigm for content-based music retrieval from acoustic input. In: Shum, H.-Y., Liao, M., Chang, S.-F. (eds.) PCM 2001. LNCS, vol. 2195, pp. 590–597. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45453-5_76CrossRefGoogle Scholar
- 6.Maezawa, A., Goto, M., Okuno, H.G.: Query-by-conducting: an interface to retrieve classical-music interpretations by real-time tempo input. In: The 11th International Society of Music Information Retrieval, pp. 477–482 (2010)Google Scholar
- 10.Smiraglia, R.P.: Musical works as information retrieval entities: epistemological perspectives. In: The 2nd International Society of Music Information Retrieval, pp. 85–91 (2001)Google Scholar