Skip to main content
SpringerLink
Log in

Emotion Annotation of Music: A Citizen Science Approach

  • Conference paper
  • First Online:
Collaboration Technologies and Social Computing (CollabTech 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12856))

Abstract

The understanding of the emotions in music has motivated research across diverse areas of knowledge for decades. In the field of computer science, there is a particular interest in developing algorithms to “predict” the emotions in music perceived by or induced to a listener. However, the gathering of reliable “ground truth” data for modeling the emotional content of music poses challenges, since tasks related with annotations of emotions are time consuming, expensive and cognitively demanding due to its inherent subjectivity and its cross-disciplinary nature. Citizen science projects have proven to be a useful approach to solve these types of problems where there is a need for recruiting collaborators for massive scale tasks. We developed a platform for annotating emotional content in musical pieces following a citizen science approach, to benefit not only the researchers, who benefit from the generated dataset, but also the volunteers, who are engaged to collaborate on the research project, not only by providing annotations but also through their self and community-awareness about the emotional perception of the music. Likewise, gamification mechanisms motivate the participants to explore and discover new music based on the emotional content. Preliminary user evaluations showed that the platform design is in line with the motivations of the general public, and that the citizen science approach offers an iterative refinement to enhance the quantity and quality of contributions by involving volunteers in the design process. The usability of the platform was acceptable, although some of the features require improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.mturk.com/.

  2. 2.

    https://www.prolific.co/.

  3. 3.

    https://www.zooniverse.org/.

  4. 4.

    https://www.w3.org/TR/annotation-model.

  5. 5.

    https://enthusiasts.trompamusic.eu/.

References

  1. Abu Amsha, O., Schneider, D.K., Fernandez-Marquez, J.L., Da Costa, J., Fuchs, B., Kloetzer, L.: Data analytics in citizen cyberscience: evaluating participant learning and engagement with analytics. Hum. Comput. 3(1), 69–97 (2016). https://doi.org/10.15346/hc.v3i1.5

  2. von Ahn, L.: Games with a purpose. Computer 39(6), 92–94 (2006). https://doi.org/10.1109/MC.2006.196

    Article  Google Scholar 

  3. Aljanaki, A., Wiering, F., Veltkamp, R.C.: Studying emotion induced by music through a crowdsourcing game. Inf. Process. Manage. 52(1), 115–128 (2016). https://doi.org/10.1016/j.ipm.2015.03.004. emotion and Sentiment in Social and Expressive Media

  4. Aljanaki, A., Yang, Y.H., Soleymani, M.: Developing a benchmark for emotional analysis of music. PLoS One 1–22 (2017)

    Google Scholar 

  5. Barrett, L.F.: How emotions are made: the secret life of the brain. Houghton Mifflin Harcourt (2017)

    Google Scholar 

  6. Barthet, M., Fazekas, G., Sandler, M.: Music emotion recognition: from content- to context-based models. In: From Sounds to Music and Emotions, pp. 228–252. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41248-6_13

    Chapter  Google Scholar 

  7. Baruch, A., May, A., Yu, D.: The motivations, enablers and barriers for voluntary participation in an online crowdsourcing platform. Comput. Hum. Behav. 64, 923–931 (2016). https://doi.org/10.1016/j.chb.2016.07.039

    Article  Google Scholar 

  8. Bonney, R., Phillips, T.B., Ballard, H.L., Enck, J.W.: Can citizen science enhance public understanding of science? Public Understand. Sci. 25(1), 2–16 (2016). https://doi.org/10.1177/0963662515607406. pMID: 26445860

  9. Brenton, P., von Gavel, S., Vogel, E., Lecoq, M.E.: Technology infrastructure for citizen science, pp. 63–80. UCL Press (2018). http://www.jstor.org/stable/j.ctv550cf2.12

  10. Brooke, J.: SUS - a quick and dirty usability scale. In: Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B. (eds.) Usability Evaluation in Industry, pp. 189–194 (1996)

    Google Scholar 

  11. Brooke, J.: Sus: a retrospective. J. Usability Stud. 8(2), 29–40 (2013)

    Google Scholar 

  12. Budd, M.: Music and the Emotion. Routledge, London (1992)

    Google Scholar 

  13. Cook, N.: Beyond the Score - Music as performance. Oxford University Press, Oxford (2013)

    Google Scholar 

  14. Eerola, T.: Music and emotions. In: Bader, R. (ed.) Springer Handbook of Systematic Musicology. SH, pp. 539–554. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-662-55004-5_29

    Chapter  Google Scholar 

  15. Eerola, T., Vuoskoski, J.K.: A comparison of the discrete and dimensional models of emotion in music. Psychol. Music 39(1), 18–49 (2011). https://doi.org/10.1177/0305735610362821

    Article  Google Scholar 

  16. Eerola, T., Vuoskoski, J.K.: A review of music and emotion studies: approaches, emotion models, and stimuli. Music Percept. Interdiscipl. J. 30(3), 307–340 (2013)

    Article  Google Scholar 

  17. Ekman, P.: Are there basic emotions. Psychol. Rev. 99(3), 550–553 (1992)

    Article  Google Scholar 

  18. English, P., Richardson, M., Garzón-Galvis, C.: From crowdsourcing to extreme citizen science: Participatory research for environmental health. Ann. Rev. Public Health 39(1), 335–350 (2018). https://doi.org/10.1146/annurev-publhealth-040617-013702. pMID: 29608871

  19. Science Europe: Science Europe briefing paper on citizen science (2018). https://www.scienceeurope.org/media/gjze3dv4/se_briefingpaper_citizenscience.pdf

  20. Gómez-Cañón, J.S., Cano, E., Herrera, P., Gómez, E.: In: Joyful for you and tender for us: the influence of individual characteristics and language on emotion labeling and classification, pp. 853–860. Montréal, Canada (2020)

    Google Scholar 

  21. Haklay, M.: Citizen science and volunteered geographic information: overview and typology of participation. In: Sui, D., Elwood, S., Goodchild, M. (eds.) Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice, pp. 105–122. Springer, Dordrecht (2013). https://doi.org/10.1007/978-94-007-4587-2_7

  22. Hallam, S., Cross, I., Thaut, M.: The Oxford Handbook of Music Psychology. Oxford University Press, Oxford (2016)

    Google Scholar 

  23. Hu, X., Downie, S., Laurier, C., Bay, M., Ehmann, A.: The 2007 mirex audio mood classification task: lessons learned. In: Proceedings 9th International Conference Music Information Retrieval, pp. 462–467 (2008)

    Google Scholar 

  24. Hu, X., Yang, Y.H.: Cross-dataset and cross-cultural music mood prediction: a case on Western and Chinese Pop songs. IEEE Trans. Affect. Comput. 8(2), 228–240 (2017)

    Article  Google Scholar 

  25. Jennett, C., Cox, A.L.: Digital citizen science and the motivations of volunteers. In: The Wiley Handbook of Human Computer Interaction, vol. chap. 39, pp. 831–841. Wiley, Hoboken (2018). https://doi.org/10.1002/9781118976005.ch39

  26. Juslin, P.N.: Handbook of Music and Emotion: Theory, Research Applications. Oxford University Press, Oxford (2010)

    Google Scholar 

  27. Juslin, P.N.: Musical Emotions Explained. Oxford University Press, Oxford (2019)

    Book  Google Scholar 

  28. Kim, Y.E., Schmidt, E.M., Emelle, L.: Moodswings: a collaborative game for music mood label collection. ISMIR 8, 231–236 (2008)

    Google Scholar 

  29. Kim, Y.E., Schmidt, E.M., Migneco, R., Morton, B.G., Richardson, P., Scott, J., Speck, J.A., Turnbull, D.: Music emotion recognition: a state of the art review. Proc. ISMIR 86, 937–952 (2010)

    Google Scholar 

  30. Krumhansl, C.L.: An exploratory study of musical emotions and psychophysiology. Canadian J. Exp. Psychol. Revue canadienne de psychologie expérimentale 51(4), 336 (1997)

    Article  Google Scholar 

  31. Law, E.L., Von Ahn, L., Dannenberg, R.B., Crawford, M.: Tagatune: a game for music and sound annotation. In: ISMIR. vol. 3, p. 2 (2007). http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.106.7184

  32. Weigl, D., et al.: Interweaving and enriching digital music collections for scholarship, performance, and enjoyment. In: 6th International Conference on Digital Libraries for Musicology, pp. 84–88. DLfM 2019, Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3358664.3358666

  33. Mandel, M.I., Ellis, D.P.: A web-based game for collecting music metadata. J. New Music Res. 37(2), 151–165 (2008). https://doi.org/10.1080/09298210802479300

    Article  Google Scholar 

  34. Nov, O., Arazy, O., Anderson, D.: Scientists@home: what drives the quantity and quality of online citizen science participation? PLoS One 9(4), 1–11 (2014). https://doi.org/10.1371/journal.pone.0090375

  35. Patel, A.D.: Music as a transformative technology of the mind: an update. In: Honing, H. (ed.) The Origins of Musicality, p. chap. 5. MIT Press, Massachusetts (2018)

    Google Scholar 

  36. Ponciano, L., Brasileiro, F.: Finding volunteers’ engagement profiles in human computation for citizen science projects. Hum. Comput. 1(2), 1–17 (2014). https://doi.org/10.15346/hc.v1i2.12

  37. Raddick, M.J., et al.: Galaxy zoo: motivations of citizen scientists. Astronomy Educ. Rev. 12(1),(2013). https://doi.org/10.3847/AER2011021

  38. Resnik, D.B., Elliott, K.C., Miller, A.K.: A framework for addressing ethical issues in citizen science. Environ. Sci. Poli. 54, 475–481 (2015). https://doi.org/10.1016/j.envsci.2015.05.008

    Article  Google Scholar 

  39. Rotman, D., Hammock, J., Preece, J., Hansen, D., Boston, C., Bowser, A., He, Y.: Motivations affecting initial and long-term participation in citizen science projects in three countries. In: iconference, pp. 110–124. Presented at the (2014). https://doi.org/10.9776/14054

  40. Russell, J.A.: A circumplex model of affect. Personality Soc. Psychol. 39(6), 1161–1178 (1980)

    Article  Google Scholar 

  41. Simperl, E., Reeves, N., Phethean, C., Lynes, T., Tinati, R.: Is virtual citizen science a game? Trans. Soc. Comput. 1(2) (2018). https://doi.org/10.1145/3209960

  42. Soleymani, M., et al.: In: Emotional analysis of music: a comparison of methods, vol. MM 2014, pp. 1161–1164. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2647868.2655019

  43. Turnbull, D., Liu, R., Barrington, L., Lanc kriet, G.R. : A game-based approach for collecting semantic annotations of music. ISMIR. 7, 535–538 (2007)

    Google Scholar 

  44. Vastenburg, M., Romero Herrera, N., Van Bel, D., Desmet, P.: Pmri: development of a pictorial mood reporting instrument. In: CHI 2011 Extended Abstracts on Human Factors in Computing Systems, pp. 2155–2160. CHI EA 2011, Association for Computing Machinery, New York (2011). https://doi.org/10.1145/1979742.1979933

  45. Wald, D.M., Longo, J., Dobell, A.R.: Design principles for engaging and retaining virtual citizen scientists. Conserv. Biol. 30(3), 562–570 (2016). https://doi.org/10.1111/cobi.12627

    Article  Google Scholar 

  46. West, S.E., Pateman, R.M.: Recruiting and retaining participants in citizen science: what can be learned from the volunteering literature? Citizen Sci. Theory Pract. 1(2),(2016). https://doi.org/10.5334/cstp.8

  47. Wiggins, A., Crowston, K.: From conservation to crowdsourcing: a typology of citizen science. In: 2011 44th Hawaii International Conference on System Sciences, pp. 1–10 (2011). https://doi.org/10.1109/HICSS.2011.207

  48. Wynn, J.: Citizen Science in the Digital Age: Rhetoric, Science, and Public Engagement. The University of Alabama Press, Tuscaloosa (2019)

    Google Scholar 

  49. Yang, Y.H., Chen, H.H.: Music Emotion Recognition. CRC Press, Boca Raton (2011)

    Google Scholar 

  50. Zentner, M., Grandjean, D., Scherer, K.R.: Emotions evoked by the sound of music: characterization, classification, and measurement. Emotion 8(4), 494 (2008). https://doi.org/10.1037/1528-3542.8.4.494

    Article  Google Scholar 

Download references

Acknowledgments

This work has been partially funded by the TROMPA project, European Union’s Horizon 2020 research and innovation programme under grant agreement no. 770376. TIDE-UPF also acknowledges the support by FEDER, the National Research Agency of the Spanish Ministry of Science and Innovation, TIN2017-85179-C3-3-R, PID2020-112584RB-C33, MDM-2015-0502, the Ramon y Cajal programme (P. Santos) and by ICREA under the ICREA Academia programme (D. Hernández-Leo, Serra Hunter).

Author information

Authors and Affiliations

  1. Universitat Pompeu Fabra, Barcelona, 08005, Spain

    Nicolás Felipe Gutiérrez Páez, Juan Sebastián Gómez-Cañón, Lorenzo Porcaro, Patricia Santos, Davinia Hernández-Leo & Emilia Gómez

Authors
  1. Nicolás Felipe Gutiérrez Páez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Sebastián Gómez-Cañón
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lorenzo Porcaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Davinia Hernández-Leo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Emilia Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicolás Felipe Gutiérrez Páez .

Editor information

Editors and Affiliations

  1. Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain

    Davinia Hernández-Leo

  2. School of Creative Science, Waseda University, Tokyo, Japan

    Reiko Hishiyama

  3. Department of Management Control and Information Systems, University of Chile, Santiago, Chile

    Gustavo Zurita

  4. Human-Computer Interaction Group, University of Trier, Trier, Germany

    Benjamin Weyers

  5. Institute of Computer Science, University of Tartu, Tartu, Estonia

    Alexander Nolte

  6. Academic Center for Computing and Media Studies, Kyoto University, Kyoto, Japan

    Hiroaki Ogata

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gutiérrez Páez, N.F., Gómez-Cañón, J.S., Porcaro, L., Santos, P., Hernández-Leo, D., Gómez, E. (2021). Emotion Annotation of Music: A Citizen Science Approach. In: Hernández-Leo, D., Hishiyama, R., Zurita, G., Weyers, B., Nolte, A., Ogata, H. (eds) Collaboration Technologies and Social Computing. CollabTech 2021. Lecture Notes in Computer Science(), vol 12856. Springer, Cham. https://doi.org/10.1007/978-3-030-85071-5_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85071-5_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85070-8

  • Online ISBN: 978-3-030-85071-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation