Skip to main content
SpringerLink
Log in

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

  • Conference paper
  • First Online:
Mathematics and Computation in Music (MCM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11502))

Included in the following conference series:

Abstract

The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications:

  • A picture can become a sort of score; graphic scores can have a different source of interpretation;

  • Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis.

The current version has been developed in a Virtual Reality (VR) environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    More specifically, a different set of values of parameters of the physical models. Depending on the RGB values, two instruments are chosen and cross-synthesized. The processe is explained more in detail in 2.5.

  2. 2.

    A programming framework providing a solid environment for graphic intensive applications [9].

  3. 3.

    The Wekinator is a software for the application of Machine Learning in the arts field [10].

  4. 4.

    A virtual straight line drawn from the controller towards infinity.

  5. 5.

    For the sake of clarity, it is worth pointing out that the model is here used only as a way to fastly produce interpolated values. Therefore, its prediction accuracy is not a main concern in this specific case.

  6. 6.

    Strategies of data extraction could include spectrographic analysis, image segmentation (by grouping sets of pixels) and feature extraction.

References

  1. Cytowic, E.R.: Synesthesia: A Union of the Senses, 2nd edn. MIT Press, Cambridge (2002). https://doi.org/10.1007/978-1-4612-3542-2

    Book  Google Scholar 

  2. Payling, D.: Visual Music Composition with Electronic Sound and Video. Staffordshire University (2014)

    Google Scholar 

  3. Shatter, G., Zueger, E., Nitschke, C.: A synaesthetic approach for a synthesizer interface based on genetic algorithm and fuzzy sets. In: Proceedings of International Computer Music Conference 2005 (ICMC 2005), pp. 664–667 (2005)

    Google Scholar 

  4. Margounakis, D., Politis, D.: Converting images to music using their colour properties. In: Proceedings of the 12th International Conference on Auditory Display (ICAD06), pp. 198–205 (2006)

    Google Scholar 

  5. Specdrums on Kickstarter. https://www.kickstarter.com/projects/364756202/specdrums-music-at-your-fingertips. Accessed 4 Jan 2019

  6. McClard, P.T.: Music generating system and method utilizing control of music based upon displayed color. US Brevetto US5689078A (1995)

    Google Scholar 

  7. Macedo, A.R.: Visible and audible spectrums-a proposal of correspondence. Proc. Artech 5, 168–171 (2010)

    Google Scholar 

  8. Rogowska, A.: Synaesthesia and Individual Differences. Cambridge University Press, Cambridge (2015)

    Book  Google Scholar 

  9. Unity. https://www.unity3d.com. Accessed 4 Jan 2019

  10. Wekinator - Instructions. http://www.wekinator.org/instructions/. Accessed 4 Jan 2019

Download references

Author information

Authors and Affiliations

  1. Hong Kong Baptist University, Kowloon Tong, Hong Kong

    Giovanni Santini

Authors
  1. Giovanni Santini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Santini .

Editor information

Editors and Affiliations

  1. Georgia State University, Atlanta, GA, USA

    Mariana Montiel

  2. Technical University of Madrid, Madrid, Spain

    Francisco Gomez-Martin

  3. Technological University of the Mixteca, Oaxaca, Mexico

    Octavio A. Agustín-Aquino

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Santini, G. (2019). Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. In: Montiel, M., Gomez-Martin, F., Agustín-Aquino, O.A. (eds) Mathematics and Computation in Music. MCM 2019. Lecture Notes in Computer Science(), vol 11502. Springer, Cham. https://doi.org/10.1007/978-3-030-21392-3_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-21392-3_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-21391-6

  • Online ISBN: 978-3-030-21392-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation