Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

  • Giovanni SantiniEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11502)


The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications:
  • A picture can become a sort of score; graphic scores can have a different source of interpretation;

  • Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis.

The current version has been developed in a Virtual Reality (VR) environment.


Synesthesia Synthesizer Physical modeling Machine Learning Virtual reality 


  1. 1.
    Cytowic, E.R.: Synesthesia: A Union of the Senses, 2nd edn. MIT Press, Cambridge (2002). Scholar
  2. 2.
    Payling, D.: Visual Music Composition with Electronic Sound and Video. Staffordshire University (2014)Google Scholar
  3. 3.
    Shatter, G., Zueger, E., Nitschke, C.: A synaesthetic approach for a synthesizer interface based on genetic algorithm and fuzzy sets. In: Proceedings of International Computer Music Conference 2005 (ICMC 2005), pp. 664–667 (2005)Google Scholar
  4. 4.
    Margounakis, D., Politis, D.: Converting images to music using their colour properties. In: Proceedings of the 12th International Conference on Auditory Display (ICAD06), pp. 198–205 (2006)Google Scholar
  5. 5.
  6. 6.
    McClard, P.T.: Music generating system and method utilizing control of music based upon displayed color. US Brevetto US5689078A (1995)Google Scholar
  7. 7.
    Macedo, A.R.: Visible and audible spectrums-a proposal of correspondence. Proc. Artech 5, 168–171 (2010)Google Scholar
  8. 8.
    Rogowska, A.: Synaesthesia and Individual Differences. Cambridge University Press, Cambridge (2015)CrossRefGoogle Scholar
  9. 9.
    Unity. Accessed 4 Jan 2019
  10. 10.
    Wekinator - Instructions. Accessed 4 Jan 2019

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Hong Kong Baptist UniversityKowloon TongHong Kong

Personalised recommendations