Multimedia Tools and Applications

, Volume 61, Issue 2, pp 489–518 | Cite as

Algorithmic approach to sonification of classical Chinese poetry

  • Chih-Fang HuangEmail author
  • Hsiang-Pin Lu
  • Jenny Ren


The classical Chinese poetry is a remarkable form of art in traditional Chinese character. However, it is difficult for people who are unfamiliar with ancient Chinese to experience the artistic content of the poetry. In this study, a sonification scheme, Tx2Ms (Text-to-Music), is proposed to extract the poetry features between lines in verses; moreover, dynamics and interval relations are modeled to map those features to the movement of multi-dimensional musical elements such as durations. This conversion is based on poetry intonation and acoustic analysis of the pronunciations of poems; and a stochastic compositional algorithm is created by applying a Markov chain. In addition, the best pentatonic mode for a specific poem is recommended according to the formants analysis. Therefore, the sonification of classical Chinese poetry not only provides a novel way for people to appreciate Chinese poetry but also enriches the state of mind and imagery in the delivery process, and the experiment results show that the proposed system is successfully accepted by most people.


Sonification Markov chain Classical Chinese poetry Algorithmic composition 



The authors would like to appreciate the support from National Science Council projects of Taiwan: NSC99-2410-H-155-035-MY2.


  1. 1.
    Bain R (2006) The AIMS Project: creative experiments in musical sonification. International Computer Music Conference ICMC 2006, New Orleans, Louisiana, USA (November)Google Scholar
  2. 2.
    Ballas A (1994) Delivery of information through sound. In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison-Wesley, Reading, pp 79–94Google Scholar
  3. 3.
    Barrass S, Kramer G (1999) Using sonification. Multimedia systems 7(1) (accessed 3 April 2011)
  4. 4.
    Blattner M, Papp L, Glinert P (1994) Sonic enhancement of two-dimensional graphics displays. In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison-Wesley, Reading, pp 447–470Google Scholar
  5. 5.
    Brewster A, Wright C, Edwards N (1993) An evaluation of earcons for use in auditory human-computer interfaces. In: Ashlund S, Mullet K, Henderson A, Hollnagel E, White T (eds) Proceedings of InterCHI’93. ACM, Amsterdam, pp 222–227Google Scholar
  6. 6.
    Chadabe J (1997) Electric sound: the past and promise of electronic. Music Prentice Hall, pp 178, ISBN 9780133032314Google Scholar
  7. 7.
    Chen H-H (2004) Introduction to composition in Chinese Poetry-Tsu. Wu-Nan Book Co., TaipeiGoogle Scholar
  8. 8.
    Clement J (1998) Learning harmonic progression using Markov models. (accessed 10 May 2009)
  9. 9.
    Coagula software website: (accessed 8 December 2010)
  10. 10.
    Cope D (1992) Computer modeling of musical intelligence in experiments in musical intelligence. Comp Music J 16(2):69–83CrossRefGoogle Scholar
  11. 11.
    Cope D (2004) virtual music: computer synthesis of musical style. MIT PressGoogle Scholar
  12. 12.
    Doornbusch P (2002) A brief survey of mapping in algorithmic composition. International Computer Music Conference, GöteborgGoogle Scholar
  13. 13.
    Durate J, Hsiao S-C, Huang C-F, and Winsor P (2006) The applications of Sieve Theory in Algorithmic Composition using MAX/MSP and BASIC. The 2nd International Conference WOCMAT, Workshop for Computer Music and Audio Technology, Taipei, Taiwan, pp 96–99 (March)Google Scholar
  14. 14.
    Fagerlönn J, Liljedahl M (2009) Awesome sound design tool: a web based utility that invites end users into the audio design process. In Proceedings of the 15th International Conference on Auditory Display (ICAD2009), Copenhagen, Denmark (May)Google Scholar
  15. 15.
    Farbood M, Schoner B (2001) Analysis and synthesis of palestrina-style counterpoint using Markov chains. International Computer Music Conference, Havana, Cuba, pp 18–22 (September)Google Scholar
  16. 16.
    Fowler A (1986) Franz Liszt’s Petrarch Sonnets: the persistent poetic problem. Indiana Theor Rev 7(2):48–68Google Scholar
  17. 17.
    Franz M (1998) Markov chains as tools for jazz improvisation analysis. Master thesis, Virginia Polytechnic Institute and State UniversityGoogle Scholar
  18. 18.
    Garzonis S, Jones S, Jay T, O’Neill E (2009) Auditory icon and earcon mobile service notifications: intuitiveness, learnability, memorability and preference. Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009,ACM, Boston, MA, USA, 4–9 April 2009Google Scholar
  19. 19.
    Gaser C, Schlaug G (2003) Brain structures differ between musicians and non-musicians. J Neurosci 23(27):9240–9245 (October)Google Scholar
  20. 20.
    Goina M, Polotti P (2008) Elementary gestalts for gesture sonification. Proceedings of the 2008 International Conference on New Interfaces for Musical Expression (NIME–08). Genova, Italy, pp 150–153 (June)Google Scholar
  21. 21.
    Hiller L, Isaacson L (1959) Experimental music. McGraw-HillGoogle Scholar
  22. 22.
    Hsu C-Y (1997) Theory on classical Chinese poetry composition. Hungyeh Publishing, TaipeiGoogle Scholar
  23. 23.
    Hunter P, Schellenberg E, Schimmack U (2008) Mixed affective responses to music with conflicting cues. Cogn Emotion 22(2):327–352CrossRefGoogle Scholar
  24. 24.
    Hussein K, Tilevich E, Bukvic I, Kim S (2009) Sonification design guidelines to enhance program comprehension. In Proceedings of ICPC‘2009, pp 120–129Google Scholar
  25. 25.
    Image-to-Sound Mapping website: (accessed 8 December 2010)
  26. 26.
    Jeon M, Davison B, Nees M, Wilson J, Walker B (2009) Enhanced auditory menu cues improve dual task performance and are preferred with in-vehicle technologies. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ‘09). ACM, New York, NY, USA, pp 91–98Google Scholar
  27. 27.
    Kaper G, Wiebel E, Tipei S (1999) Data Sonification and Sound Visualization. Computing in Science and Engineering 1, no. 4, (accessed 9 May 2009)
  28. 28.
    Kramer G (1994) Some organizing principles for representing data with sound. In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison-Wesley, Reading, pp 185–221Google Scholar
  29. 29.
    Kramer G (1994) An introduction to auditory display. In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison-Wesley, Reading, pp 1–77Google Scholar
  30. 30.
    Kramer G et al (1999) Sonification report: status of the field and research agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display. (accessed 3 April 2011)
  31. 31.
    Langston P (1989) Six techniques for algorithmic music composition. 15th International Computer Music Conference (ICMC) in Columbus, Ohio (November)Google Scholar
  32. 32.
    Lerdahl F (2001) The sounds of poetry viewed as music. Ann N Y Acad Sci 337–354 (June)Google Scholar
  33. 33.
    McAlpine K, Miranda E, Hoggar S (1999) Making music with algorithms: a case-study system. Comp Music J 23(2):19–30CrossRefGoogle Scholar
  34. 34.
    McCormack J (1996) Grammar Based Music Composition. In: Stocker R et al (eds) Complex systems 96: from local interactions to global phenomena. ISO, Amsterdam, pp 320–336Google Scholar
  35. 35.
    Mitchell D (1985) Gustav Mahler: songs and symphonies of life and death: interpretations and annotations. University of California Press (November)Google Scholar
  36. 36.
    Moore R (1990) Chapter five: composing. In Elements of computer music. Prentice-Hall, Englewood CliffsGoogle Scholar
  37. 37.
    Rahn J (1980) Basic atonal theory. New York: Longman; London and Toronto: Prentice Hall International, ISBN 0-02-873160-3. Reprinted 1987, New York: Schirmer Books; London: Collier MacmillanGoogle Scholar
  38. 38.
    Rothstein J (1995) MIDI: A Comprehensive Introduction, Computer Music and Digital Audio Series, Vol.7, A-R Editions, Inc.Google Scholar
  39. 39.
    Saue S (2000) A model for interaction in exploratory sonification displays. In Proceedings of the International Conference on Auditory Display (ICAD2000), in Atlanta, Georgia (April)Google Scholar
  40. 40.
    Scaletti C (1994) Sound synthesis algorithms for auditory data representations. In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison-Wesley, Reading, pp 223–251Google Scholar
  41. 41.
    Shan K, Chiu S-C (2010) Algorithmic compositions based on discovered musical patterns. J Multimed Tools Appl 46:1–23CrossRefGoogle Scholar
  42. 42.
    Spiliotopoulos D, Stavropoulou P, Kouroupetroglou G (2009) Acoustic rendering of data tables using earcons and prosody for document accessibility. In: Stephanidis C (ed) Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction. Part III: Applications and Services (UAHCI ‘09). Springer, Berlin, pp 587–596CrossRefGoogle Scholar
  43. 43.
    Sturm L (2001) Synthesis and algorithmic composition techniques derived from particle physics. Proc. 8th Biennial Arts Tech. Symp., Connecticut College, New LondonGoogle Scholar
  44. 44.
    Vazquez-Alvarez Y, Oakley I, and Brewster A (2010) Urban sound gardens: supporting overlapping audio landmarks in exploratory environments. In Proceedings of multimodal location based techniques for extreme navigation workshop, Pervasive, Helsinki, Finland, pp 37–38Google Scholar
  45. 45.
    Verbeurgt K, Dinolfo M, and Fayer M (2004) Extracting patterns in music for composition via Markov chains. Innovations in Applied Artificial Intelligence. 17th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, IEA/AIE 2004 Ottawa, Canada, pp 1123–1132 (May)Google Scholar
  46. 46.
    Verron C, Aramaki M, Kronland-Martinet R, Pallone G (2009) Analysis/synthesis and spatialization of noisy environmental sounds. In Proceedings of the 15th International Conference on Auditory Display (ICAD2009), Copenhagen, Denmark, pp 36–40 (May)Google Scholar
  47. 47.
    Walker N, Cothran T (2003) Sonification sandbox: a graphical toolkit for auditory graphs. In Proceedings of the International Conference on Auditory Display (ICAD2003), Boston, USA (July)Google Scholar
  48. 48.
    Walker B, Kogan A (2009) Spearcon performance and preference for auditory menus on a mobile phone. In: Stephanidis C (ed) Proceedings of the 5th International on ConferenceUniversal Access in Human-Computer Interaction. Part II: Intelligent and Ubiquitous Interaction Environments (UAHCI ‘09). Springer, Berlin, pp 445–454Google Scholar
  49. 49.
    Winsor P (1989) The computer composer’s toolbox. Windcrest Books, Blue Ridge SummitGoogle Scholar
  50. 50.
    Winsor P (1990) Computer music in C. Blue Ridge Summit, Penn. Windcrest BooksGoogle Scholar
  51. 51.
    Winsor P (1992) Automated music composition. University of North Texas Press, DentonGoogle Scholar
  52. 52.
    Wu X, Li Z-N (2008) A study of image-based music composition. 2008 IEEE International Conference on Multimedia and Expo, ICME 2008 Proceedings, Burnaby, BC, pp 1345–1348Google Scholar
  53. 53.
    Yang C-C (2004) A study of constructing chinese poetry knowledge base—Su-Shi’s classical poetry knowledge base system. Master thesis, National Chiao Tung UniversityGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Department of Information CommunicationYuan Ze UniversityChung-LiRepublic of China
  2. 2.Department of Mechanical EngineeringNational Chiao-Tung UniversityHsinchuRepublic of China
  3. 3.Institute of MusicNational Chiao-Tung UniversityHsinchuRepublic of China

Personalised recommendations