Making Music Composing Easier for Amateurs: A Hybrid Machine Learning Approach

  • Jiaming XuEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 800)


Creating your own musical pieces is one of the most attractive ways to enjoy music. However, many musically untrained people lack the basic musical skills to do so. In this paper, we seek to explore how machine learning algorithms can enable musically untrained users to create their own music.

To achieve this, we propose a Neural Hidden Markov Model (NHMM). It is a hybrid of a Hidden Markov model (HMM) and Convolution neural network (CNN) with a Long Short-Term Memory (LSTM) neural network. This model takes users’ original musical ideas in an easy intuitive way and automatically modifies the input to generate musically appropriate melodies as output. We further extend the model to allow users to specify the magnitude of revision, duration of music segment to be revised, choice of music genres, popularity of songs, and co-creation of songs in social settings. These extensions enhance user understanding of music theory, enrich their experience of self-learning, and enable social aspects of music creation. The model is trained using MIDI files of existing songs. We also conduct experiments on melody generation.

We also hope to design a mobile application with an intuitive, interactive, and graphical user interface, which is suitable for the elderly and young children. Different from most existing literature focusing on computer music composition itself, our research and application aim at using computers to aid human composition and enriching the music education of musically untrained people.


  1. 1.
    Power, S.D., Falk, T.H., Chau, T.: Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy. J. Neural Eng. 7(7), 26002 (2010)CrossRefGoogle Scholar
  2. 2.
    Gillick, J., Tang, K., Keller, R.M.: Machine learning of Jazz grammars. Comput. Music J. 34(3), 56–66 (2010)CrossRefGoogle Scholar
  3. 3.
    Kitahara, T., Tsuchiya, Y.: A Machine Learning Approach to Support Music Creation by Musically Untrained PeopleGoogle Scholar
  4. 4.
    Mozer, M.C., Soukup, T.: Connectionist music composition based on melodic and stylistic constraints. In: Advances in Neural Information Processing Systems (1990)Google Scholar
  5. 5.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  6. 6.
    Eck, D., Schmidhuber, J.: A first look at music composition using LSTM recurrent neural networks. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale (2007)Google Scholar
  7. 7.
    Sturm, B.L., et al.: Music transcription modelling and composition using deep learning (2016)Google Scholar
  8. 8.
    Liang, F., et al.: Automatic stylistic composition of Bach Chorales with deep LSTM (2017)Google Scholar
  9. 9.
    Coca, A.E., Correa, D.C., Zhao, L.: Computer-aided music composition with LSTM neural network and chaotic inspiration. In: International Joint Conference on Neural Networks (2014)Google Scholar
  10. 10.
    Zhang, W., et al.: Parallel distributed processing model with local space-invariant interconnections and its optical architecture. Appl. Opt. 29(32), 4790–4797 (1990)CrossRefGoogle Scholar
  11. 11.
    Mirghafori, N., Morgan, N., Bourlard, H.: Parallel training of MLP probability estimators for speech recognition: a gender-based approach. In: Neural Networks for Signal Processing (1994)Google Scholar
  12. 12.
    Krakovna, V., Doshivelez, F.: Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models (2016)Google Scholar
  13. 13.
    Ke, T., et al.: Unsupervised Neural Hidden Markov Models (2016)Google Scholar
  14. 14.
    Deng, J., Kwok, Y.K.: Automatic Chord estimation on seventhsbass Chord vocabulary using deep neural network. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2016)Google Scholar
  15. 15.
    Verbeurgt, K., Dinolfo, M., Fayer, M.: Extracting Patterns in Music for Composition via Markov Chains, pp. 1123–1132. Springer, Berlin/Heidelberg (2004)CrossRefGoogle Scholar
  16. 16.
    Trentin, E., Gori, M.: A survey of hybrid ANN/HMM models for automatic speech recognition. Neurocomputing 37(1), 91–126 (2001)CrossRefGoogle Scholar
  17. 17.
    Rabiner, L., Juang, B.: An introduction to hidden Markov models. Current protocols in bioinformatics/editoral board, Andreas D. Baxevanis …[et al.], 2007. Appendix 3(Appendix 3): p. Appendix 3AGoogle Scholar
  18. 18.
    Netzer, O., Lattin, J.M., Srinivasan, V.: A hidden Markov model of customer relationship dynamics. Mark. Sci. 27(2), 185–204 (2008)CrossRefGoogle Scholar
  19. 19.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Mozer, M.C.: Induction of multiscale temporal structure. In: International Conference on Neural Information Processing Systems (1991)Google Scholar
  21. 21.
    Todd, P.M., Loy, G.: Creation by Refinement and the Problem of Algorithmic Music Composition (1991)Google Scholar
  22. 22.
    Hild, H., Feulner, J., Menzel, W.: HARMONET: a neural net for harmonizing chorales in the style of J. S. Bach. In: Advances in Neural Information Processing Systems (1991)Google Scholar
  23. 23.
    Bellgard, M.I., Tsang, C.P.: Harmonising music using a network of Boltzmann machines (1992)Google Scholar
  24. 24.
    Johnson, M.: Why doesn’t EM find good HMM POS-taggers? In: EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Prague, 28–30 June 2007Google Scholar
  25. 25.
    Sutskever, I.: Training Recurrent Neural Networks. Doctoral (2013)Google Scholar
  26. 26.
    Liu, I., Ramakrishnan, B.: Bach in 2014: Music Composition with Recurrent Neural Network. Eprint Arxiv (2014)Google Scholar
  27. 27.
    Bengio, Y., et al.: Global optimization of a neural network-hidden Markov model hybrid. IEEE Trans. Neural Netw. 3(2), 252–259 (1992)CrossRefGoogle Scholar
  28. 28.
    Welch, L.R.: Hidden Markov models and the Baum-Welch algorithm. IEEE Inf. Theory Soc. Newsl. 53(2), 194–211 (2003)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Columbia Business SchoolNew YorkUSA

Personalised recommendations