When Siri Knows How You Feel: Study of Machine Learning in Automatic Sentiment Recognition from Human Speech

  • L. ZhangEmail author
  • E. Y. K. Ng
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 887)


Opinions and sentiments are essential to human activities and have a wide variety of applications. As many decision makers turn to social media due to large volume of opinion data available, efficient and accurate sentiment analysis is necessary to extract those data. Hence, text sentiment analysis has recently become a popular field and has attracted many researchers. However, extracting sentiments from audio speech remains a challenge. This project explored the possibility of applying supervised Machine Learning in recognizing sentiments in English utterances on a sentence level. In addition, the project also aimed to examine the effect of combining acoustic and linguistic features on classification accuracy. Six audio tracks were randomly selected to be training data from 40 YouTube videos (monologue) with strong presence of sentiments. Speakers expressed sentiments towards products, films, or political events. These sentiments were manually labelled as negative and positive based on independent judgment of three experimenters. A wide range of acoustic and linguistic features were then analyzed and extracted using sound editing and text mining tools, respectively. A novel approach was proposed, which used a simplified sentiment score to integrate linguistic features and estimate sentiment valence. This approach improved negation analysis and hence increased overall accuracy. Results showed that when both linguistic and acoustic features were used, accuracy of sentiment recognition improved significantly, and that excellent prediction was achieved when the four classifiers were trained, respectively, namely, kNN, SVM, Neural Network, and Naïve Bayes. Possible sources of error and inherent challenges of audio sentiment analysis were discussed to provide potential directions for future research.


Sentiment analysis Natural language processing Machine learning Affective computing Data analytics Speech processing Computational linguistic 



I would like to express my sincere gratitude to my project supervisor Professor Eddie Ng for being open-minded about my project topic, without which I could not have been able to delve deep into my field of interest. His insightful suggestions and unwavering support has guided me through doubts and difficulties.


  1. 1.
    Liu, B.: Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press, Cambridge (2015)CrossRefGoogle Scholar
  2. 2.
    Lee, C.M., Narayanan, S.S.: Toward detecting emotions in spoken dialogs. IEEE Trans. Speech Audio Process. 13(2), 293 (2005)CrossRefGoogle Scholar
  3. 3.
    Kaushik, L., Sangwan, A., Hansen, J.H.L.: A Holistic Lexicon-Based Approach to Opinion Mining. IEEE (2013)Google Scholar
  4. 4.
    Ding, X., Liu, B., Yu, P.S.: A holistic lexicon-based approach to opinion mining. In: Proceedings of the Conference on Web Search and Web Data Mining (WSDM-2008) (2008)Google Scholar
  5. 5.
    Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2004), Aug 22–25, 2004, Seattle, Washington, USAGoogle Scholar
  6. 6.
    Demšar, J., Curk, T., Erjavec, A.: Orange: data mining toolbox in Python. J. Mach. Learn. Res. 14, 2349–2353 (2013)zbMATHGoogle Scholar
  7. 7.
  8. 8.
    Tape, T.G.: Interpreting diagnostic tests. University of Nebraska Medical Center (n.d).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Anglo-Chinese Junior CollegeSingaporeSingapore
  2. 2.College of EngineeringNanyang Technological University (NTU)SingaporeSingapore

Personalised recommendations