An Interpreter of a Human Emotional State Based on a Neural-Like Hierarchical Structure

  • Konstantin V. SidorovEmail author
  • Natalya N. Filatova
  • Pavel D. Shemaev
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 874)


The paper considers possibility of hybrid system utilization that integrates the strategies of neural network models and methods of fuzzy data processing, for production rules construction and interpretation of biomedical signals by using the developed interpreter. The signals interpreter is based on the idea of growing pyramidal networks, which was adapted to work with fuzzy descriptions of objects. Objects fuzzy descriptions are generated by using informative attributes extracted from the time series attractors reconstructions. During training process, the class models are forming in interpreter’s hierarchical structure and then transforming into fuzzy statements serving as a set of production rules for fuzzy logic inference system. Fuzzy statements reflect the main characteristics of all objects from training set and presented in verbal terms understandable to experts. This paper provides detailed description of interpreter’s software realization and demonstrates construction algorithm of the neural-like hierarchical structure (NLHS) with production rules forming procedure. Article also contains the results of interpreter’s program implementation on biomedical signals represented by electroencephalography (EEG) and speech signals. All collected biomedical signals (from training and test sets) are characterize changes in emotional reactions of subjects undergoing audiovisual stimulation. Mathematical apparatus of attractor reconstruction for EEG and speech signals allows to monitor changes in the individual properties of this structures on perception stage and after stimulation. As a result of monitoring a number of regularities determined, which are reveal themselves in attractor’s characteristics and variations of emotional reactions correlated with them, and also with experts estimations as well as subjects self-estimations.


Interpreter of emotions Human emotions Software tool EEG Speech signal Attractor Neural-like hierarchical structure Fuzzy set Fuzzy sign Production rule Training set Test set 


  1. 1.
    Rangayyan, R.M.: Biomedical Signal Analysis, 2nd edn. Wiley-IEEE Press, New York (2015)CrossRefGoogle Scholar
  2. 2.
    Farhoudi, Z., Setayeshi, S., Rabiee, A.: Using learning automata in brain emotional learning for speech emotion recognition. Int. J. Speech Technol. 20(3), 553–562 (2017)CrossRefGoogle Scholar
  3. 3.
    Abhang, P.A., Gawali, B.W.: Correlation of EEG images and speech signals for emotion analysis. Br. J. Appl. Sci. Technol. 10(5), 1–13 (2015). Scholar
  4. 4.
    Filatova, N.N., Sidorov, K.V., Iliasov, L.V.: Automated system for analyzing and interpreting nonverbal information. Int. J. Appl. Eng. Res. 10(24), 45741–45749 (2015)Google Scholar
  5. 5.
    Fortin, P.E., Cooperstock, J.R.: Laughter and tickles: toward novel approaches for emotion and behavior elicitation. IEEE TAC 8(4), 508–521 (2017)Google Scholar
  6. 6.
    Lan, Z., Sourina, O., Wang, L., Liu, Y.: Real-time EEG-based emotion monitoring using stable features. Vis. Comput. 32(3), 347–358 (2016). Scholar
  7. 7.
    Filatova, N.N., Sidorov, K.V., Terekhin, S.A.: A software package for interpretation of nonverbal information by analyzing speech patterns or electroencephalogram. Softw. Syst. 111(3), 22–27 (2015). (in Russia, Programmnye Produkty i Sistemy)
  8. 8.
    Okada, G., Yonezawa, T., Kurita, K., Tsumura, N.: Monitoring emotion by remote measurement of physiological signals using an RGB camera. ITE Trans. MTA 6(1), 131–137 (2018)Google Scholar
  9. 9.
    Bobkov, A.S., Dmitriev, A.S., Zaboleeva-Zotova, A.V., Orlova, Y., Rozaliev, V.L.: Detailed analysis of postures and gestures for the identification of human emotional reactions. WASJ 24, 151–158 (2013)Google Scholar
  10. 10.
    Kolesnikov, A.V.: Gibrid intelligent systems: a theory and development technology. St. Petersburg State Polytech. Univ. Publ., St. Petersburg (2001). (in Russia, Gibridnye Intellektualnye Sistemy: Teoriya i Tekhnologiya Razrabotki)Google Scholar
  11. 11.
    Filatova, N.N., Khaneev, D.M.: Use of neurolike structures for automatic generation of hypotheses for classification rules. Fuzzy Syst. Soft Comput. 8(1), 27–44 (2013). (in Russia, Nechetkie Sistemy i Mjagkie Vychislenija)Google Scholar
  12. 12.
    Khaneev D.M., Filatova N.N.: The pyramidal network for classification of objects, presented by fuzzy features. Eng. Sci. 134(9), 45–49 (2012). Izvestiya SFedU. (in Russia, Izvestiya SFedU. Tekhnicheskie Nauki)Google Scholar
  13. 13.
    Gladun, V.P.: Growing pyramidal networks. News Artif. Intell. 2(1), 30–40 (2004). (in Russia, Novosti Iskusstvennogo Intellekta)Google Scholar
  14. 14.
    Filatova, N.N., Khaneev, D.M., Sidorov, K.V.: Signals interpreter based on neural-like hierarchical structure. Softw. Syst. 105(1), 92–97 (2014). (in Russia, Programmnye Produkty i Sistemy)Google Scholar
  15. 15.
    Rabinovich, M.I., Muezzinoglu, M.K.: Nonlinear dynamics of the brain: emotion and cognition. Adv. Phys. Sci. 180(4), 371–387 (2010). (in Russia, Uspekhi Fizicheskikh Nauk)CrossRefGoogle Scholar
  16. 16.
    Filatova N.N., Sidorov K.V., Shemaev P.D.: Prediction properties of attractors based on their fuzzy trend. In: Abraham, A., et al. (eds.) Proceedings of the Second International Scientific Conference “Intelligent Information Technologies for Industry” (IITI 2017) Advances in Intelligent Systems and Computing, vol. 679, pp. 244–253. Springer, Switzerland (2018). Scholar
  17. 17.
    Filatova, N.N., Sidorov, K.V., Terekhin, S.A., Vinogradov, G.P.: The system for the study of the dynamics of human emotional response using fuzzy trends. In: Abraham, A., et al. (eds.) Proceedings of the First International Scientific Conference “Intelligent Information Technologies for Industry” (IITI 2016), Advances in Intelligent Systems and Computing, vol. 451, part III, pp. 175–184. Springer, Switzerland (2016). Scholar
  18. 18.
    Filatova, N.N., Sidorov, K.V.: Interpretation of the emotion characteristics through the analysis of attractors reconstructed on EEG signals. Fuzzy Syst. Soft Comput. 11(1), 57–76 (2016). (in Russia, Nechetkie Sistemy i Mjagkie Vychislenija)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Konstantin V. Sidorov
    • 1
    Email author
  • Natalya N. Filatova
    • 1
  • Pavel D. Shemaev
    • 1
  1. 1.Department of Information TechnologiesTver State Technical UniversityTverRussia

Personalised recommendations