Abstract
Recently, how to infer one’s emotional state by nonverbal components has attracted great attention from the scientific community. If we can decode and visualize the emotional states from human brain activity, what an amazing thing that would be? The research in this paper found a way to decode and visualize different emotional states from human brain activity. In our experiments, at first, the power spectral density (PSD) was extracted from EEG signals evoked by visual stimulation of different emotional facial images. PSD can be viewed as a clue containing specific emotional states in human brain. After that, we use the conditional variational auto-encoder (VAE) to decode and visualize the emotional state, which takes the extracted PSD feature as input and generates the corresponding image. Specifically, VAE is a framework consisting of an encoder and a decoder. The former is used to learn low-dimension potential features of specific emotional state from the input PSD, and the later outputs an image containing the corresponding emotional state. Finally, our method was trained and tested on the EEG data from six subjects while they were looking at images from the Chinese Facial Affective Picture System (CFAPS) and obtained some promising results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bao, F.S., Liu, X., Zhang, C.: PyEEG: an open source python module for EEG/MEG feature extraction. Comput. Intell. Neurosci. (2011)
DeYoe, E.A., et al.: Mapping striate and extrastriate visual areas in human cerebral cortex. Proc. Natl. Acad. Sci. 93(6), 2382–2386 (1996)
Dieleman, S., van den Oord, A., Simonyan, K.: The challenge of realistic music generation: modelling raw audio at scale. In: Advances in Neural Information Processing Systems, pp. 7989–7999 (2018)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Gupta, A., Agarwal, A., Singh, P., Rai, P.: A deep generative framework for paraphrase generation. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Haynes, J.D., Rees, G.: Neuroimaging: decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7(7), 523 (2006)
Huth, A.G., de Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600), 453 (2016)
Huth, A.G., Lee, T., Nishimoto, S., Bilenko, N.Y., Vu, A.T., Gallant, J.L.: Decoding the semantic content of natural movies from human brain activity. Front. Syst. Neurosci. 10, 81 (2016)
Kavasidis, I., Palazzo, S., Spampinato, C., Giordano, D., Shah, M.: Brain2image: converting brain signals into images. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1809–1817. ACM (2017)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Lin, Y.P., et al.: EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)
Miller, E.K., Cohen, J.D.: An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24(1), 167–202 (2001)
Naselaris, T., Prenger, R.J., Kay, K.N., Oliver, M., Gallant, J.L.: Bayesian reconstruction of natural images from human brain activity. Neuron 63(6), 902–915 (2009)
Ray, W.J., Cole, H.W.: EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228(4700), 750–752 (1985)
Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-resolution images with VQ-VAE (2019)
Rozgić, V., Vitaladevuni, S.N., Prasad, R.: Robust EEG emotion classification using segment level decision fusion. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1286–1290. IEEE (2013)
Serban, I.V., et al.: A hierarchical latent variable encoder-decoder model for generating dialogues. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)
Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., Shah, M.: Deep learning human mind for automated visual classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6809–6817 (2017)
Tirupattur, P., Rawat, Y.S., Spampinato, C., Shah, M.: Thoughtviz: visualizing human thoughts using generative adversarial network. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 950–958. ACM (2018)
Vandenhende, S., De Brabandere, B., Neven, D., Van Gool, L.: A three-player GAN: generating hard samples to improve classification networks. arXiv preprint arXiv:1903.03496 (2019)
Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex 28(12), 4136–4160 (2017)
Xu, H., Plataniotis, K.N.: Affective states classification using EEG and semi-supervised deep learning approaches. In: 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6. IEEE (2016)
Yan, X., Yang, J., Sohn, K., Lee, H.: Attribute2Image: conditional image generation from visual attributes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 776–791. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_47
Acknowledgement
This work was supported by National Key R&D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), National Natural Science Foundation of China (61671193), Key Science and Technology Program of Zhejiang Province (2018C04012), Science and technology platform construction project of Fujian science and Technology Department (2015Y2001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Long, Y., Kong, W., Jin, X., Shang, J., Yang, C. (2019). Visualizing Emotional States: A Method Based on Human Brain Activity. In: Zeng, A., Pan, D., Hao, T., Zhang, D., Shi, Y., Song, X. (eds) Human Brain and Artificial Intelligence. HBAI 2019. Communications in Computer and Information Science, vol 1072. Springer, Singapore. https://doi.org/10.1007/978-981-15-1398-5_18
Download citation
DOI: https://doi.org/10.1007/978-981-15-1398-5_18
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1397-8
Online ISBN: 978-981-15-1398-5
eBook Packages: Computer ScienceComputer Science (R0)