A GAN-Based Data Augmentation Method for Multimodal Emotion Recognition

  • Yun Luo
  • Li-Zhen Zhu
  • Bao-Liang LuEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11554)


The lack of training data is an obstacle to build satisfactory multimodal emotion recognition models. Generative adversarial network (GAN) has recently shown great successes in generating realistic-like data. In this paper, we propose a GAN-based data augmentation method for enhancing the performance of multimodal emotion recognition models. We adopt conditional Boundary Equilibrium GAN (cBEGAN) to generate artificial differential entropy features of electroencephalography signal, eye movement data and their direct concatenations. The main advantage of cBEGAN is that it can overcome the instability of conventional GAN and has very quick converge speed. We evaluate our proposed method on two multimodal emotion datasets. The experimental results demonstrate that our proposed method achieves 4.6% and 8.9% improvements of mean accuracies on classifying three and five emotions, respectively.


EEG Eye movement Emotion recognition Generative adversarial network Data augmentation 



This work was supported in part by grants from the National Key Research and Development Program of China (Grant No. 2017YFB1002501), the National Natural Science Foundation of China (Grant No. 61673266), and the Fundamental Research Funds for the Central Universities.


  1. 1.
    Berthelot, D., Schumm, T., Metz, L.: BEGAN: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717 (2017)
  2. 2.
    Duan, R.N., Zhu, J.Y., Lu, B.L.: Differential entropy feature for EEG-based emotion classification. In: IEEE EMBS NER 2013, pp. 81–84. IEEE (2013)Google Scholar
  3. 3.
    Dyk, D.A.V., Meng, X.L.: The art of data augmentation. J. Comput. Graph. Stat. 10(1), 1–50 (2001)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS 2014, pp. 2672–2680 (2014)Google Scholar
  5. 5.
    Hartmann, K.G., Schirrmeister, R.T., Ball, T.: EEG-GAN: generative adversarial networks for electroencephalograhic (EEG) brain signals (2018)Google Scholar
  6. 6.
    Krell, M.M., Su, K.K.: Rotational data augmentation for electroencephalographic data. In: Engineering in Medicine and Biology Society, pp. 471–474 (2017)Google Scholar
  7. 7.
    Liu, W., Zheng, W.-L., Lu, B.-L.: Emotion recognition using multimodal deep learning. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9948, pp. 521–529. Springer, Cham (2016). Scholar
  8. 8.
    Lotte, F.: Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based brain-computer interfaces. Proc. IEEE 103(6), 871–890 (2015)CrossRefGoogle Scholar
  9. 9.
    Lu, Y., Zheng, W.L., Li, B., Lu, B.L.: Combining eye movements and EEG to enhance emotion recognition. In: IJCAI 2015, pp. 1170–1176 (2015)Google Scholar
  10. 10.
    Luo, Y., Lu, B.L.: EEG data augmentation for emotion recognition using a conditional Wasserstein GAN. In: IEEE EMBS 2018, pp. 2535–2538. IEEE (2018)Google Scholar
  11. 11.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  12. 12.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)CrossRefGoogle Scholar
  13. 13.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR 2016 (2016)Google Scholar
  14. 14.
    Wang, F., Zhong, S., Peng, J., Jiang, J., Liu, Y.: Data augmentation for EEG-based emotion recognition with deep convolutional neural networks. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10705, pp. 82–93. Springer, Cham (2018). Scholar
  15. 15.
    Zhao, L.M., Li, R., Zheng, W.L., Lu, B.L.: Classification of five emotions from EEG and eye movement signals: complementary representation properties. In: IEEE NER 2019. IEEE (2019)Google Scholar
  16. 16.
    Zheng, W.L., Lu, B.L.: Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transact. Auton. Ment. Dev. 7(3), 162–175 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Center for Brain-Like Computing and Machine Intelligence, Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognition EngineeringShanghai Jiao Tong UniversityShanghaiChina
  3. 3.Brain Science and Technology Research CenterShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations