Deep Transfer Learning for Cross-subject and Cross-experiment Prediction of Image Rapid Serial Visual Presentation Events from EEG Data

  • Mehdi HajinorooziEmail author
  • Zijing Mao
  • Yuan-Pin Lin
  • Yufei Huang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


Transfer learning (TL) has gained significant interests recently in brain computer interface (BCI) as a key approach to design robust predictors for cross-subject and cross-experiment prediction of the brain activities in response to cognitive events. We carried out in this.aper the first comprehensive investigation of the transferability of deep convolutional neural network (CNN) for cross-subject and cross-experiment prediction of image Rapid Serial Visual Presentation (RSVP) events. We show that for both cross-subject and cross-experiment predictions, all convolutional layers and fully connected layers contain both general and subject/experiment-specific features and transfer learning with weights fine-tuning can improve the prediction performance over that without transfer. However, for cross-subject prediction, the convolutional layers capture more subject-specific features, whereas for cross-experiment prediction, the convolutional layers capture more general features across experiment. Our study provides important information that will guide the design of more sophisticated deep transfer learning algorithms for EEG based classifications in BCI applications.


Transfer learning Deep convolutional neural networks EEG signals 


  1. 1.
    Bigdely-Shamlo, N., Vankov, A., Ramirez, R.R., Makeig, S.: Brain activity-based image classification from rapid serial visual presentation. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 432–441 (2008)CrossRefGoogle Scholar
  2. 2.
    Cecotti, H., Eckstein, M.P., Giesbrecht, B.: Single-trial classification of event-related potentials in rapid serial visual presentation tasks using supervised spatial filtering. IEEE Trans. Neural Netw. Learn. Syst. 25, 2030–2042 (2014)CrossRefGoogle Scholar
  3. 3.
    Lei, X., Yang, P., Yao, D.: An empirical Bayesian framework for brain–computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 17, 521–529 (2009)CrossRefGoogle Scholar
  4. 4.
    Rivet, B., Souloumiac, A., Attina, V., Gibert, G.: xDAWN algorithm to enhance evoked potentials: application to brain–computer interface. IEEE Trans. Biomed. Eng. 56, 2035–2043 (2009)CrossRefGoogle Scholar
  5. 5.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  6. 6.
    Cook, D., Feuz, K.D., Krishnan, N.C.: Transfer learning for activity recognition: a survey. Knowl. Inf. Syst. 36(3), 537–556 (2013)CrossRefGoogle Scholar
  7. 7.
    Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.-R., Jaitly, N., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012)CrossRefGoogle Scholar
  8. 8.
    Shao, L., Zhu, F., Li, X.: Transfer learning for visual categorization: a survey (2014)Google Scholar
  9. 9.
    Razavian, A.S., et al.: CNN features off-the-shelf: an astounding baseline for recognition. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE (2014)Google Scholar
  10. 10.
    Donahue, J., et al.: Decaf: a deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531 (2013)
  11. 11.
    Yosinski, J., et al.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems (2014)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  13. 13.
    Patel, A.B., Nguyen, T., Baraniuk, R.G.: A probabilistic theory of deep learning. arXiv preprint arXiv:1504.00641 (2015)
  14. 14.
    Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Mirowski, P.W., et al.: Comparing SVM and convolutional networks for epileptic seizure prediction from intracranial EEG. In: IEEE Workshop on Machine Learning for Signal Processing, MLSP 2008. IEEE (2008)Google Scholar
  16. 16.
    Aytar, Y., Zisserman, A.: Tabula rasa: model transfer for object category detection. In: 2011 IEEE International Conference on Computer Vision (ICCV). IEEE (2011)Google Scholar
  17. 17.
    Li, X.: Regularized adaptation: theory, algorithms and applications. Ph.D. thesis, University of Washington, USA (2007)Google Scholar
  18. 18.
    Yang, J., Yan, R., Hauptmann, A.: Adapting SVM classifiers to data with shifted distributions. In: ICDM Workshops 2007 (2007)Google Scholar
  19. 19.
    U.S Department of Defense Office of the Secretary of Defense: Code of federal regulations, protection of human subjects. 32 CFR 219 (1999)Google Scholar
  20. 20.
    U.S. Department of the Army. Use of volunteers as subjects of research. AR 70-25. Government Printing Office, Washington, DC (1990)Google Scholar
  21. 21.
    Hajinoroozi, M., et al.: Feature extraction with deep belief networks for driver’s cognitive states prediction from EEG data. In: 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP). IEEE (2015)Google Scholar
  22. 22.
    Hajinoroozi, M., Mao, Z., Huang, Y.: Prediction of driver’s drowsy and alert states from EEG signals with deep learning. In: IEEE 6th International Workshop on Computational Advances in Multi-sensor Adaptive Processing (CAMSAP). IEEE (2015)Google Scholar
  23. 23.
    Hajinoroozi, M., Mao, Z., Jung, T.P., Lin, C.T., Huang, Y.: EEG-based prediction of driver’s cognitive performance by deep convolutional neural network. Signal Process. Image Commun. (2016)Google Scholar
  24. 24.
    Touryan, J., Apker, G., Kerick, S., Lance, B., Ries, A.J., McDowell, K.: Translation of EEG-based performance prediction models to rapid serial visual presentation tasks. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS (LNAI), vol. 8027, pp. 521–530. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39454-6_56 CrossRefGoogle Scholar
  25. 25.
    Touryan, J., Apker, G., Lance, B.J., Kerick, S. E., Ries, A. J. McDowell, K.: Estimating endogenous changes in task performance from EEG. In: Using Neurophysiological Signals That Reflect Cognitive or Affective State, p. 268 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Mehdi Hajinoroozi
    • 1
    Email author
  • Zijing Mao
    • 1
  • Yuan-Pin Lin
    • 2
  • Yufei Huang
    • 1
  1. 1.University of Texas at San AntonioSan AntonioUSA
  2. 2.Institute of Medical Science and TechnologyNational Sun Yat-sen UniversityKaohsiungTaiwan

Personalised recommendations