Decoding Generic Visual Representations from Human Brain Activity Using Machine Learning

  • Angeliki PapadimitriouEmail author
  • Nikolaos Passalis
  • Anastasios Tefas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11131)


Among the most impressive recent applications of neural decoding is the visual representation decoding, where the category of an object that a subject either sees or imagines is inferred by observing his/her brain activity. Even though there is an increasing interest in the aforementioned visual representation decoding task, there is no extensive study of the effect of using different machine learning models on the decoding accuracy. In this paper we provide an extensive evaluation of several machine learning models, along with different similarity metrics, for the aforementioned task, drawing many interesting conclusions. That way, this paper (a) paves the way for developing more advanced and accurate methods and (b) provides an extensive and easily reproducible baseline for the aforementioned decoding task.


Neural decoding Deep visual representations 


  1. 1.
    Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincott Williams & Wilkins, Philadelphia (2005)Google Scholar
  2. 2.
    Huettel, S.A., Song, A.W., McCarthy, G., et al.: Functional Magnetic Resonance Imaging, vol. 1. Sinauer Associates Sunderland, MA (2004)Google Scholar
  3. 3.
    Tagliazucchi, E., Laufs, H.: Decoding wakefulness levels from typical fMRI resting-state data reveals reliable drifts between wakefulness and sleep. Neuron 82(3), 695–708 (2014)CrossRefGoogle Scholar
  4. 4.
    Horikawa, T., Tamaki, M., Miyawaki, Y., Kamitani, Y.: Neural decoding of visual imagery during sleep. Science 340(6132), 639–642 (2013)CrossRefGoogle Scholar
  5. 5.
    Gwin, J.T., Gramann, K., Makeig, S., Ferris, D.P.: Removal of movement artifact from high-density EEG recorded during walking and running. J. Neurophysiol. 103(6), 3526–3534 (2010)CrossRefGoogle Scholar
  6. 6.
    Presacco, A., Goodman, R., Forrester, L., Contreras-Vidal, J.L.: Neural decoding of treadmill walking from noninvasive electroencephalographic signals. J. Neurophysiol. 106(4), 1875–1887 (2011)CrossRefGoogle Scholar
  7. 7.
    Fink, A., Graif, B., Neubauer, A.C.: Brain correlates underlying creative thinking: EEG alpha activity in professional vs. novice dancers. NeuroImage 46(3), 854–862 (2009)CrossRefGoogle Scholar
  8. 8.
    Cruz-Garza, J.G., Hernandez, Z.R., Nepaul, S., Bradley, K.K., Contreras-Vidal, J.L.: Neural decoding of expressive human movement from scalp electroencephalography (EEG). Front. Hum. Neurosci. 8, 188 (2014)CrossRefGoogle Scholar
  9. 9.
    Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nature Commun. 8, 15037 (2017)CrossRefGoogle Scholar
  10. 10.
    Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Cortex 1–25Google Scholar
  11. 11.
    Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S., van Lier, R., van Gerven, M.A.: Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 4246–4257 (2017)Google Scholar
  12. 12.
    Wen, H., Shi, J., Chen, W., Liu, Z.: Transferring and generalizing deep-learning-based neural encoding models across subjects. NeuroImage 176, 152–163 (2018)CrossRefGoogle Scholar
  13. 13.
    Nasrabadi, N.M.: Pattern recognition and machine learning. J. Electron. Imaging 16(4), 049901 (2007)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hastie, T., Tibshirani, R.: Discriminant adaptive nearest neighbor classification and regression. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 409–415 (1996)CrossRefGoogle Scholar
  15. 15.
    Mosteller, F., Tukey, J.W.: Data Analysis and Regression: A Second Course in Statistics. Addison-Wesley Series in Behavioral Science: Quantitative Methods. Addison-Wesley, Mass (1977)Google Scholar
  16. 16.
    Hoerl, A.E., Kennard, R.W.: Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1), 55–67 (1970)CrossRefGoogle Scholar
  17. 17.
    Shawe-Taylor, J., Cristianini, N., et al.: Kernel Methods for Pattern Analysis. Cambridge University Press, New York (2004)CrossRefGoogle Scholar
  18. 18.
    Haykin, S.S.: Neural Networks and Learning Machines, vol. 3. Pearson Education, Upper Saddle River (2009)Google Scholar
  19. 19.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (2014)Google Scholar
  20. 20.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  22. 22.
    Vovk, V.: Kernel ridge regression. In: Empirical Inference, pp. 105–116 (2013)CrossRefGoogle Scholar
  23. 23.
    Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Chollet, F., et al.: Keras (2015).
  25. 25.
    Graves, A., Mohamed, A.-r., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)Google Scholar
  26. 26.
    Pan, S.J., Yang, Q., et al.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  27. 27.
    Passalis, N., Tefas, A.: Learning deep representations with probabilistic knowledge transfer. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 283–299. Springer, Cham (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Artificial Intelligence and Information Analysis LaboratoryAristotle University of ThessalonikiThessalonikiGreece

Personalised recommendations