Human-centered image classification via a neural network considering visual and biological features

  • Kazaha HoriiEmail author
  • Keisuke Maeda
  • Takahiro Ogawa
  • Miki Haseyama


In this paper, we propose a human-centered image classification via a neural network considering visual and biological features. The proposed method has two novelties. Firstly, we apply Group-Sparse Local Fisher Discriminant Analysis (GS-LFDA) to biological features. GS-LFDA realizes dimensionality reduction and noise elimination for biological features with consideration of local structures and class information. Secondly, we construct a Canonical Correlation Analysis (CCA)-based hidden layer via Discriminative Locality Preserving CCA (DLPCCA). DLPCCA transforms visual features into effective features by considering the relationships with biological information and class information. The CCA-based hidden layer enables transformation of visual features into effective features for image classification from a small number of training samples. Furthermore, once the projection can be obtained in the training phase, elimination of the need for biological data acquisition in the test phase is realized. This is another merit of our method.


Image classification Neural network Biological information Group-sparse local fisher discriminant analysis Discriminative locality preserving canonical correlation analysis 



In this research, we used inspection data that were provided by East Nippon Expressway Company Limited. This work was partly supported by JSPS KAKENHI Grant Number JP17H01744 and MIC/SCOPE #181601001 and MIC/SCOPE #181503004.


  1. 1.
    Ahmed S, Merino LM, Mao Z, Meng J, Robbins K, Huang Y (2013) A deep learning method for classification of images RSVP events with EEG data. In: Proceedings of IEEE global conference on signal and information processing, pp 33–36Google Scholar
  2. 2.
    Balconi M, Vanutelli M (2016) Hemodynamic (fNIRS) and EEG (N200) correlates of emotional inter-species interactions modulated by visual and auditory stimulation. Sci Rep 6:23083CrossRefGoogle Scholar
  3. 3.
    Barron A (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39(3):930–945MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bhatia R (2013) Matrix analysis, vol 169. Springer Science & Business MediaGoogle Scholar
  5. 5.
    Bigdely N, Vankov A, Ramirez R, Makeig S (2008) Brain activity-based image classification from rapid serial visual presentation. IEEE Trans Neural Syst Rehabil Eng 16(5):432–441CrossRefGoogle Scholar
  6. 6.
    Carlson TA, Hogendoorn H, Kanai R, Mesik J, Turret J (2011) High temporal resolution decoding of object position and category. J Vis 11(10):9–9CrossRefGoogle Scholar
  7. 7.
    Carlson T, Tovar DA, Alink A, Kriegeskorte N (2013) Representational dynamics of object vision: the first 1000 ms. J Vis 13(10):1–1CrossRefGoogle Scholar
  8. 8.
    Christoudias C, Urtasun R, Darrell T (2012) Multi-view learning in the presence of view disagreement. arXiv:1206.3242
  9. 9.
    Ding C, Zhou D, He X, Zha H (2006) R 1-pca: rotational invariant l 1-norm principal component analysis for robust subspace factorization. In: Proceedings of international conference on machine learning, pp 281–288Google Scholar
  10. 10.
    Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: Proceedings of international conference on machine learning, pp 647–655Google Scholar
  11. 11.
    Gui J, Tao D, Sun Z, Luo Y, You X, Tang Y (2014) Group sparse multiview patch alignment framework with view consistency for image classification. IEEE Trans Image Process 23(7):3126–3137MathSciNetCrossRefGoogle Scholar
  12. 12.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034Google Scholar
  13. 13.
    Hernandez J, Li Y, Rehg JM, Picard RW (2014) Bioglass: physiological parameter estimation using a head-mounted wearable device. In: Proceedings of EAI international conference on wireless mobile communication and healthcare (Mobihealth), pp 55–58Google Scholar
  14. 14.
    Hirshfield L, Chauncey K, Gulotta R, Girouard A, Solovey E, Jacob R, Sassaroli A, Fantini S (2009) Combining electroencephalograph and functional near infrared spectroscopy to explore users mental workload. In: Proceedings of international conference on foundations of augmented cognition, pp 239–247Google Scholar
  15. 15.
    Horii K, Maeda K, Ogawa T, Haseyama M (2018) A human-centered neural network moldel with discriminative locality preserving canonical correlation analysis for image classification. In: Proceedings of IEEE international conference on image processing, pp 2366–2370Google Scholar
  16. 16.
    Hotelling H (1936) Relations between two sets of variates. Biometrika 28 (3/4):321–377CrossRefGoogle Scholar
  17. 17.
    Hsu CW, Chang CC, Lin CJ (2003) A practical guide to support vector classification. Technical Report. Depatment of Computer Science, National Taiwan UniversityGoogle Scholar
  18. 18.
    Huang G, Zhu Q, Siew C (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Proceedings of IEEE international joint conference on neural networks, vol 2, pp 985–990Google Scholar
  19. 19.
    Huang G, Zhu Q, Siew C (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501CrossRefGoogle Scholar
  20. 20.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of ACM international conference on multimedia, pp 675–678Google Scholar
  21. 21.
    Jolliffe I (2002) Principal component analysis. Wiley Online LibraryGoogle Scholar
  22. 22.
    Kaneshiro B, Guimaraes MP, Kim HS, Norcia AM, Suppes P (2015) A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. Plos One 10(8):e0135697CrossRefGoogle Scholar
  23. 23.
    Karessli N, Akata Z, Schiele B, Bulling A (2017) Gaze embeddings for zero-shot image classification. In: Proceedings of the IEEE international conference on computer vision and pattern recognition, p 1611.09309Google Scholar
  24. 24.
    Kim J, Kim J, Jang GJ, Lee M (2017) Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection. Neural Netw 87:109–121CrossRefGoogle Scholar
  25. 25.
    Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, 1097–1105Google Scholar
  26. 26.
    Lee G, Singanamalli A, Wang H, Feldman MD, Master SR, Shih NN, Spangler E, Rebbeck T, Tomaszewski JE, Madabhushi A (2015) Supervised multi-view canonical correlation analysis (sMVCCA): integrating histologic and proteomic features for predicting recurrent prostate cancer. IEEE Trans Med Imaging 34(1):284–297CrossRefGoogle Scholar
  27. 27.
    Li Q, Cai W, Wang X, et al (2014) Medical image classification with convolutional neural network. In: IEEE International conference on control automation robotics & vision, pp 844–848Google Scholar
  28. 28.
    Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi F (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26CrossRefGoogle Scholar
  29. 29.
    Mary NAB, Dharma D (2017) Coral reef image classification employing improved LDP for feature extraction. Journal of Visual Communication and Image RepresentationGoogle Scholar
  30. 30.
    Mary NAB, Dejey D (2018) Classification of coral reef submarine images and videos using a novel Z with tilted Z local binary pattern. Wirel Pers Commun 98(3):2427–2459CrossRefGoogle Scholar
  31. 31.
    Mary NAB, Dharma D (2018) Coral reef image/video classification employing novel octa-angled pattern for triangular sub region and pulse coupled convolutional neural network (PCCNN). Multimed Tools Appl, 1–35Google Scholar
  32. 32.
    Nielsen AA (2002) Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing data. IEEE Trans Image Process 11(3):293–305MathSciNetCrossRefGoogle Scholar
  33. 33.
    Sharif A, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 806–813Google Scholar
  34. 34.
    Shensa M (1992) The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Trans Signal Process 40(10):2464–2482CrossRefGoogle Scholar
  35. 35.
    Slovák P, Janssen J, Fitzpatrick G (2012) Understanding heart rate sharing: towards unpacking physiosocial space. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 859–868Google Scholar
  36. 36.
    Spampinato C, Palazzo S, Kavasidis I, Giordano D, Shah M, Souly N (2016) Deep learning human mind for automated visual classification. arXiv:1609.00344
  37. 37.
    Sugiyama M (2006) Local fisher discriminant analysis for supervised dimensionality reduction. In: Proceedings of international conference on machine learning, pp 905–912Google Scholar
  38. 38.
    Tai K, Chau T (2009) Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface. J Neuroeng Rehabil 6(1):39CrossRefGoogle Scholar
  39. 39.
    Yang J, Zhang X (2012) Feature-level fusion of fingerprint and finger-vein for personal identification. Pattern Recogn Lett 33(5):623–628CrossRefGoogle Scholar
  40. 40.
    Yeh Y, Huang C, Wang Y (2014) Heterogeneous domain adaptation and classification by exploiting the correlation subspace. IEEE Trans Image Process 23(5):2009–2018MathSciNetCrossRefGoogle Scholar
  41. 41.
    Zhang X, Guan N, Luo Z, Lan L (2012) Discriminative locality preserving canonical correlation analysis. In: Proceedings of Chinese conference on pattern recognition, pp 341–349Google Scholar
  42. 42.
    Zhang Z, Xu Y, Yang J, Li X, Zhang D (2015) A survey of sparse representation: algorithms and applications. IEEE Access 3:490–530CrossRefGoogle Scholar
  43. 43.
    Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. Adv Neural Inf Process Syst, 487–495Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Kazaha Horii
    • 1
    Email author
  • Keisuke Maeda
    • 1
  • Takahiro Ogawa
    • 1
  • Miki Haseyama
    • 1
  1. 1.Graduate School of Information Science and TechnologyHokkaido UniversitySapporoJapan

Personalised recommendations