Nonlinear Subspace Feature Enhancement for Image Set Classification

  • Mohammed E. FathyEmail author
  • Azadeh Alavi
  • Rama Chellappa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)


While several methods have been proposed for modeling and recognizing image sets, the success of these methods relies heavily on how well the image data follows the assumptions of the underlying models. Among the models that have been utilized by many image set classification methods, the physically inspired subspace model assumes that the images of an object lie on a union of low-dimensional subspaces. Despite their successful performance in controlled environments, the performance of such subspace-based classifiers suffers in practical unconstrained settings, where the data may not strictly follow the assumptions necessary for the subspace model to hold. In this paper, we propose Nonlinear Subspace Feature Enhancement (NSFE), an approach for nonlinearly embedding image sets into a space where they adhere to a more discriminative subspace structure. In turn, this improves the performance of subspace-based classifiers such as sparse representation-based classification. We describe how the structured loss function of NSFE can be optimized in a batch-by-batch fashion by a two-step alternating algorithm. The algorithm makes very few assumptions about the form of the embedding to be learned and is compatible with stochastic gradient descent and back-propagation. This makes NSFE usable with deep, feed-forward embeddings and trainable in an end-to-end fashion. We experiment with two different types of features and nonlinear embeddings over three image set datasets and we show that our method compares favorably to state-of-the-art image set classification methods.



This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.


  1. 1.
    Wang, R., Shan, S., Chen, X., Gao, W.: Manifold-manifold distance with application to face recognition based on image set. In: CVPR, pp. 1–8 (2008)Google Scholar
  2. 2.
    Wang, R., Chen, X.: Manifold discriminant analysis. In: CVPR, pp. 429–436 (2009)Google Scholar
  3. 3.
    Cevikalp, H., Triggs, B.: Face recognition based on image sets. In: CVPR, pp. 2567–2573 (2010)Google Scholar
  4. 4.
    Hu, Y., Mian, A.S., Owens, R.: Sparse approximated nearest points for image set classification. In: CVPR, pp. 121–128 (2011)Google Scholar
  5. 5.
    Harandi, M.T., Sanderson, C., Shirazi, S., Lovell, B.C.: Graph embedding discriminant analysis on Grassmannian manifolds for improved image set matching. In: CVPR, pp. 2705–2712 (2011)Google Scholar
  6. 6.
    Mahmood, A., Mian, A.: Hierarchical sparse spectral clustering for image set classification. In: BMVC, pp. 1–11 (2012)Google Scholar
  7. 7.
    Chen, Y.-C., Patel, V.M., Phillips, P.J., Chellappa, R.: Dictionary-based face recognition from video. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 766–779. Springer, Heidelberg (2012). Scholar
  8. 8.
    Wang, R., Guo, H., Davis, L.S., Dai, Q.: Covariance discriminative learning: a natural and efficient approach to image set classification. In: CVPR, pp. 2496–2503 (2012)Google Scholar
  9. 9.
    Harandi, M., Sanderson, C., Shen, C., Lovell, B.C.: Dictionary learning and sparse coding on Grassmann manifolds: an extrinsic solution. In: ICCV, pp. 3120–3127 (2013)Google Scholar
  10. 10.
    Ortiz, E.G., Wright, A., Shah, M.: Face recognition in movie trailers via mean sequence sparse representation-based classification. In: CVPR, pp. 3531–3538 (2013)Google Scholar
  11. 11.
    Chen, S., Sanderson, C., Harandi, M.T., Lovell, B.C.: Improved image set classification via joint sparse approximated nearest subspaces. In: CVPR, pp. 452–459 (2013)Google Scholar
  12. 12.
    Chen, L.: Dual linear regression based classification for face cluster recognition. In: CVPR, pp. 2673–2680 (2014)Google Scholar
  13. 13.
    Mahmood, A., Mian, A., Owens, R.: Semi-supervised spectral clustering for image set classification. In: CVPR, pp. 121–128 (2014)Google Scholar
  14. 14.
    Hayat, M., Bennamoun, M., An, S.: Learning non-linear reconstruction models for image set classification. In: CVPR, pp. 1915–1922 (2014)Google Scholar
  15. 15.
    Hayat, M., Bennamoun, M., An, S.: Reverse training: an efficient approach for image set classification. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 784–799. Springer, Cham (2014). Scholar
  16. 16.
    Lu, J., Wang, G., Deng, W., Moulin, P.: Simultaneous feature and dictionary learning for image set based face recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 265–280. Springer, Cham (2014). Scholar
  17. 17.
    Lu, J., Wang, G., Deng, W., Moulin, P., Zhou, J.: Multi-manifold deep metric learning for image set classification. In: CVPR (2015) 1137–1145Google Scholar
  18. 18.
    Huang, Z., Wang, R., Shan, S., Li, X., Chen, X.: Log-Euclidean metric learning on symmetric positive definite manifold with application to image set classification. In: ICML, pp. 720–729 (2015)Google Scholar
  19. 19.
    Wang, W., Wang, R., Huang, Z., Shan, S., Chen, X.: Discriminant analysis on Riemannian manifold of Gaussian distributions for face recognition with image sets. In: CVPR, pp. 2048–2057 (2015)Google Scholar
  20. 20.
    Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. PAMI 25, 218–233 (2003)CrossRefGoogle Scholar
  21. 21.
    Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. PAMI 31, 210–227 (2009)CrossRefGoogle Scholar
  22. 22.
    Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. JMLR 12, 2121–2159 (2011)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Sutskever, I., Martens, J., Dahl, G.E., Hinton, G.E.: On the importance of initialization and momentum in deep learning. In: ICML, pp. 1139–1147 (2013)Google Scholar
  24. 24.
    Hamm, J., Lee, D.D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: ICML, pp. 376–383 (2008)Google Scholar
  25. 25.
    Huang, Z., Wang, R., Shan, S., Chen, X.: Projection metric learning on Grassmann manifold with application to video based face recognition. In: CVPR, pp. 140–149 (2015)Google Scholar
  26. 26.
    Zhu, P., Zhang, L., Zuo, W., Zhang, D.: From point to set: extend the learning of distance metrics. In: ICCV, pp. 2664–2671 (2013)Google Scholar
  27. 27.
    Harandi, M., Salzmann, M., Baktashmotlagh, M.: Beyond Gauss: image-set matching on the Riemannian manifold of PDFs. In: ICCV, pp. 4112–4120 (2015)Google Scholar
  28. 28.
    Hayat, M., Bennamoun, M., An, S.: Deep reconstruction models for image set classification. PAMI 37, 713–727 (2015)CrossRefGoogle Scholar
  29. 29.
    Zhang, L., Yang, M., Feng, X.: Sparse representation or collaborative representation: which helps face recognition? In: ICCV, pp. 471–478 (2011)Google Scholar
  30. 30.
    Zhu, P., Zuo, W., Zhang, L., Shiu, S.C.K., Zhang, D.: Image set-based collaborative representation for face recognition. IEEE Trans. Inf. Forens. Secur. 9, 1120–1132 (2014)CrossRefGoogle Scholar
  31. 31.
    Zhang, H., Zhang, Y., Huang, T.S.: Simultaneous discriminative projection and dictionary learning for sparse representation based classification. Pattern Recogn. 46, 346–354 (2013)CrossRefGoogle Scholar
  32. 32.
    Qiu, Q., Sapiro, G.: Learning transformations for clustering and classification. JMLR 16, 187–225 (2015)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Fathy, M.E., Alavi, A., Chellappa, R.: Discriminative Log-Euclidean feature learning for sparse representation-based recognition of faces from videos, pp. 3359–3367 (2016)Google Scholar
  34. 34.
    Weinberger, K.Q., Blitzer, J., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. In: NIPS, pp. 1473–1480 (2005)Google Scholar
  35. 35.
    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR, pp. 815–823 (2015)Google Scholar
  36. 36.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  37. 37.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)Google Scholar
  38. 38.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: ICML, pp. 689–696 (2009)Google Scholar
  39. 39.
    Kim, M., Kumar, S., Pavlovic, V., Rowley, H.: Face tracking and recognition with visual constraints in real-world videos. In: CVPR, pp. 1–8 (2008)Google Scholar
  40. 40.
    Viola, P., Jones, M.J.: Robust real-time face detection. IJCV 57, 137–154 (2004)CrossRefGoogle Scholar
  41. 41.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: CVPR, pp. 3444–3451 (2013)Google Scholar
  42. 42.
    Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: CVPR, pp. 529–534 (2011)Google Scholar
  43. 43.
    Fathy, M.E., Patel, V.M., Chellappa, R.: Face-based active authentication on mobile devices. In: ICASSP, pp. 1687–1691 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mohammed E. Fathy
    • 1
    Email author
  • Azadeh Alavi
    • 1
  • Rama Chellappa
    • 1
  1. 1.Center for Automation ResearchUniversity of MarylandCollege ParkUSA

Personalised recommendations