Learning CCA Representations for Misaligned Data

  • Hichem SahbiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11132)


Canonical correlation analysis (CCA) is a statistical learning method that seeks to build view-independent latent representations from multi-view data. This method has been successfully applied to several pattern analysis tasks such as image-to-text mapping and view-invariant object/action recognition. However, this success is highly dependent on the quality of data pairing (i.e., alignments) and mispairing adversely affects the generalization ability of the learned CCA representations.

In this paper, we address the issue of alignment errors using a new variant of canonical correlation analysis referred to as alignment-agnostic (AA) CCA. Starting from erroneously paired data taken from different views, this CCA finds transformation matrices by optimizing a constrained maximization problem that mixes a data correlation term with context regularization; the particular design of these two terms mitigates the effect of alignment errors when learning the CCA transformations. Experiments conducted on multi-view tasks, including multi-temporal satellite image change detection, show that our AA CCA method is highly effective and resilient to mispairing errors.


Canonical correlation analysis Learning compact representations Misalignment resilience Change detection 


  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  2. 2.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Boujemaa, N., Fleuret, F., Gouet, V., Sahbi, H.: Visual content extraction for automatic semantic annotation of video news. In: The proceedings of the SPIE Conference, San Jose, CA, vol. 6 (2004)Google Scholar
  4. 4.
    Wang, L., Sahbi, H.: Directed acyclic graph kernels for action recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3168–3175 (2013)Google Scholar
  5. 5.
    Tollari, S., et al.: A comparative study of diversity methods for hybrid text and image retrieval approaches. In: Peters, C., et al. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 585–592. Springer, Heidelberg (2009). Scholar
  6. 6.
    Hussain, M., Chen, D., Cheng, A., Wei, H., Stanley, D.: Change detection from remotely sensed images: from pixel-based to object-based approaches. ISPRS J. Photogrammetry Remote Sens. 80, 91–106 (2013)CrossRefGoogle Scholar
  7. 7.
    Bourdis, N., Marraud, D., Sahbi, H.: Spatio-temporal interaction for aerial video change detection. In: 2012 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 2253–2256. IEEE (2012)Google Scholar
  8. 8.
    Sahbi, H., Boujemaa, N.: Coarse-to-fine support vector classifiers for face detection. In: null, p. 30359. IEEE (2002)Google Scholar
  9. 9.
    Li, X., Sahbi, H.: Superpixel-based object class segmentation using conditional random fields. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1101–1104. IEEE (2011)Google Scholar
  10. 10.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  11. 11.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. Book in preparation for MIT Press (2016).
  12. 12.
    Jiu, M., Sahbi, H.: Nonlinear deep kernel learning for image annotation. IEEE Trans. Image Process. 26(4), 1820–1832 (2017)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11, 625–660 (2010)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems, pp. 3320–3328 (2014)Google Scholar
  15. 15.
    Doersch, C., Zisserman, A.: Multi-task self-supervised visual learning. arXiv preprint arXiv:1708.07860 (2017)
  16. 16.
    Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part II. LNCS, vol. 9906, pp. 102–118. Springer, Cham (2016). Scholar
  17. 17.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)CrossRefGoogle Scholar
  18. 18.
    Anderson, T.W.: An Introduction to Multivariate Statistical Analysis, vol. 2. Wiley, New York (1958)zbMATHGoogle Scholar
  19. 19.
    Hardoon, D.R., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16(12), 2639–2664 (2004)CrossRefGoogle Scholar
  20. 20.
    Sahbi, H.: Interactive satellite image change detection with context-aware canonical correlation analysis. IEEE Geosci. Remote Sens. Lett. 14(5), 607–611 (2017)CrossRefGoogle Scholar
  21. 21.
    Melzer, T., Reiter, M., Bischof, H.: Appearance models based on kernel canonical correlation analysis. Patt. Recogn. 36(9), 1961–1971 (2003)CrossRefGoogle Scholar
  22. 22.
    Hardoon, D.R., Shawe-Taylor, J.: Sparse canonical correlation analysis. Mach. Learn. 83(3), 331–353 (2011)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Witten, D.M., Tibshirani, R.J.: Extensions of sparse canonical correlation analysis with applications to genomic data. Stat. Appl. Genet. Mol. Biol. 8(1), 1–27 (2009)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Zhang, Z., Zhao, M., Chow, T.W.: Binary-and multi-class group sparse canonical correlation analysis for feature extraction and classification. IEEE Trans. Knowl. Data Eng. 25(10), 2192–2205 (2013)CrossRefGoogle Scholar
  25. 25.
    Vía, J., Santamaría, I., Pérez, J.: A learning algorithm for adaptive canonical correlation analysis of several data sets. Neural Netw. 20(1), 139–152 (2007)CrossRefGoogle Scholar
  26. 26.
    Sun, T., Chen, S.: Locality preserving cca with applications to data visualization and pose estimation. Image Vis. Comput. 25(5), 531–543 (2007)CrossRefGoogle Scholar
  27. 27.
    Zhai, D., Zhang, Y., Yeung, D.Y., Chang, H., Chen, X., Gao, W.: Instance-specific canonical correlation analysis. Neurocomputing 155, 205–218 (2015)CrossRefGoogle Scholar
  28. 28.
    Yger, F., Berar, M., Gasso, G., Rakotomamonjy, A.: Adaptive canonical correlation analysis based on matrix manifolds. arXiv preprint arXiv:1206.6453 (2012)
  29. 29.
    De la Torre, F.: A unification of component analysis methods. In: Handbook of Pattern Recognition and Computer Vision, pp. 3–22 (2009)Google Scholar
  30. 30.
    Sun, J., Keates, S.: Canonical correlation analysis on data with censoring and error information. IEEE Trans. Neural Netw. Learn. Syst. 24(12), 1909–1919 (2013)CrossRefGoogle Scholar
  31. 31.
    Sun, L., Ji, S., Ye, J.: Canonical correlation analysis for multilabel classification: a least-squares formulation, extensions, and analysis. IEEE Trans. Patt. Anal. Mach. Intell. 33(1), 194–200 (2011)CrossRefGoogle Scholar
  32. 32.
    Haghighat, M., Abdel-Mottaleb, M.: Low resolution face recognition in surveillance systems using discriminant correlation analysis. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), pp. 912–917. IEEE (2017)Google Scholar
  33. 33.
    Ferecatu, M., Sahbi, H.: Multi-view object matching and tracking using canonical correlation analysis. In: 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 2109–2112. IEEE (2009)Google Scholar
  34. 34.
    Loy, C.C., Xiang, T., Gong, S.: Multi-camera activity correlation analysis. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 1988–1995. IEEE (2009)Google Scholar
  35. 35.
    Zhou, F., Torre, F.: Canonical time warping for alignment of human behavior. In: Advances in Neural Information Processing Systems, pp. 2286–2294 (2009)Google Scholar
  36. 36.
    Zhou, F., De la Torre, F.: Generalized canonical time warping. IEEE Trans. Patt. Anal. Mach. Intell. 38(2), 279–294 (2016)CrossRefGoogle Scholar
  37. 37.
    Kim, T.K., Cipolla, R.: Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Trans. Patt. Anal. Mach. Intell. 31(8), 1415–1428 (2009)CrossRefGoogle Scholar
  38. 38.
    Fischer, B., Roth, V., Buhmann, J.M.: Time-series alignment by non-negative multiple generalized canonical correlation analysis. BMC Bioinform. 8(10), S4 (2007)CrossRefGoogle Scholar
  39. 39.
    Trigeorgis, G., Nicolaou, M., Zafeiriou, S., Schuller, B.: Deep canonical time warping for simultaneous alignment and representation learning of sequences. IEEE Trans. Patt. Anal. Mach. Intell. 40(5), 1128–1138 (2017)CrossRefGoogle Scholar
  40. 40.
    Ham, J., Lee, D.D., Saul, L.K.: Semisupervised alignment of manifolds. In: AISTATS, pp. 120–127 (2005)Google Scholar
  41. 41.
    Lafon, S., Keller, Y., Coifman, R.R.: Data fusion and multicue data matching by diffusion maps. IEEE Trans. Patt. Anal. Mach. Intell. 28(11), 1784–1797 (2006)CrossRefGoogle Scholar
  42. 42.
    Wang, C., Mahadevan, S.: Manifold alignment using procrustes analysis. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1120–1127. ACM (2008)Google Scholar
  43. 43.
    Luo, B., Hancock, E.R.: Iterative procrustes alignment with the EM algorithm. Image Vis. Comput. 20(5), 377–396 (2002)CrossRefGoogle Scholar
  44. 44.
    Feuz, K.D., Cook, D.J.: Collegial activity learning between heterogeneous sensors. Knowl. Inf. Syst. 1–28 (2017)Google Scholar
  45. 45.
    Sahbi, H., Li, X.: Context-based support vector machines for interconnected image annotation. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part I. LNCS, vol. 6492, pp. 214–227. Springer, Heidelberg (2011). Scholar
  46. 46.
    Sahbi, H.: CNRS-TELECOM ParisTech at imageCLEF 2013 scalable concept image annotation task: winning annotations with context dependent SVMs. In: CLEF (Working Notes) (2013)Google Scholar
  47. 47.
    Sahbi, H.: Discriminant canonical correlation analysis for interactive satellite image change detection. In: IGARSS, pp. 2789–2792 (2015)Google Scholar
  48. 48.
    Bourdis, N., Denis, M., Sahbi, H.: Constrained optical flow for aerial image change detection. In: 2011 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 4176–4179 (2011)Google Scholar
  49. 49.
    Bourdis, N., Denis, M., Sahbi, H.: Camera pose estimation using visual servoing for aerial video change detection. In: 2012 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3459–3462 (2012)Google Scholar
  50. 50.
    Sahbi, H.: Relevance feedback for satellite image change detection. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1503–1507. IEEE (2013)Google Scholar
  51. 51.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  52. 52.
    Thiemert, S., Sahbi, H., Steinebach, M.: Applying interest operators in semi-fragile video watermarking. In: Security, Steganography, and Watermarking of Multimedia Contents VII, vol. 5681, International Society for Optics and Photonics, pp. 353–363 (2005)Google Scholar
  53. 53.
    Kim, T., Im, Y.J.: Automatic satellite image registration by combination of matching and random sample consensus. IEEE Trans. Geosci. Remote Sens. 41(5), 1111–1117 (2003)CrossRefGoogle Scholar
  54. 54.
    Lu, L.Z., Pearce, C.E.M.: Some new bounds for singular values and eigenvalues of matrix products. Ann. Oper. Res. 98(1), 141–148 (2000)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.CNRS, Sorbonne UniversityParisFrance

Personalised recommendations