Hybrid Euclidean-and-Riemannian Metric Learning for Image Set Classification

Abstract

We propose a novel hybrid metric learning approach to combine multiple heterogenous statistics for robust image set classification. Specifically, we represent each set with multiple statistics – mean, covariance matrix and Gaussian distribution, which generally complement each other for set modeling. However, it is not trivial to fuse them since the mean vector with $d$-dimension often lies in Euclidean space $\mathbb {R}^d$, whereas the covariance matrix typically resides on Riemannian manifold $Sym^+_{d}$. Besides, according to information geometry, the space of Gaussian distribution can be embedded into another Riemannian manifold $Sym^+_{d+1}$. To fuse these statistics from heterogeneous spaces, we propose a Hybrid Euclidean-and-Riemannian Metric Learning (HERML) method to exploit both Euclidean and Riemannian metrics for embedding their original spaces into high dimensional Hilbert spaces and then jointly learn hybrid metrics with discriminant constraint. The proposed method is evaluated on two tasks: set-based object categorization and video-based face recognition. Extensive experimental results demonstrate that our method has a clear superiority over the state-of-the-art methods.