Learning Image Representations Tied to Egomotion from Unlabeled Video

Abstract

Understanding how images of objects and scenes behave in response to specific egomotions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose a new “embodied” visual learning paradigm, exploiting proprioceptive motor signals to train visual representations from egocentric video with no manual supervision. Specifically, we enforce that our learned features exhibit equivariance i.e., they respond predictably to transformations associated with distinct egomotions. With three datasets, we show that our unsupervised feature learning approach significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks. In the most challenging test, we show that features learned from video captured on an autonomous driving platform improve large-scale scene recognition in static images from a disjoint domain.

Appendix

1.1 Optimization and Hyperparameter Selection

(Elaborating on para titled “Network architectures and Optimization” in Sect. 4.1)

We use Nesterov-accelerated stochastic gradient descent as implemented in Caffe (Jia et al. 2014), starting from weights randomly initialized according to Glorot and Bengio (2010). The base learning rate and regularization $\lambda$s are selected with greedy cross-validation. Specifically, for each task, the optimal base learning rate (from 0.1, 0.01, 0.001, 0.0001) was identified for clsnet. Next, with this base learning rate fixed, the optimal regularizer weight (for drlim, temporal and equiv) was selected from a logarithmic grid (steps of $10^{0.5}$). For equiv + drlim, the drlim loss regularizer weight fixed for drlim was retained, and only the equiv loss weight was cross-validated. The contrastive loss margin parameter $\delta$ in Eq. (8) in drlim, temporal and equiv were set uniformly to 1.0. Since no other part of these objectives (including the softmax classification loss) depends on the scale of features,¹⁵ different choices of margins $\delta$ in these methods lead to objective functions with equivalent optima—the features are only scaled by a factor. For equiv + drlim, we set the drlim and equiv margins respectively to 1.0 and 0.1 to reflect the fact that the equivariance maps $M_g$ of Eq. (7) applied to the representation $\mathbf {z_{\varvec{\theta }}}(g\varvec{x})$ of the transformed image must bring it closer to the original image representation $\mathbf {z_{\varvec{\theta }}}(\varvec{x})$ than it was before i.e., $\Vert M_g \mathbf {z_{\varvec{\theta }}}(g\varvec{x})- \mathbf {z_{\varvec{\theta }}}(\varvec{x})\Vert _2 < \Vert \mathbf {z_{\varvec{\theta }}}(g\varvec{x}) - \mathbf {z_{\varvec{\theta }}}(\varvec{x})\Vert _2$.

Fig. 12

Slowness AUROC on training (left) and testing (right) data for (top) drlim (bottom) coherence, showing the weakness of slowness prior

In addition, to allow fast and thorough experimentation, we set the number of training epochs for each method on each dataset based on a number of initial runs to assess the scale of time usually taken before the classification softmax loss on validation data began to rise i.e., overfitting began. All future runs for that method on that data were run to roughly match (to the nearest 5000) the number of epochs identified above. For most cases, this number was of the order of 50,000. Batch sizes (for both the classification stack and the Siamese networks) were set to 16 (found to have no major difference from 4 or 64) for NORB-NORB and KITTI-KITTI, and to 128 (selected from 4, 16, 64, 128) for KITTI-SUN, where we found it necessary to increase batch size so that meaningful classification loss gradients were computed in each SGD iteration, and training loss began to fall, despite the large number (397) of classes.

On a single Tesla K-40 GPU machine, NORB-NORB training tasks took $\approx$15 min, KITTI-KITTI tasks took $\approx$30 min, and KITTI-SUN tasks took $\approx$2 h. The purely unsupervised training runs with large “KITTI-227” images took up to  15 h per run.

1.2 The Weakness of the Slow Feature Analysis Prior

We now present evidence supporting our claim in the paper that the principle of slowness, which penalizes feature variation within small temporal windows, provides a prior that is rather weak. In every stochastic gradient descent (SGD) training iteration for the drlim and temporal networks, we also computed a “slowness” measure that is independent of feature scaling (unlike the drlim and temporal losses of Eq. (9) themselves), to better understand the shortcomings of these methods.

Given training pairs $(\varvec{x}_i,\varvec{x}_j)$ annotated as neighbors or non-neighbors by $n_{ij}=\mathbbm {1}(|t_i-t_j|\le T)$ (cf. Eq. (9) in the paper), we computed pairwise distances $\varDelta _{ij}=d(\mathbf {z}_{\varvec{\theta }(s)}(\varvec{x}_i), \mathbf {z}_{\varvec{\theta }(s)}(\varvec{x}_j))$, where $\varvec{\theta }(s)$ is the parameter vector at SGD training iteration s, and d(., .) is set to the $\ell _2$ distance for drlim and to the $\ell _1$ distance for temporal (cf. Sect. 4).

We then measured how well these pairwise distances $\varDelta _{ij}$ predict the temporal neighborhood annotation $n_{ij}$, by measuring the Area Under Receiver Operating Characteristic (AUROC) when varying a threshold on $\varDelta _{ij}$.

These “slowness AUROC”s are plotted as a function of training iteration number in Fig. 12, for drlim and coherence networks trained on the KITTI-SUN task. Compared to the standard random AUROC value of 0.5, these slowness AUROCs tend to be near 0.9 already even before optimization begins, and reach peak AUROCs very close to 1.0 on both training and testing data within about 4000 iterations (batch size 128). This points to a possible weakness in these methods—even with parameters (temporal neighborhood size, regularization $\lambda$) cross-validated for recognition, the slowness prior is too weak to regularize feature learning effectively, since strengthening it causes loss of discriminative information. In contrast, our method requires systematic feature space responses to egomotions, and offers a stronger prior.