Slow Feature Analysis
Slow feature analysis (SFA) is an unsupervised learning algorithm for extracting slowly varying features from a multidimensional input signal in time. It is not based on low-pass filtering, i.e., temporal averaging, but combines input components of single time steps into temporally stable features. SFA can be used for nonlinear dimensionality reduction and learning of invariant representations. Algorithmically, it is closely related to principal component analysis (PCA).
The slowness principle is based on the observation that different representations of a sensorial input vary on different time scales. For instance, a zebra grazing in the savanna is a scene that changes slowly. This scene is represented in the eyes of an observer in terms of activities of retinal receptors, which, due to the black-and-white stripes of the zebra, change quickly between high and low values whenever the zebra moves or the gaze of the observer changes. However, in higher brain areas of the observer, there is a high-level representation of the zebra grazing, which changes slowly again. This difference in time scales is true for most dynamic scenes. The slowness principle therefore states that high-level representations can be learned from the receptor activities simply by extracting features that vary slowly over time (without using the trivial option of low-pass filtering).
The idea of the slowness principle has probably first been mentioned by Geoffrey Hinton in 1989; early algorithms were presented by Peter Földiák and by Graeme Mitchison, both in 1991; slow feature analysis (SFA) has first been introduced by Laurenz Wiskott in 1998; see (Wiskott et al. 2011) for references.
Slow Feature Analysis
Most learning algorithms based on the slowness principle are online learning rules, i.e., they improve the extracted features incrementally with each time step of the training data. Slow feature analysis (SFA) (Wiskott et al. 2011) in contrast is an algorithm that takes all training data into account at once.
SFA first uses PCA to normalize the input signal such that it has zero mean and unit variance in all directions, which is also known as whitening or sphering. It then calculates the time derivative (in the discrete case the difference between successive time points) of the whitened signal and uses PCA again to find the directions of smallest variance, which are the directions of slowest variation and correspond to the slow features to be extracted.
As SFA is based on PCA, it is basically a linear algorithm. However, one typically applies a predefined set of nonlinear functions to the input signal first, which is known as nonlinear expansion, and then applies SFA to this expanded input signal. Thus, SFA learns linear combinations of fixed nonlinear functions, which results in nonlinear functions to extract features from the input.
It has been shown analytically that SFA is closely related to spike-timing-dependent plasticity (STDP) (Sprekeler et al. 2007).
Applications in Computational Neuroscience
SFA has been developed for learning invariant representations of moving objects in a feedforward model of the visual system. It has been shown that a hierarchical network of SFA nodes extracts features from a video signal of moving objects that permit invariant object recognition with a simple classifier and also extraction of positional and pose information with linear regression (Wiskott et al. 2011). Interestingly, on the first SFA layer units share many properties with complex cells in the primary visual cortex, such as phase invariance, orientation selectivity, and frequency tuning. One also finds end- and side-inhibited units there. Complex cell-like units emerge also if learning is based on simulated retinal waves rather than moving objects (Dähne et al. 2014).
If combined with independent component analysis, an SFA network trained with input from a simulated rat running around in a static environment develops units that behave like place cells and others that behave like head-direction cells in the hippocampal formation of rats (Wiskott et al. 2011). Place cells fire if a rat is at a particular location independently of its head direction; head-direction cells fire if a rat is oriented in a particular direction independently of its location. SFA can extract location and head direction because these variables vary slowly compared to individual pixel values of the visual input sequence as the rat runs through the environment.
SFA hast also been used for technical applications (Escalante-B and Wiskott 2012).
- Dähne S, Wilbert N, Wiskott L (2014) Slow feature analysis on retinal waves leads to VI complex cells. PLoS Comput Biol, in pressGoogle Scholar