Aligning Sequences and Actions by Maximizing Space-Time Correlations

  • Yaron Ukrainitz
  • Michal Irani
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3953)


We introduced an algorithm for sequence alignment, based on maximizing local space-time correlations. Our algorithm aligns sequences of the same action performed at different times and places by different people, possibly at different speeds, and wearing different clothes. Moreover, the algorithm offers a unified approach to the problem of sequence alignment for a wide range of scenarios (e.g., sequence pairs taken with stationary or jointly moving cameras, with the same or different photometric properties, with or without moving objects). Our algorithm is applied directly to the dense space-time intensity information of the two sequences (or to filtered versions of them). This is done without prior segmentation of foreground moving objects, and without prior detection of corresponding features across the sequences. Examples of challenging sequences with complex actions are shown, including ballet dancing, actions in the presence of other complex scene dynamics (clutter), as well as multi-sensor sequence pairs.


Video Sequence Input Sequence Alignment Algorithm Temporal Derivative Pyramid Level 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Burt, P., Hingorani, R., Kolczynski, R.: Mechanisms for isolating component patterns in the sequential analysis of multiple motion. In: Workshop on Visual Motion (1991)Google Scholar
  2. 2.
    Carlsson, S.: Recognizing walking people. IJRR 22, 359–370 (2003)Google Scholar
  3. 3.
    Caspi, Y., Irani, M.: Aligning non-overlapping sequences. IJCV 48 (2002)Google Scholar
  4. 4.
    Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. T-PAMI (2002)Google Scholar
  5. 5.
    Caspi, Y., Simakov, D., Irani, M.: Feature-based sequence-to-sequence matching. In: VMODS (2002)Google Scholar
  6. 6.
    Giese, M.A., Poggio, T.: Synthesis and recognition of biological motion patterns based on linear superposition of prototypical motion sequences. In: IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (1999)Google Scholar
  7. 7.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Addison-Wesley, Reading (1993)Google Scholar
  8. 8.
    Irani, M., Anandan, P.: Robust multi-sensor image alignment. In: ICCV (1998)Google Scholar
  9. 9.
    Press, W., Flannery, B., Teukolsky, S., Vetterling, W.: Numerical Recipes in C. Cambridge University Press, Cambridge (1988)zbMATHGoogle Scholar
  10. 10.
    Rao, C., Gritai, A., Shah, M., Syeda-Mahmood, T.: View-invariant alignment and matching of video sequences. In: ICCV, pp. 939–945 (2004)Google Scholar
  11. 11.
    Shechtman, E., Irani, M.: Space-time behavior based correlation. In: CVPR (2005)Google Scholar
  12. 12.
    Thomas, G., Finney, R.: Calculus and Analytic Geometry, 9th edn. Addison-Wesley, Reading (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Yaron Ukrainitz
    • 1
  • Michal Irani
    • 1
  1. 1.Department of Computer Science and Applied MathematicsThe Weizmann Institute of ScienceRehovotIsrael

Personalised recommendations