Towards Space-Time Semantics in Two Frames

  • Karla Brkić
  • Axel Pinz
  • Zoran Kalafatić
  • Siniša Šegvić
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7585)


We present a novel, low-level scheme to analyze spatial and temporal change within a local support region. Assuming available region correspondences between two adjacent frames, we divide each region into a regular grid of patches. Depending on the change of an image function inside the patch over time, each patch is assigned weights for the following four labels: “C” for a constant patch, “O” when new information originates from outside the support region, “I” for “inner” changes, and “N” for information from neighboring patches. Our method goes beyond optical flow, as it provides an additional semantic level of understanding the changes in space-time. We demonstrate how our novel “COIN” scheme can be used to categorize local space-time events in image pairs, including locally planar support regions, 3D discontinuities, and virtual vs. real crossings of 3D structures.


Interest Point Image Patch Synthetic Image Support Region Depth Discontinuity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Doretto, G., Chiuso, A., Wu, Y., Soatto, S.: Dynamic textures. Int. J. of Comp. Vis. 51, 91–109 (2003)zbMATHCrossRefGoogle Scholar
  2. 2.
    Derpanis, K., Wildes, R.: Spacetime texture representation and recognition based on a spatiotemporal orientation analysis. IEEE TPAMI (2012)Google Scholar
  3. 3.
    Doulamis, A., Doulamis, N., Ntalianis, K., Kollias, S.: An efficient fully unsupervised video object segmentation scheme using an adaptive neural-network classifier architecture. Trans. Neur. Netw. 14, 616–630 (2003)CrossRefGoogle Scholar
  4. 4.
    Ogale, A.S., Fermüller, C., Aloimonos, Y.: Motion segmentation using occlusions. IEEE Trans. Pattern Anal. Mach. Intell. 27, 988–992 (2005)CrossRefGoogle Scholar
  5. 5.
    Stein, A.N., Hebert, M.: Local detection of occlusion boundaries in video. Image Vision Comput. 27, 514–522 (2009)CrossRefGoogle Scholar
  6. 6.
    Derpanis, K.G., Wildes, R.P.: Detecting Spatiotemporal Structure Boundaries: Beyond Motion Discontinuities. In: Zha, H., Taniguchi, R.-I., Maybank, S. (eds.) ACCV 2009, Part II. LNCS, vol. 5995, pp. 301–312. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Dollár, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. In: VS-PETS (2005)Google Scholar
  8. 8.
    Ke, Y., Sukthankar, R., Hebert, M.: Efficient visual event detection using volumetric features. In: Proc. ICCV, vol. 1, pp. 166–173 (2005)Google Scholar
  9. 9.
    Laptev, I., Lindeberg, T.: Space-time interest points. In: Proc. ICCV (2003)Google Scholar
  10. 10.
    Laptev, I.: On space-time interest points. Int. J. of Comp. Vis. 64, 107–123 (2005)CrossRefGoogle Scholar
  11. 11.
    Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In: Proc. CVPR, pp. 3361–3368 (2011)Google Scholar
  12. 12.
    Wang, H., Ullah, M.M., Kläser, A., Laptev, I., Schmid, C.: Evaluation of local spatio-temporal features for action recognition. In: Proc. BMVC, p. 127 (2009)Google Scholar
  13. 13.
    Kläser, A., Marszałek, M., Schmid, C.: A spatio-temporal descriptor based on 3d-gradients. In: Proc. BMVC, pp. 995–1004 (2008)Google Scholar
  14. 14.
    Brkić, K., Pinz, A., Šegvić, S., Kalafatić, Z.: Histogram-Based Description of Local Space-Time Appearance. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 206–217. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  15. 15.
    Aanæs, H., Dahl, A., Steenstrup Pedersen, K.: Interesting interest points. Int. J. of Comp. Vis. 97, 18–35 (2012)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Karla Brkić
    • 1
  • Axel Pinz
    • 2
  • Zoran Kalafatić
    • 1
  • Siniša Šegvić
    • 1
  1. 1.Faculty of Electrical Engineering and ComputingUniversity of ZagrebCroatia
  2. 2.Graz University of TechnologyAustria

Personalised recommendations