Attending to Motion: Localizing and Classifying Motion Patterns in Image Sequences

  • John K. Tsotsos
  • Marc Pomplun
  • Yueju Liu
  • Julio C. Martinez-Trujillo
  • Evgueni Simine
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2525)


The Selective Tuning Model is a proposal for modelling visual attention in primates and humans. Although supported by significant biological evidence, it is not without its weaknesses. The main one addressed by this paper is that the levels of representation on which it was previously demonstrated (spatial Gaussian pyramids) were not biologically plausible. The motion domain was chosen because enough is known about motion processing to enable a reasonable attempt at defining the feedforward pyramid. The effort is unique because it seems that no past model presents a motion hierarchy plus attention to motion. We propose a neurally-inspired model of the primate visual motion system attempting to explain how a hierarchical feedforward network consisting of layers representing cortical areas V1, MT, MST, and 7a detects and classifies different kinds of motion patterns. The STM model is then integrated into this hierarchy demonstrating that successfully attending to motion patterns, results in localization and labelling of those patterns.


Visual Attention Optic Flow Attentional Modulation Direction Selectivity Middle Temporal 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, J. K., Cai, Q., Liao, W., Sabata, B. (1998). Nonrigid motion analysis: Articulated and elastic motion, Computer Vision and Image Understanding 70(2), p142–156.CrossRefGoogle Scholar
  2. 2.
    Shah, M., Jain, R. (1997). Visual recognition of activities, gestures, facial expressions and speech: an introduction and a perspective, in Motion-Based Recognition, ed. by M. Shah and R. Jain, Kluwer Academic Publishers.Google Scholar
  3. 3.
    Cedras, C., Shah, M. (1994). A survey of motion analysis from moving light displays, IEEE CVPR-94, Seattle, Washington, p214–221.Google Scholar
  4. 4.
    Cedras, C., Shah, M. (1995). Motion-based recognition: A survey, Image and Vision Computing, 13(2), p129–155.CrossRefGoogle Scholar
  5. 5.
    Hildreth, E. Royden, C. (1995). Motion Perception, in The Handbook of Brain Theory and Neural Networks, ed. by M. Arbib, MIT Press, p585–588.Google Scholar
  6. 6.
    Aggarwal, J. K., Cai, Q. (1999). Human motion analysis: A Review, Computer Vision and Image Understanding 73(3), p428–440.CrossRefGoogle Scholar
  7. 7.
    Gavrila, D. M. (1999). The visual analysis of human movement: A Survey, Computer Vision and Image Understanding 73(1), p82–98.CrossRefzbMATHGoogle Scholar
  8. 8.
    Tsotsos, J. K., (2001). Motion Understanding: Task-Directed Attention and Representations that link Perception with Action, Int. J. of Computer Vision 45:3, 265–280.CrossRefzbMATHGoogle Scholar
  9. 9.
    Siskind, J. M. (1995). Grounding Language in Perception. Artificial Intelligence Review 8, p371–391.CrossRefGoogle Scholar
  10. 10.
    Mann, R., Jepson, A., Siskind, J. (1997). The computational perception of scene dynamics, Computer Vision and Image Understanding, 65(2), p113–128.CrossRefGoogle Scholar
  11. 11.
    Pinhanez, C., Bobick, A. (1997). Human action detection using PNF propagation of temporal constraints, MIT Media Lab TR 423, April.Google Scholar
  12. 12.
    Tsotsos, J. K. (1980). A framework for visual motion understanding, Ph.D. Thesis, Dept. of Computer Science, University of Toronto, May.Google Scholar
  13. 13.
    Dickmanns, E. D., Wünsche, H. J. (1999). Dynamic vision for perception and control of motion, Handbook of Computer Vision and Applications Vol. 2, ed by B. Jahne, H. Haubeccker, P. Geibler, Academic Press.Google Scholar
  14. 14.
    Dreschler, L., Nagel, H. H. (1982). On the selection of critical points and local curvature extrema of region boundaries for interframe matching, Proc. Int. Conf. Pattern Recognition, Munich, p542–544.Google Scholar
  15. 15.
    Wachter, S., Nagel, H. H. (1999). Tracking persons in monocular image sequences, Computer Vision and Image Understanding 74(3), p174–192.CrossRefGoogle Scholar
  16. 16.
    Tsotsos, J. K. (1990). Analyzing vision at the complexity level, Behavioral and Brain Sciences 13–3, p423–445.CrossRefGoogle Scholar
  17. 17.
    Desimone, R., Duncan, J., (1995). Neural Mechanisms of Selective Attention, Annual Review of Neuroscience 18, p193–222.CrossRefGoogle Scholar
  18. 18.
    Treue, S., Martinez-Trujillo, J. C., (1999). Feature-based attention influences motion processing gain in macaque visual cortex, nayure, 399, 575–579.Google Scholar
  19. 19.
    Tsotsos, J. K., Culhane, S. M., Wai, W. Y. K., Lai, Y., Davis, N. & Nuflo, F. (1995). Modeling visual attention via selective tuning. Artificial Intelligence, 78, 507–545.CrossRefGoogle Scholar
  20. 20.
    Koch, C., Ullman, S., (1985). Shifts in selective visual attention: Towards the underlying neural circuitry, Hum. Neurobiology 4, p219–227.Google Scholar
  21. 21.
    Moran, J., Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex, Science 229, p782–784.CrossRefGoogle Scholar
  22. 22.
    Kastner, S., De Weerd, P., Desimone, R., Ungerleider, L. (1998). Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRI, Science 282, p108–111.CrossRefGoogle Scholar
  23. 23.
    Vanduffel, W., Tootell, R., Orban, G. (2000). Attention-dependent suppression ofmetabolic activity in the early stages of the macaque visual system, Cerebral Cortex 10, p109–126.CrossRefGoogle Scholar
  24. 24.
    Brefczynski J. A., DeYoe E. A. (1999). A physiological correlate of the’ spotlight’ of visual attention. Nat Neurosci. Apr;2(4), p370–374.CrossRefGoogle Scholar
  25. 25.
    Gandhi S. P., Heeger D. J., Boynton G. M. (1999). Spatial attention affects brain activity in human primary visual cortex, Proc Natl Acad Sci U S A, Mar 16;96(6), p3314–9.CrossRefGoogle Scholar
  26. 26.
    Smith, A., Singh, K., Greenlee, M. (2000). Attentional suppression of activity in the human visual cortex, NeuroReport, Vol 11 No 2 7, p271–277.CrossRefGoogle Scholar
  27. 27.
    Reynolds, J., Chelazzi, L., Desimone, R. (1999). Competitive mechanisms subserve attention in macaque areas V2 and V4, The Journal of Neuroscience, 19(5), p1736–1753.Google Scholar
  28. 28.
    Caputo, G., Guerra, S. (1998). Attentional selection by distractor suppression, Vision Research 38(5), p669–689.CrossRefGoogle Scholar
  29. 29.
    Bahcall, D., Kowler, E. (1999). Attentional interference at small spatial separations, Vision Research 39(1), p71–86.CrossRefGoogle Scholar
  30. 30.
    Tsotsos, J. K., Culhane, S., Cutzu, F. (2001). From theoretical foundations to a hierarchical circuit for selective attention, Visual Attention and Cortical Circuits, ed. by J. Braun, C. Koch and J. Davis, p285–306, MIT Press.Google Scholar
  31. 31.
    Cutzu, F., Tsotsos, J. K., The selective tuning model of visual attention: Testing the predictions arisiing from the inhibitory surround mechanism, Vision Research, (in press)Google Scholar
  32. 32.
    Chelazzi, L., Duncan, J., Miller, E., Desimone, R. (1998). Responses of neurons in inferior temporal cortex during memory-guided visual search, J. Neurophysiology 80, p2918–2940.Google Scholar
  33. 33.
    Roelfsema, P., Lamme, V., Spekreijse, H. (1998). Object-based attention in the primary visual cortex of the macaque monkey, Nature 395, p376–380.CrossRefGoogle Scholar
  34. 34.
    Simoncelli, E. P. & Heeger, D. J. (1998). A model of neuronal responses in visual area MT. Vision Research, 38 (5), 743–761.CrossRefGoogle Scholar
  35. 35.
    Beardsley, S. A. & Vaina, L. M. (1998). Computational modeling of optic flow selectivity in MSTd neurons. Network: Computation in Neural Systems, 9, 467–493.CrossRefzbMATHGoogle Scholar
  36. 36.
    Giese, M. A. (2000). Neural field model for the recognition of biological motion. Paper presented at the Second International ICSC Symposium on Neural Computation (NC 2000), Berlin, Germany.Google Scholar
  37. 37.
    Meese, T. S. & Anderson, S. J. (2002). Spiral mechanisms are required to account for summation of complex motion components. Vision Research, 42, 1073–1080.CrossRefGoogle Scholar
  38. 38.
    Nowlan, S. J., Sejnowski, T. J., (1995). A Selection Model for Motion Processing in Area MT of Primates, The Journal of Neuroscience 15 (2), p 1195–1214.Google Scholar
  39. 39.
    Grossberg, S., Mingolla, E. & Viswanathan, L. (2001). Neural dynamics of motion integration and segmentation within and across apertures. Vision Research, 41, 2521–2553.CrossRefGoogle Scholar
  40. 40.
    Zemel, R. S., Sejnowski, T. J., (1998). A Model for Encoding Multiple Object Motions and Self-Motion in area MST of Primate visual cortex, The Journal of Neuroscience, 18(1), 531–547.Google Scholar
  41. 41.
    Pack, C., Grossberg, S. Mingolla, E., (2001). A nerual model of smooth pursuit control and motion perception by cortical area MST, Journal of Cognitive Neuroscience, 13(1), 102–120.CrossRefGoogle Scholar
  42. 42.
    Perrone, J. A. & Stone, L. S. (1998) Emulating the visual receptive field properties of MST neurons with a template model of heading estimation. The Journal of Neuroscience, 18, 5958–5975.Google Scholar
  43. 43.
    Orban, G. A., Kennedy, H. & Bullier, J. (1986). Velocity sensitivity and direction sensitivity of neurons in areas V1 and V2 of the monkey: Influence of eccentricity. Journal of Neurophysiology, 56 (2), 462–480.Google Scholar
  44. 44.
    Heeger, D. J. (1988). Optical flow using spatiotemporal filters. International Journal of Computer Vision, 1 (4), 279–302.CrossRefGoogle Scholar
  45. 45.
    Lagae, L., Raiguel, S. & Orban, G. A. (1993). Speed and direction selectivity of Macaque middle temporal neurons. Journal of Neurophysiology, 69 (1), 19–39.Google Scholar
  46. 46.
    Felleman, D. J. & Kaas, J. H. (1984). Receptive field properties of neurons in middle temporal visual area (MT) of owl monkeys. Journal of Neurophysiology, 52, 488–513.Google Scholar
  47. 47.
    Treue, S. & Andersen, R. A. (1996). Neural responses to velocity gradients in macaque cortical area MT. Visual Neuroscience, 13, 797–804.CrossRefGoogle Scholar
  48. 48.
    Graziano, M. S., Andersen, R. A. & Snowden, R. J. (1994). Tuning of MST neurons to spiral motions. Journal of Neuroscience, 14 (1), 54–67.Google Scholar
  49. 49.
    Duffy, C. J. & Wurtz, R. H. (1997). MST neurons respond to speed patterns in optic flow. Journal of Neuroscience, 17(8), 2839–2851.Google Scholar
  50. 50.
    Siegel, R. M. & Read, H. L. (1997). Analysis of optic flow in the monkey parietal area 7a. Cerebral Cortex, 7, 327–346CrossRefGoogle Scholar
  51. 51.
    Treue, S. & Maunsell, J. H. R. (1996). Attentional modulation of visual motion processing in cortical areas MT and MST. Nature, 382, 539–541.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • John K. Tsotsos
    • 1
  • Marc Pomplun
    • 2
  • Yueju Liu
    • 1
  • Julio C. Martinez-Trujillo
    • 1
  • Evgueni Simine
    • 1
  1. 1.Centre for Vision ResearchYork UniversityTorontoCanada
  2. 2.Department of Computer ScienceUniversity of Massachusetts at BostonBostonUSA

Personalised recommendations