An Integration Principle for Multimodal Sensor Data Based on Temporal Coherence of Self-Organized Patterns

  • Emilia I. Barakova
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2085)


The world around us offers continuously huge amounts of information, from which living organisms can elicit the knowledge and understanding they need for survival or well-being. A fundamental cognitive feature, that makes this possible is the ability of a brain to integrate the inputs it receives from different sensory modalities into a coherent description of its surrounding environment. By analogy, artificial autonomous systems are designed to record continuously large amounts of data with various sensors. A major design problem by the last is the lack of reference of how the information from the different sensor streams can be integrated into a consistent description. This paper focuses on the development of a sinergistic integration principle, supported by the synchronization of the multimodal information streams on temporal coherence principle. The processing of the individual information streams is done by a self organizing neural algorithm, known as Neural gas algorithm. The integration itself uses a supervised learning method to allow the various information streams to interchange their knowledge as emerged experts. Two complementary data streams, recorded by exploration of autonomous robot of unprepared environments are used to simultaneously illustrate and motivate in a concrete sense the developed integration approach.


Processing Stream Autonomous Robot Dynamic Trajectory Surrounding World Temporal Synchronization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Barakova, E.I. & Zimmer, U.R. Dynamical situation and trajectory discrimination by means of clustering and accumulation of raw range measurements, Proc. of the Intl. Conf. on Advances in Intelligent Systems, Canberra, Australia, February 2000.Google Scholar
  2. [2]
    Barakova, E.I. & Zimmer, U.R. “Global spatial modelling based on dynamics identification according to discriminated static sensations” Underwater Technology 2000, Tokyo, Japan, May 23-26 2000.Google Scholar
  3. [3]
    Becker, S. and Hinton, G.E. (1992). A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355:161–163.CrossRefGoogle Scholar
  4. [4]
    Grossberg, S. The complementary brain: A Unifying View of Brain Specialization and Modularity.Google Scholar
  5. [5]
    Elfes, A. “Sonar-based real-world mapping and navigation”, IEEE Journal of Robotics and Automation, RA-3(3):249–265, June 1987.CrossRefGoogle Scholar
  6. [6]
    Eagleman, D.M. & Sejnowski, T.J. (2000) Motion integration and postdiction in visual awareness. Science. 287(5460): 2036–8.CrossRefGoogle Scholar
  7. [7]
    Cramer, K.C. and Mriganka Sur. Activity-dependent remodeling of connections in the mammalian visual system. Current Opinion in Neurobiology, 5:106,111, 1995.CrossRefGoogle Scholar
  8. [8]
    Martinetz. T.M., S.G. Berkovich, K.J. Schulten, ‘Neural-Gas’ Network for Vector Quantization and its Application to Time-Series Prediction, IEEE Transactions on Neural Networks, Vol. 4,No.4, July 1993, pp. 558–569.CrossRefGoogle Scholar
  9. [9]
    von der Malsburg, C. The correlation theory of brain function., in Models of Neural networks II, edited by E. Domany, J.L. van Hemmen, and K. Schulten (Springer, Berlin, 1994) Chapter 2, pp. 95, 119.Google Scholar
  10. [10]
    von der Malsburg, C. Binding in Models of Perception and Brain Function. Current Opinion in Neurobiology, 5:520, 526, 1995.CrossRefGoogle Scholar
  11. [11]
    McGurk, H. and MacDonald, J. (1976)., Hearing lips and seeing voices., Nature, 264:746–748.CrossRefGoogle Scholar
  12. [12]
    de Sa, V.R. and Dana H. Ballard, Category Learning through Multi-Modality Sensing, In Neural Computation 10(5) 1998.Google Scholar
  13. [13]
    Triesch, J. (2000). Democratic Integration: A theory of adaptive sensory integration. Technical Report NRL TR 00.1, Nat’l. Resource Lab’y. for the Study of Brain and Behavior,U. Rochester.Google Scholar
  14. [14]
    Zeki, S. Functional specialization in the visual cortex of the rhesus monkey, Nature (Lond.)274,423–8.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Emilia I. Barakova
    • 1
  1. 1.GMD - Japan Research LaboratoryKitakyushu-cityJapan

Personalised recommendations