Information Fusion for Perceptual Feedback: A Brain Activity Sonification Approach

  • Tomasz M. Rutkowski
  • Andrzej Cichocki
  • Danilo Mandic

When analysing multichannel processes, it is often convenient to use some sort of “visualisation” to help understand and interpret spatio-temporal dependencies between the channels, and to perform input variable selection. This is particularly advantageous when the levels of noise are high, the “active” channel changes its spatial location with time, and also for spatio-temporal processes where several channels contain meaningful information, such as in the case of electroencephalogram (EEG)-based brain activity monitoring. To provide insight into the dynamics of brain electrical responses, spatial sonification of multichannel EEG is performed, whereby the information from active channels is fused into music-like audio. Owing to its “data fusion via fission” mode of operation, empirical mode decomposition (EMD) is employed as a time-frequency analyser, and the brain responses to visual stimuli are sonified to provide audio feedback. Such perceptual feedback has enormous potential in multimodal brain computer and brain machine interfaces (BCI/BMI).


Empirical Mode Decomposition Auditory Feedback Information Fusion Intrinsic Mode Function Brain Computer Interface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Eerola, T., Toiviainen, P.: MIR in Matlab: The MIDI toolbox. In: Proceedings of the 5th International Conference on Music Information Retrieval, ISMIR2004, pp. 22–27. Audiovisual Institute, Universitat Pompeu Fabra, Barcelona, Spain (2004)Google Scholar
  2. 2.
    Fahle, M., Poggio, T. (eds.): Perceptual Learning. MIT, Cambridge, MA (2002)Google Scholar
  3. 3.
    Huang, N., Shen, Z., Long, S., Wu, M., Shih, H., Zheng, Q., Yen, N.C., Tung, C., Liu, H.: The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 454(1971), 903–995 (1998)MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Kelly, S.P., Lalor, E.C., Finucane, C., McDarby, G., Reilly, R.B.: Visual spatial attention control in an independent brain–computer interface. IEEE Transactions on Biomedical Engineering 52(9), 1588–1596 (2005)CrossRefGoogle Scholar
  5. 5.
    Miranda, E., Brouse, A.: Interfacing the brain directly with musical systems: On developing systems for making music with brain signals. Leonardo 38(4), 331–336 (2005)CrossRefGoogle Scholar
  6. 6.
    Palaniappan, R., Mandic, D.P.: Biometric from the brain electrical activity: A machine learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(4), 738–742 (2007)CrossRefGoogle Scholar
  7. 7.
    Przybyszewski, A., Rutkowski, T.: Processing of the incomplete representation of the visual world. In: Proceedings of the First Warsaw International Seminar on Intelligent Systems, WISIS’04. Warsaw, Poland (2004)Google Scholar
  8. 8.
    Rilling, G., Flandrin, P., Goncalves, P.: On empirical mode decomposition and its algorithms. In: Proceedings of IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, NSIP-03. IEEE (2003)Google Scholar
  9. 9.
    Rutkowski, T.M., Vialatte, F., Cichocki, A., Mandic, D., Barros, A.K.: Knowledge-Based Intelligent Information and Engineering Systems, Lecture Notes in Artificial Intelligence, vol. 4253, chap. Auditory Feedback for Brain Computer Interface Management – An EEG Data Sonification Approach, pp. 1232–1239. Springer, Berlin, Heidelberg, New York (2006)Google Scholar
  10. 10.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain–computer interfaces for communication and control. Clinical Neurophysiology 113, 767–791 (2002)CrossRefGoogle Scholar
  11. 11.
    Wolpaw, J.R., McFarland, D.J.: Control of a two-dimensional movement signal by a noninvasive brain–computer interface in humans. Proceedings of National Academy of Sciences of the United States America 101(51), 17849–17854 (2004)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Tomasz M. Rutkowski
    • 1
  • Andrzej Cichocki
    • 1
  • Danilo Mandic
    • 2
  1. 1.Brain Science InstituteJapan
  2. 2.Imperial College LondonLondonUK

Personalised recommendations