Bimodal Incremental Self-Organizing Network (BiSON) with Application to Learning Chinese Characters

  • Andrew P. Papliński
  • William M. Mount
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8226)


We present a recurrent learning system that can incrementally integrate stimuli in two modalities, visual and auditory. The system consists of five self-organizing modules, each mapping input stimuli into respective latent spaces. Two sensory modules convert the input stimuli into an internal 3-D “neuronal code”. The central module integrates the bimodal information, and through modulatory top-down feedback influences the organization of data in two unimodal association units. Two feedback gains control the strength of the feedback connection. As an example we selected a set of Chinese characters and related spoken words. It is shown that the learning system can build a stable neuronal structure for incrementally applied visual and auditory stimuli.


Multimodal Learning Visual and Auditory stimuli Recurrent networks Self-organization Chinese characters 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Atteveldt, N.M.: Speech Meets Script: FMRI Studies on the Integration of Letters and Speech Sounds. University of Maastricht (2006)Google Scholar
  2. 2.
    Bookheimer, S.: How the brain reads Chinese characters. Neuroreport 12(1) (January 2001)Google Scholar
  3. 3.
    Calvert, G.A., Brammer, M.J., Iversen, S.D.: Crossmodal identification. Current Biology 2, 247–253 (1998)Google Scholar
  4. 4.
    Chou, S., Papliński, A.P., Gustafsson, L.: Speaker-dependent bimodal integration of Chinese phonemes and letters using multimodal self-organizing networks. In: Proc. Int. Joint Conf. Neural Networks, Orlando, Florida, pp. 248–253 (August 2007)Google Scholar
  5. 5.
    Dehaene, S.: Reading in the Brain. Viking (2009),
  6. 6.
    Druckmann, S., Chklovskii, D.B.: Over-complete representations on recurrent neural networks can support persistent percepts. Advances in Neural Information Processing Systems (2010)Google Scholar
  7. 7.
    Fitch, W.T.: The Evolution of Language. Cambridge University Press (2010)Google Scholar
  8. 8.
    Jantvik, T., Gustafsson, L., Papliński, A.P.: A self-organized artificial neural network architecture for sensory integration with applications to letter–phoneme integration. Neural Computation 23, 2101–2139 (2011)CrossRefGoogle Scholar
  9. 9.
    Kohonen, T.: Self-Organising Maps, 3rd edn. Springer, Berlin (2001)Google Scholar
  10. 10.
    Papliński, A.P.: Rotation invariant categorization of visual objects using Radon transform and self-organizing modules. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds.) ICONIP 2010, Part II. LNCS, vol. 6444, pp. 360–366. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Papliński, A.P.: Incremental self-organizing map (iSOM) in categorization of visual objects. In: Huang, T., Zeng, Z., Li, C., Leung, C.S. (eds.) ICONIP 2012, Part II. LNCS, vol. 7664, pp. 125–132. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Papliński, A.P.: Rotation invariant categorization of colour images using Radon transform. In: Proc. WCCI–IJCNN, pp. 1408–1413. IEEE (2012)Google Scholar
  13. 13.
    Papliński, A.P.: The angular integral of the radon transform (aniRT) as a feature vector in categorization of visual objects. In: Guo, C., Hou, Z.-G., Zeng, Z. (eds.) ISNN 2013, Part I. LNCS, vol. 7951, pp. 523–531. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Papliński, A.P., Gustafsson, L., Mount, W.M.: A model of binding concepts to spoken names. Aust. Journal of Intelligent Information Processing Systems 11(2), 1–5 (2010)Google Scholar
  15. 15.
    Papliński, A.P., Gustafsson, L., Mount, W.M.: A recurrent multimodal network for binding written words and sensory-based semantics into concepts. In: Lu, B.-L., Zhang, L., Kwok, J. (eds.) ICONIP 2011, Part I. LNCS, vol. 7062, pp. 413–422. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  16. 16.
    Siok, W., Fletcher, P.: The role of phonological awareness and visual orthographic skills in Chinese reading acquisition. Developmental Psychology 37(6), 886–889 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Andrew P. Papliński
    • 1
  • William M. Mount
    • 2
  1. 1.Monash UniversityAustralia
  2. 2.University of New South WalesAustralia

Personalised recommendations