Modeling Complex Symbolic Sequences with Neural Based Systems

  • P. Tiňo
  • V. Vojtek
Conference paper


We study the problem of modeling long, complex symbolic sequences with recurrent neural networks (RNNs) and stochastic machines (SMs). RNNs are trained to predict the next symbol and the training process is monitored with information theory based performance measures. SMs are constructed using Kohonen self-organizing map quantizing RNN state space. We compare generative models through entropy spectra computed from sequences, or directly from the machines.


Recurrent Neural Network Finite State Machine Symbolic Sequence State Transition Matrix Cross Entropy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    J.P. Crutchfield and K. Young. Computation at the onset of chaos. In W.H. Zurek, editor, Complexity, Entropy, and the physics of Information, SFI Studies in the Sciences of Complexity, vol 8, pages 223–269. Addison-Wesley, 1990.Google Scholar
  2. [2]
    J. Hertz, A. Krogh, and R.G. Palmer. Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA, 1991.Google Scholar
  3. [3]
    T. Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464–1479, 1990.Google Scholar
  4. [4]
    P. Tino and M. Koteles. Modeling complex sequences with neural and hybrid neural based approaches. Technical Report STUFEI-DCSTR-96-49, Slovak University of Technology, Bratislava, Slovakia, September 1996.Google Scholar
  5. [5]
    P. Tino and J. Sajda. Learning and extracting initial mealy machines with a modular neural network model. Neural Computation, 7(4):822–844, 1995.CrossRefGoogle Scholar
  6. [6]
    K. Young and J.P. Crutchfield. Fluctuation spectroscopy. In W. Ebeling, editor, Chaos, Solitons, and Fractals, special issue on Complexity, 1993.Google Scholar
  7. [7]
    J. Ziv and N. Merhav. A measure of relative entropy between individual sequences with application to universal classification. IEEE Transactions on Information Theory, 39(4):1270–1279, 1993.MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Wien 1998

Authors and Affiliations

  • P. Tiňo
    • 1
  • V. Vojtek
    • 1
  1. 1.Department of Computer Science and EngineeringSlovak University of TechnologyBratislavaSlovakia

Personalised recommendations