Advertisement

The Role of Structure and Complexity on Reservoir Computing Quality

  • Matthew DaleEmail author
  • Jack Dewhirst
  • Simon O’Keefe
  • Angelika Sebald
  • Susan Stepney
  • Martin A. Trefzer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11493)

Abstract

We explore the effect of structure and connection complexity on the dynamical behaviour of Reservoir Computers (RC). At present, considerable effort is taken to design and hand-craft physical reservoir computers. Both structure and physical complexity are often pivotal to task performance, however, assessing their overall importance is challenging. Using a recently proposed framework, we evaluate and compare the dynamical freedom (referring to quality) of neural network structures, as an analogy for physical systems. The results quantify how structure affects the range of behaviours exhibited by these networks. It highlights that high quality reached by more complex structures is often also achievable in simpler structures with greater network size. Alternatively, quality is often improved in smaller networks by adding greater connection complexity. This work demonstrates the benefits of using abstract behaviour representation, rather than evaluation through benchmark tasks, to assess the quality of computing substrates, as the latter typically has biases, and often provides little insight into the complete computing quality of physical systems.

Keywords

Reservoir computing Unconventional computing Echo state networks Structure Complexity 

Notes

Acknowledgments

This work is part of the SpInspired project, funded by EPSRC grant EP/R032823/1. Jack Dewhirst is funded by an EPSRC DTP PhD studentship.

References

  1. 1.
    Bala, A., Ismail, I., Ibrahim, R., Sait, S.M.: Applications of metaheuristics in reservoir computing techniques: a review. IEEE Access 6, 58012–58029 (2018)CrossRefGoogle Scholar
  2. 2.
    Adamatzky, A.: Game of Life Cellular Automata, vol. 1. Springer, London (2010).  https://doi.org/10.1007/978-1-84996-217-9CrossRefzbMATHGoogle Scholar
  3. 3.
    Appeltant, L., et al.: Information processing using a single dynamical node as complex system. Nature Commun. 2, 468 (2011)CrossRefGoogle Scholar
  4. 4.
    Büsing, L., Schrauwen, B., Legenstein, R.: Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Comput. 22(5), 1272–1311 (2010)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Crutchfield, J.P.: The calculi of emergence. Physica D 75(1–3), 11–54 (1994)CrossRefGoogle Scholar
  6. 6.
    Dale, M.: Neuroevolution of hierarchical reservoir computers. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 410–417. ACM (2018)Google Scholar
  7. 7.
    Dale, M., Miller, J.F., Stepney, S., Trefzer, M.A.: Evolving carbon nanotube reservoir computers. In: Amos, M., Condon, A. (eds.) UCNC 2016. LNCS, vol. 9726, pp. 49–61. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-41312-9_5CrossRefGoogle Scholar
  8. 8.
    Dale, M., Miller, J.F., Stepney, S., Trefzer, M.A.: Reservoir computing in materio: an evaluation of configuration through evolution. In: 2016 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8, December 2016Google Scholar
  9. 9.
    Dale, M., Miller, J.F., Stepney, S., Trefzer, M.A.: Reservoir computing in materio: a computational framework for in materio computing. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2178–2185, May 2017Google Scholar
  10. 10.
    Dale, M., Miller, J.F., Stepney, S., Trefzer, M.A.: A substrate-independent framework to characterise reservoir computers. arXiv preprint arXiv:1810.07135 (2018)
  11. 11.
    Gallicchio, C., Micheli, A., Pedrelli, L.: Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017)CrossRefGoogle Scholar
  12. 12.
    Goudarzi, A., Lakin, M.R., Stefanovic, D.: DNA reservoir computing: a novel molecular computing approach. In: Soloveichik, D., Yurke, B. (eds.) DNA 2013. LNCS, vol. 8141, pp. 76–89. Springer, Cham (2013).  https://doi.org/10.1007/978-3-319-01928-4_6CrossRefzbMATHGoogle Scholar
  13. 13.
    Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. German National Research Center for Information Technology GMD Technical Report 148:34, Bonn, Germany (2001)Google Scholar
  14. 14.
    Jaeger, H.: Short term memory in echo state networks. GMD-Forschungszentrum Informationstechnik (2001)Google Scholar
  15. 15.
    Lavis, D.A.: Equilibrium statistical mechanics of lattice models. Springer, Dordrecht (2015).  https://doi.org/10.1007/978-94-017-9430-5CrossRefzbMATHGoogle Scholar
  16. 16.
    Legenstein, R., Maass, W.: Edge of chaos and prediction of computational performance for neural circuit models. Neural Networks 20(3), 323–334 (2007)CrossRefGoogle Scholar
  17. 17.
    Lehman, J., Stanley, K.O.: Exploiting open-endedness to solve problems through the search for novelty. In: ALIFE, pp. 329–336 (2008)Google Scholar
  18. 18.
    Lloyd, S.: Ultimate physical limits to computation. Nature 406(6799), 1047 (2000)CrossRefGoogle Scholar
  19. 19.
    Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-35289-8_36CrossRefGoogle Scholar
  20. 20.
    Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)CrossRefGoogle Scholar
  21. 21.
    Paquot, Y., et al.: Optoelectronic reservoir computing. Scientific Reports, 2 (2012)Google Scholar
  22. 22.
    Pearson, J.E.: Complex patterns in a simple system. Science 261(5118), 189–192 (1993)CrossRefGoogle Scholar
  23. 23.
    Rodan, A., Tiňo, P.: Simple deterministically constructed recurrent neural networks. In: Fyfe, C., Tino, P., Charles, D., Garcia-Osorio, C., Yin, H. (eds.) IDEAL 2010. LNCS, vol. 6283, pp. 267–274. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15381-5_33CrossRefGoogle Scholar
  24. 24.
    Rodan, A., Tino, P.: Minimum complexity echo state network. IEEE Trans. Neural Networks 22(1), 131–144 (2011)CrossRefGoogle Scholar
  25. 25.
    Rodan, A., Tiňo, P.: Simple deterministically constructed cycle reservoirs with regular jumps. Neural Comput. 24(7), 1822–1852 (2012)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Schrauwen, B., Verstraeten, D., Van Campenhout, J.: An overview of reservoir computing: theory, applications and implementations. In: Proceedings of the 15th European Symposium on Artificial Neural Networks. Citeseer (2007)Google Scholar
  27. 27.
    Stepney, S.: The neglected pillar of material computation. Physica D 237(9), 1157–1164 (2008)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Tanaka, G., et al.: Recent advances in physical reservoir computing: a review. arXiv preprint arXiv:1808.04962 (2018)
  29. 29.
    Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Networks 20(3), 391–403 (2007)CrossRefGoogle Scholar
  30. 30.
    Xue, Y., Yang, L., Haykin, S.: Decoupled echo state networks with lateral inhibition. Neural Networks 20(3), 365–376 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Matthew Dale
    • 1
    • 4
    Email author
  • Jack Dewhirst
    • 1
    • 4
  • Simon O’Keefe
    • 1
    • 4
  • Angelika Sebald
    • 2
    • 4
  • Susan Stepney
    • 1
    • 4
  • Martin A. Trefzer
    • 3
    • 4
  1. 1.Department of Computer ScienceUniversity of YorkYorkUK
  2. 2.Department of ChemistryUniversity of YorkYorkUK
  3. 3.Department of Electronic EngineeringUniversity of YorkYorkUK
  4. 4.York Cross-disciplinary Centre for Systems AnalysisYorkUK

Personalised recommendations