Abstract
While the No-Prop (no back propagation) algorithm uses the delta rule to train the output layer of a feed-forward network, No-Prop-fast employs fast linear regression learning using the Hopf-Wiener solution. Ten times faster learning speeds can be achieved on large datasets like the MNIST benchmark, compared to one of the fastest backpropagation algorithm known. Additionally, the plain feed-forward network No-prop-fast can distinguish gaze movements on cartoons with and without text, as well as age-specific attention shifts between text and picture areas with minimal pre-processing.
Continuously learning mobile robots and adaptive intelligent systems require such fast learning algorithms. Almost real-time learning speeds enable lower turn-around cycles in product development and data analysis.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Widrow, B., Hoff Jr., M.E.: Adaptive switching circuits. IRE WESCON Convention Record 4, 96–104 (1960)
Widrow, B., Greenblatt, A., Kim, Y., Park, D.: The no-prop algorithm: A new learning algorithm for multilayer neural networks. Neural Networks 37, 182–188 (2013)
Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: Kremer, S.C., Kolen, J.F. (eds.) A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press (2001)
LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backProp. In: Orr, G.B., Müller, K.-R. (eds.) NIPS-WS 1996. LNCS, vol. 1524, pp. 9–50. Springer, Heidelberg (1998)
Jäger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004)
Wang, Z.Q., Manry, M., Schiano, J.: Lms learning algorithms: misconceptions and new results on converence. IEEE Transactions on Neural Networks 11(1), 47–56 (2000)
Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes in C: The Art of Scientific Computing, 2nd edn. Cambridge University Press (1992)
Møller, M.F.: A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks 6(4), 525–533 (1993)
Ranzato, M., Poultney, C., Chopra, S., LeCun, Y.: Efficient learning of sparse representations with an energy-based model. In: Platt, J., et al. (eds.) Advances in Neural Information Processing Systems (NIPS 2006), vol. 19. MIT Press (2006)
Ciresan, D.C., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. CoRR abs/1202.2745 (2012)
Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Deep big simple neural nets excel on handwritten digit recognition. CoRR (2010)
Krause, A.F., Dürr, V., Bläsing, B., Schack, T.: Evolutionary optimization of echo state networks: multiple motor pattern learning. In: Madani, K. (ed.) 6th ANNIIP 2010, Funchal, Madeira, pp. 63–71 (June 2010)
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye tracking - A comprehensive guide to methods and measures. Oxford University Press, New York (2011)
Essig, K., Pomplun, M., Ritter, H.: A neural network for 3d gaze recording with binocular eye trackers. International Journal of Parallel, Emergent and Distributed Systems 21(2), 79–95 (2006)
Zhang, Y., Zhao, X., Fu, H., Liang, Z., Chi, Z., Zhao, X., Feng, D.: A time delay neural network model for simulating eye gaze data. Journal of Experimental & Theoretical Artificial Intelligence 23(1), 11–126 (2011)
Macaš, M., Lhotská, L., Novák, D.: Bio-inspired methods for analysis and classification of reading eye movements of dyslexic children. Technical report, University in Prague, Algarve, Portugal, October 3-5 (2005)
Sommer, D., Hink, T., Golz, M.: Application of learning vector quantization to detect drivers dozing-off. In: European Symposium on Intelligent Technologies, Hybrid Systems and their implementation on Smart Adaptive Systems, pp. 119–123 (2002)
Vo, T., Mendis, B.S.U., Gedeon, T.: Gaze pattern and reading comprehension. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds.) ICONIP 2010, Part II. LNCS, vol. 6444, pp. 124–131. Springer, Heidelberg (2010)
Zhu, D., Mendis, B.S.U., Gedeon, T., Asthana, A., Goecke, R.: A hybrid fuzzy approach for human eye gaze pattern recognition. In: Köppen, M., Kasabov, N., Coghill, G. (eds.) ICONIP 2008, Part II. LNCS, vol. 5507, pp. 655–662. Springer, Heidelberg (2009)
Krause, A.F., Essig, K., Essig-Shih, L.-Y., Schack, T.: Classifying the differences in gaze patterns of alphabetic and logographic L1 readers – A neural network approach. In: Iliadis, L., Jayne, C. (eds.) EANN/AIAI 2011, Part I. IFIP AICT, vol. 363, pp. 78–83. Springer, Heidelberg (2011)
Buckner, R.: Memory and executive functioning in aging and ad: Multiple factors that cause decline and reserve factors that compensate. Neuron 44, 195–208 (2004)
Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124, 372–422 (1998)
Henderson, J., Ferreira, F.: Scene perception for psycholinguists. In: The Interface of Language, Vision and Action: Eye Movements and the Visual World, pp. 1–58. Psychology Press, New York (2004)
Plauen, E.: Vater und Sohn (3 Bde.). Südverlag Konstanz, Konstanz (2000)
Watterson, B.: Calvin und Hobbes: Der Jubelband: 10 Jahre: 10 Jahre Jubel Buch. Carlsen Verlag, Hamburg (2008)
Kramer, A., Hahn, S., Irwin, D., Theuuwes, J.: Age differences in the control of looking behavior: Do you know where your eyes have been? Psychological Science 11, 210–217 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Krause, A.F., Essig, K., Piefke, M., Schack, T. (2013). No-Prop-fast - A High-Speed Multilayer Neural Network Learning Algorithm: MNIST Benchmark and Eye-Tracking Data Classification. In: Iliadis, L., Papadopoulos, H., Jayne, C. (eds) Engineering Applications of Neural Networks. EANN 2013. Communications in Computer and Information Science, vol 383. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41013-0_46
Download citation
DOI: https://doi.org/10.1007/978-3-642-41013-0_46
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-41012-3
Online ISBN: 978-3-642-41013-0
eBook Packages: Computer ScienceComputer Science (R0)