Abstract
We present a modelling framework for the investigation of prototype-based classifiers in non-stationary environments. Specifically, we study Learning Vector Quantization (LVQ) systems trained from a stream of high-dimensional, clustered data. We consider standard winner-takes-all updates known as LVQ1. Statistical properties of the input data change on the time scale defined by the training process. We apply analytical methods borrowed from statistical physics which have been used earlier for the exact description of learning in stationary environments. The suggested framework facilitates the computation of learning curves in the presence of virtual and real concept drift. Here we focus on time-dependent class bias in the training data. First results demonstrate that, while basic LVQ algorithms are suitable for the training in non-stationary environments, weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Zliobaite I, Pechenizkiy M, Gama J (2016) An overview of concept drift applications. In: Big data analysis new algorithms for a new society. Springer
Hastie, T, Tibshirani, R, Friedman, J (2001) The elements of statistical learning: data mining, inference, and prediction. Springer
Amunts K, et al (ed) (2014) Brain-inspired computing. In: Second international workshop BrainComp 2015. LNCS, vol 10087. Springer
Losing V, Hammer B, Wersing H (2017) Incremental on-line learning: a review and of state of the art algorithms. Neurocomputing 275:1261–1274
Ditzler G, Roveri M, Alippi C, Polikar R (2015) Learning in nonstationary environment: a survey. Comput Intell Mag 10(4):12–25
Joshi J, Kulkarni P (2012) Incremental learning: areas and methods - a survey. Int J Data Mining Knowl Manag Process 2(5):43–51
Ade R, Desmukh P (2013) Methods for incremental learning - a survey. Int J Data Mining Knowl Manag Process 3(4):119–125
Straat M, Abadi F, Göpfert C, Hammer B, Biehl M (2018) Statistical mechanics of on-line learning under concept drift. Entropy 20(10):775
Kohonen, T (2001) Self-organizing maps. Springer series in information sciences, 2nd edn., vol 30. Springer
Nova D, Estevez PA (2014) A review of learning vector quantization classifiers. Neural Comput Appl 25(3–4):511–524
Biehl M, Hammer B, Villmann T (2016) Prototype-based models in machine learning. Cognit Sci 7(2):92–111 Wiley Interdisciplinary Reviews
Biehl M, Ghosh A, Hammer B (2007) Dynamics and generalization ability of LVQ algorithms. J Mach Learn Res 8:323–360
Saad D (ed) (1999) On-line learning in neural networks. Cambridge University Press, New York
Engel A, van den Broeck C (2001) The statistical mechanics of learning. Cambridge University Press, New York
Watkin TLH, Rau A, Biehl M (1993) The statistical mechanics of learning a rule. Rev Mod Phys 65(2):499–556
Biehl M, Freking A, Reents G (1997) Dynamics of on-line competitive learning. Europhys Lett 38:73–78
Barkai N, Seung HS, Sompolinsky H (1993) Scaling laws in learning of classification tasks. Phys Rev Lett 70(20):L97–L103
Marangi C, Biehl M, Solla SA (1995) Supervised learning from clustered input examples. Europhys Lett 30:117–122
Biehl M, Schwarze H (1993) Learning drifting concepts with neural networks. J Phys A Math Gen 26:2651–2665
Vicente R, Caticha N (1998) Statistical mechanics of on-line learning of drifting concepts: a variational approach. Mach Learn 32(2):179–201
Reents G, Urbanczik R (1998) Self-averaging and on-line learning. Phys Rev Lett 80(24):5445–5448
Mezard M, Nadal JP, Toulouse G (1986) Solvable models of working memories. J de Phys (Paris) 47(9):1457–1462
van Hemmen JL, Keller G, Kühn R (1987) Forgetful memories. Europhys Lett 5(7):663–668
Saad D, Solla SA (1997) Learning with noise and regularizers in multilayer neural networks. In: Neural information processing system (NIPS 9). MIT Press, pp 260–266
Wang, S, Minku LL, Yao X (2017) A systematic study of online class imbalance learning with concept drift. CoRR abs/1703.06683
Acknowledgement
The authors would like to thank A. Ghosh, A. Witoelar and G.-J. de Vries for useful discussions of earlier projects on LVQ training in stationary environments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Biehl, M., Abadi, F., Göpfert, C., Hammer, B. (2020). Prototype-Based Classifiers in the Presence of Concept Drift: A Modelling Framework. In: Vellido, A., Gibert, K., Angulo, C., Martín Guerrero, J. (eds) Advances in Self-Organizing Maps, Learning Vector Quantization, Clustering and Data Visualization. WSOM 2019. Advances in Intelligent Systems and Computing, vol 976. Springer, Cham. https://doi.org/10.1007/978-3-030-19642-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-19642-4_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-19641-7
Online ISBN: 978-3-030-19642-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)