Abstract
The Mixture of Experts (ME) model is a type of modular artificial neural network (MANN) specially suitable when the search space is stratified and whose architecture is composed by different kinds of networks which compete to learn several aspects of a complex problem. Training a ME architecture can be treated as a maximum likelihood estimation problem, where the Expectation Maximization (EM) algorithm decouples the estimation process in a manner that fits well with the modular structure of the ME architecture. However, the learning process relies on the data and so is the performance. When the data is exposed to outliers, the model is affected by being sensible to these deviations obtaining a poor performance as it is shown in this work. This paper proposes a Robust Expectation Maximization algorithm for learning a ME model (REM-ME) based on M-estimators. We show empirically that the REM-ME for these architectures prevents performance deterioration due to outliers and yields significantly faster convergence than other approaches.
This work was supported in part by Research Grant Fondecyt 1010101 and 7010101, in part by Research Grant CHL-99/023 from the German Ministry of Education and Research (BMBF) and in part by Research Grant DGIP-UTFSM and in part by the Intership grant CONICYT-INRIA
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
H. Allende, C. Moraga, and R. Salas, Robust estimator for the learning process in neural networks applied in time series, LNCS 2415 (2002), 1080–1086.
A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, 39 (1977), 1–38.
F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel, Robust statistics, Wiley Series in Probability and Mathematical Statistics, 1986.
Peter J. Huber, Robust statistics, Wiley Series in probability and mathematical statistics, 1981.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, Adaptive mixtures of local experts, Neural Computation 3 (1991), no. 1, 79–87.
M. I. Jordan and L. Xu, Convergence properties of the EM approach to learning in mixture-of-experts architectures, Neural Networks 8 (1995), 1409–1431.
M. I. Jordan and R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm, Neural Computation 6 (1994), no. 2, 181–214.
N. Cambell, Mixture models and atypical values, Math. Geol. 16 (1984), 465–477.
L. Prechelt, Proben1— a set of benchmarks and benchmarking rules for neural training algorithms., Technical Report 21/94, Fakultaet fur Informatik, Universitaet Karlsruhe, D-76128 Karlsruhe, Germany (1994).
R. Torres, R. Salas, H. Allende, and C. Moraga, Estimador robusto en modelos de mezcla de expertos locales, CLATSE V (2002).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Torres, R., Salas, R., Allende, H., Moraga, C. (2003). Robust Expectation Maximization Learning Algorithm for Mixture of Experts. In: Mira, J., Álvarez, J.R. (eds) Computational Methods in Neural Modeling. IWANN 2003. Lecture Notes in Computer Science, vol 2686. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44868-3_31
Download citation
DOI: https://doi.org/10.1007/3-540-44868-3_31
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40210-7
Online ISBN: 978-3-540-44868-6
eBook Packages: Springer Book Archive