Abstract
Minimum divergence estimators are derived through the dual form of the divergence in parametric models. These estimators generalize the classical maximum likelihood ones. Models with unobserved data, as mixture models, can be estimated with EM algorithms, which are proved to converge to stationary points of the likelihood function under general assumptions. This paper presents an extension of the EM algorithm based on minimization of the dual approximation of the divergence between the empirical measure and the model using a proximal-type algorithm. The algorithm converges to the stationary points of the empirical criterion under general conditions pertaining to the divergence and the model. Robustness properties of this algorithm are also presented. We provide another proof of convergence of the EM algorithm in a two-component gaussian mixture. Simulations on Gaussian and Weibull mixtures are performed to compare the results with the MLE.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
More investigation is needed here since we may use asymmetric kernels to overcome this difficulty.
- 2.
Normally, \(\mu _2^l\) is bounded; still, we can extract a subsequence which converges.
References
Al Mohamad, D.: Towards a better understanding of the dual representation of phi divergences. ArXiv e-prints (2015)
Broniatowski, M., Keziou, A.: Minimization of divergences on sets of signed measures. Studia Sci. Math. Hungar. 43(4), 403–442 (2006)
Broniatowski, M., Keziou, A.: Parametric estimation and tests through divergences and the duality technique. J. Multivar. Anal. 100(1), 16–36 (2009)
Chretien, S., Hero, A.O.: Acceleration of the EM algorithm via proximal point iterations. In: Proceedings of the 1998 IEEE International Symposium on Information Theory, 1998, p. 444 (1998)
Chrétien, S., Hero, A.O.: Generalized proximal point algorithms and bundle implementations. Technical report, Department of Electrical Engineering and Computer Science, The University of Michigan (1998)
Chrétien, S., Hero, A.O.: On EM algorithms and their proximal generalizations. ESAIM: Probab. Stat. 12, 308–326 (2008)
Csiszár, I.: Eine informationstheoretische Ungleichung und ihre anwendung auf den Beweis der ergodizität von Markoffschen Ketten. Publications of the Mathematical Institute of Hungarian Academy of Sciences 8, 95–108 (1963)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc. Ser. B 39(1), 1–38 (1977)
Goldstein, A.A., Russak, I.B.: How good are the proximal point algorithms? Numer. Funct. Anal. Optim. 9(7–8), 709–724 (1987)
Liese, F., Vajda, I.: On divergences and informations in statistics and information theory. IEEE Trans. Inf. Theor. 52(10), 4394–4412 (2006)
McLachlan, G., Krishnan, T.: The EM Algorithm and Extensions. Wiley Series in Probability and Statistics. Wiley, Hoboken (2007)
Ostrowski, A.M.: Solution of Equations and Systems of Equations. Pure and Applied Mathematics. Academic Press, New York (1966)
Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Die Grundlehren der mathematischen Wissenschaften in Einzeldarstellungen, 3rd edn. Springer, Heidelberg (1998)
Toma, A., Broniatowski, M.: Dual divergence estimators and tests: robustness results. J. Multivar. Anal. 102(1), 20–36 (2011)
Tseng, P.: An analysis of the EM algorithm and entropy-like proximal point methods. Math. Oper. Res. 29(1), 27–44 (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Al Mohamad, D., Broniatowski, M. (2015). Generalized EM Algorithms for Minimum Divergence Estimation. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2015. Lecture Notes in Computer Science(), vol 9389. Springer, Cham. https://doi.org/10.1007/978-3-319-25040-3_45
Download citation
DOI: https://doi.org/10.1007/978-3-319-25040-3_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25039-7
Online ISBN: 978-3-319-25040-3
eBook Packages: Computer ScienceComputer Science (R0)