Abstract
A finite mixture model has been used to fit the data from heterogeneous populations to many applications. An Expectation Maximization (EM) algorithm is the most popular method to estimate parameters in a finite mixture model. A Bayesian approach is another method for fitting a mixture model. However, the EM algorithm often converges to the local maximum regions, and it is sensitive to the choice of starting points. In the Bayesian approach, the Markov Chain Monte Carlo (MCMC) sometimes converges to the local mode and is difficult to move to another mode. Hence, in this paper we propose a new method to improve the limitation of EM algorithm so that the EM can estimate the parameters at the global maximum region and to develop a more effective Bayesian approach so that the MCMC chain moves from one mode to another more easily in the mixture model. Our approach is developed by using both simulated annealing (SA) and adaptive rejection metropolis sampling (ARMS). Although SA is a well-known approach for detecting distinct modes, the limitation of SA is the difficulty in choosing sequences of proper proposal distributions for a target distribution. Since ARMS uses a piecewise linear envelope function for a proposal distribution, we incorporate ARMS into an SA approach so that we can start a more proper proposal distribution and detect separate modes. As a result, we can detect the maximum region and estimate parameters for this global region. We refer to this approach as ARMS annealing. By putting together ARMS annealing with the EM algorithm and with the Bayesian approach, respectively, we have proposed two approaches: an EM-ARMS annealing algorithm and a Bayesian-ARMS annealing approach. We compare our two approaches with traditional EM algorithm alone and Bayesian approach alone using simulation, showing that our two approaches are comparable to each other but perform better than EM algorithm alone and Bayesian approach alone. Our two approaches detect the global maximum region well and estimate the parameters in this region. We demonstrate the advantage of our approaches using an example of the mixture of two Poisson regression models. This mixture model is used to analyze a survey data on the number of charitable donations.
Similar content being viewed by others
References
Böhning D, Dietz E, Schlattmann P, Mendonca L, Kirchner U (1995) The zero-inflated Poisson model and the decayed, missing and filled teeth index in dental epidemiology. J R Stat Soc Ser A 162:194–209
Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B 39:1–38
Duncan B (1999) Modeling charitable contributions of time and money. J Public Econ 77:213–242
Gilks WR, Wild P (1992) Adaptive rejection sampling for Gibbs sampling. Appl Stat 41:337–348
Gilks WR, Best NG, Tan KKC (1995) Adaptive rejection Metropolis sampling. Appl Stat 44:455–472
Hastings WK (1970) Monte Carlo sampling methods using Markove chains and their applications. Biometrika 57:97–109
Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Sci New Ser 220:671–680
McLachlan G, Peel D (2000) Finite mixture models. Wiley, New York
Metropolis M, Rosenbluth AW, Rosenbluth MN, Teller AH, Teller E (1953) Equation of state calculations by fast computing machines. J Chem Phys 21:1087–1093
Park T, Park S (2004) An economic study on charitable giving of individuals in Korea: some new findings from 2002 survey data. Presented at the 6th ISTR conference, Toronto, Canada, 11–14 July 2004
Ripley BD (1987) Stochastic simulation. Wiley, New York
Smith VH, Kchoe MR, Creamer ME (1999) The private provision of public goods: altruism and voluntary giving. J Public Econ 77:213–242
Swendsen RH, Wang JS (1986) Replica Monte Carlo simulation of spin glasses. Phys Rev Lett 57:2607–2609
van den Broek J (1995) A score test for zero inflated in a Poisson regression. Biometrics 51:738–743
von Neumann J (1951) Various techniques used in connection with random digits: Monte Carlo methods. Natl Bur Stan AMS 12:36–38
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, H., Kim, I. Adaptive Rejection Metropolis Simulated Annealing for Detecting Global Maximum Regions. Methodol Comput Appl Probab 18, 1–19 (2016). https://doi.org/10.1007/s11009-014-9395-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11009-014-9395-6
Keywords
- Adaptive rejection metropolis sampling
- Expectation Maximization algorithm
- Finite mixture models
- Simulated annealing