Adaptive Rejection Metropolis Simulated Annealing for Detecting Global Maximum Regions

  • Huaiye Zhang
  • Inyoung Kim


A finite mixture model has been used to fit the data from heterogeneous populations to many applications. An Expectation Maximization (EM) algorithm is the most popular method to estimate parameters in a finite mixture model. A Bayesian approach is another method for fitting a mixture model. However, the EM algorithm often converges to the local maximum regions, and it is sensitive to the choice of starting points. In the Bayesian approach, the Markov Chain Monte Carlo (MCMC) sometimes converges to the local mode and is difficult to move to another mode. Hence, in this paper we propose a new method to improve the limitation of EM algorithm so that the EM can estimate the parameters at the global maximum region and to develop a more effective Bayesian approach so that the MCMC chain moves from one mode to another more easily in the mixture model. Our approach is developed by using both simulated annealing (SA) and adaptive rejection metropolis sampling (ARMS). Although SA is a well-known approach for detecting distinct modes, the limitation of SA is the difficulty in choosing sequences of proper proposal distributions for a target distribution. Since ARMS uses a piecewise linear envelope function for a proposal distribution, we incorporate ARMS into an SA approach so that we can start a more proper proposal distribution and detect separate modes. As a result, we can detect the maximum region and estimate parameters for this global region. We refer to this approach as ARMS annealing. By putting together ARMS annealing with the EM algorithm and with the Bayesian approach, respectively, we have proposed two approaches: an EM-ARMS annealing algorithm and a Bayesian-ARMS annealing approach. We compare our two approaches with traditional EM algorithm alone and Bayesian approach alone using simulation, showing that our two approaches are comparable to each other but perform better than EM algorithm alone and Bayesian approach alone. Our two approaches detect the global maximum region well and estimate the parameters in this region. We demonstrate the advantage of our approaches using an example of the mixture of two Poisson regression models. This mixture model is used to analyze a survey data on the number of charitable donations.


Adaptive rejection metropolis sampling Expectation Maximization algorithm Finite mixture models Simulated annealing 

AMS 2000 Subject Classification



Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Böhning D, Dietz E, Schlattmann P, Mendonca L, Kirchner U (1995) The zero-inflated Poisson model and the decayed, missing and filled teeth index in dental epidemiology. J R Stat Soc Ser A 162:194–209Google Scholar
  2. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B 39:1–38MathSciNetMATHGoogle Scholar
  3. Duncan B (1999) Modeling charitable contributions of time and money. J Public Econ 77:213–242CrossRefGoogle Scholar
  4. Gilks WR, Wild P (1992) Adaptive rejection sampling for Gibbs sampling. Appl Stat 41:337–348CrossRefMATHGoogle Scholar
  5. Gilks WR, Best NG, Tan KKC (1995) Adaptive rejection Metropolis sampling. Appl Stat 44:455–472CrossRefMATHGoogle Scholar
  6. Hastings WK (1970) Monte Carlo sampling methods using Markove chains and their applications. Biometrika 57:97–109CrossRefMATHGoogle Scholar
  7. Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Sci New Ser 220:671–680MathSciNetMATHGoogle Scholar
  8. McLachlan G, Peel D (2000) Finite mixture models. Wiley, New YorkCrossRefMATHGoogle Scholar
  9. Metropolis M, Rosenbluth AW, Rosenbluth MN, Teller AH, Teller E (1953) Equation of state calculations by fast computing machines. J Chem Phys 21:1087–1093CrossRefGoogle Scholar
  10. Park T, Park S (2004) An economic study on charitable giving of individuals in Korea: some new findings from 2002 survey data. Presented at the 6th ISTR conference, Toronto, Canada, 11–14 July 2004Google Scholar
  11. Ripley BD (1987) Stochastic simulation. Wiley, New YorkCrossRefMATHGoogle Scholar
  12. Smith VH, Kchoe MR, Creamer ME (1999) The private provision of public goods: altruism and voluntary giving. J Public Econ 77:213–242Google Scholar
  13. Swendsen RH, Wang JS (1986) Replica Monte Carlo simulation of spin glasses. Phys Rev Lett 57:2607–2609CrossRefMathSciNetGoogle Scholar
  14. van den Broek J (1995) A score test for zero inflated in a Poisson regression. Biometrics 51:738–743CrossRefMathSciNetMATHGoogle Scholar
  15. von Neumann J (1951) Various techniques used in connection with random digits: Monte Carlo methods. Natl Bur Stan AMS 12:36–38Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of StatisticsVirginia Polytechnic Institute and State UniversityBlacksburgUSA

Personalised recommendations