Skip to main content
SpringerLink
Log in

The EM Algorithm, Its Randomized Implementation and Global Optimization: Some Challenges and Opportunities for Operations Research

  • Chapter
Perspectives in Operations Research

Part of the book series: Operations Research/Computer Science Interfaces Series ((ORCS,volume 36))

Summary

The EM algorithm is a very powerful optimization method and has become popular in many fields. Unfortunately, EM is only a local optimization method and can get stuck in sub-optimal solutions. While more and more contemporary data/model combinations yield multiple local optima, there have been only very few attempts at making EM suitable for global optimization. In this paper we review the basic EM algorithm, its properties and challenges, and we focus in particular on its randomized implementation. The randomized EM implementation promises to solve some of the contemporary data/model challenges, and it is particularly well-suited for a wedding with global optimization ideas, since most global optimization paradigms are also based on the principles of randomization. We review some of the challenges of the randomized EM implementation and present a new algorithm that combines the principles of EM with that of the Genetic Algorithm. While this new algorithm shows some promising results for clustering of an online auction database of functional objects, the primary goal of this work is to bridge a gap between the field of statistics, which is home to extensive research on the EM algorithm, and the field of operations research, in which work on global optimization thrives, and to stimulate new ideas for joint research between the two.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. S. Amari. Information geometry of the EM and EM algorithms for neural networks. Neural Networks, 8:1379–1408, 1995.

    Article  Google Scholar 

  2. J. Booth and J. Hobert. Standard errors of prediction in generalized linear mixed models. Journal of the American Statistical Association, 93:262–272, 1998.

    Article  MathSciNet  Google Scholar 

  3. J. Booth and J. Hobert. Maximizing generalized linear mixed model likelihoods with an automated Monte Carlo EM algorithm. Journal of the Royal Statistical Society B, 61:265–285, 1999.

    Article  Google Scholar 

  4. J. Booth, J. Hobert and W. Jank. A survey of Monte Carlo algorithms for maximizing the likelihood of a two-stage hierarchical model. Statistical Modelling, 1:333–349, 2001.

    Article  Google Scholar 

  5. Z. Botev and D. Kroese. Global likelihood optimization via the cross-entropy method with an application to mixture models. In Proceedings of the 2004 Winter Simulation Conference, pages 529–535. IEEE Press, 2004.

    Google Scholar 

  6. R. Boyles. On the convergence of the EM algorithm. Journal of the Royal Statistical Society B, 45:47–50, 1983.

    MathSciNet  Google Scholar 

  7. S. Caffo, J. Booth and A. Davison. Empirical sup rejection sampling. Biometrika, 89:745–754, 2002.

    Article  MathSciNet  Google Scholar 

  8. B. Caffo, W. Jank and G. Jones. Ascent-Based Monte Carlo EM. Journal of the Royal Statistical Society, Series B, 67:235–252, 2005.

    Article  MathSciNet  Google Scholar 

  9. G. Celeux and J. Diebolt. A stochastic approximation type EM algorithm for the mixture problem. Stochastics and Stochastics Reports, 41:127–146, 1992.

    MathSciNet  Google Scholar 

  10. K. Chan and J. Ledolter. Monte Carlo EM estimation for time series models involving counts. Journal of the American Statistical Association, 90:242–252, 1995.

    Article  MathSciNet  Google Scholar 

  11. B. Delyon, M. Lavielle and E. Moulines. Convergence of a stochastic approximation version of the EM algorithm. The Annals of Statistics, 27:94–128, 1999.

    Article  MathSciNet  Google Scholar 

  12. A. Dempster, N. Laird and D. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39:1–22, 1977.

    MathSciNet  Google Scholar 

  13. M. Feder and E. Weinstein. Parameter estimation of superimposed signals using the EM algorithm. Acoustics, Speech, and Signal Processing, 36:477–489, 1988.

    Article  Google Scholar 

  14. G. Fort and E. Moulines. Convergence of the Monte Carlo expectation maximization for curved exponential families. The Annals of Statistics, 31:1220–1259, 2003.

    Article  MathSciNet  Google Scholar 

  15. M. Gu and S. Li. A stochastic approximation algorithm for maximum likelihood estimation with incomplete data. Canadian Journal of Statistics, 26:567–582, 1998.

    Article  MathSciNet  Google Scholar 

  16. M. Gu and H.-T. Zhu. Maximum likelihood estimation for spatial models by Markov chain Monte Carlo stochastic approximation. Journal of the Royal Statistical Society B, 63:339–355, 2001.

    Article  MathSciNet  Google Scholar 

  17. J. Heath, M. Fu and W. Jank. Global optimization with MRAS, Cross Entropy and the EM algorithm. Working Paper, Smith School of Business, University of Maryland, 2006.

    Google Scholar 

  18. J. Holland. Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor, MI, 1975.

    Google Scholar 

  19. M. Jamshidian and R. Jennrich. Conjugate gradient acceleration of the EM algorithm. Journal of the American Statistical Association, 88:221–228, 1993.

    Article  MathSciNet  Google Scholar 

  20. M. Jamshidian and R. Jennrich. Acceleration of the EM algorithm by using Quasi-Newton methods. Journal of the Royal Statistical Society B, 59:569–587, 1997.

    Article  MathSciNet  Google Scholar 

  21. W. Jank and G. Shmueli. Dynamic profiling of online auctions using curve clustering. Technical report, Smith School of Business, University of Maryland, 2003.

    Google Scholar 

  22. W. Jank and J. Booth. Efficiency of Monte Carlo EM and simulated maxi mum likelihood in two-stage hierarchical models. Journal of Computational and Graphical Statistics, in print, 2002.

    Google Scholar 

  23. W. Jank. Implementing and diagnosing the stochastic approximation EM algorithm. Technical report, University of Maryland, 2004.

    Google Scholar 

  24. W. Jank. Quasi-Monte Carlo Sampling to Improve the Efficiency of Monte Carlo EM. Computational Statistics and Data Analysis, 48:685–701, 2004.

    Article  MathSciNet  Google Scholar 

  25. W. Jank. Ascent EM for fast and global model-based clustering: An application to curve-clustering of online auctions. Technical report, University of Maryland, 2005.

    Google Scholar 

  26. M. Jordan and R. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6:181–214, 1994.

    Article  Google Scholar 

  27. K. Lange. A gradient algorithm locally equivalent to the EM algorithm. Journal of the Royal Statistical Society B, 57:425–437, 1995.

    Google Scholar 

  28. S. Lauritzen. The EM algorithm for graphical association models with missing data. Computational Statistics and Data Analysis, 19:191–201, 1995.

    Article  Google Scholar 

  29. P. L’Ecuyer and C. Lemieux. Recent advances in randomized Quasi-Monte Carlo Methods. In M Dror, P L’Ecuyer, and F Szidarovszki, editors, Modeling Uncertainty: An Examination of Stochastic Theory, Methods, and Applications, pages 419–474. Kluwer Academic Publishers, 2002.

    Google Scholar 

  30. C. Lemieux and P. L’Ecuyer. Efficiency improvement by lattice rules for pricing asian options. In Proceedings of the 1998 Winter Simulation Conference, pages 579–586. IEEE Press, 1998.

    Google Scholar 

  31. R. Levine and G. Casella. Implementations of the Monte Carlo EM algorithm. Journal of Computational and Graphical Statistics, 10:422–439, 2001.

    Article  MathSciNet  Google Scholar 

  32. R. Levine and J. Fan. An automated (Markov Chain) Monte Carlo EM algorithm. Journal of Statistical Computation and Simulation, 74:349–359, 2004.

    Article  MathSciNet  Google Scholar 

  33. C. Liu and D. Rubin. The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence. Biometrika, 81:633–648, 1994.

    Article  MathSciNet  Google Scholar 

  34. C. McCulloch. Maximum likelihood algorithms for generalized linear mixed models. Journal of the American Statistical Association, 92:162–170, 1997.

    Article  MathSciNet  Google Scholar 

  35. C. McCulloch and S. Searle. Generalized, Linear and Mixed Models. Wiley, New-York, 2001.

    MATH  Google Scholar 

  36. G. McLachlan and D. Peel. Finite Mixture Models. Wiley, New York, 2000.

    Book  MATH  Google Scholar 

  37. X.-L. Meng. On the rate of convergence of the ECM algorithm. The Annals of Statistics, 22:326–339, 1994.

    Article  MathSciNet  Google Scholar 

  38. X.-L. Meng and D. Rubin. Maximum likelihood estimation via the ECM algorithm: A general framework. Biometrika, 80:267–278, 1993.

    Article  MathSciNet  Google Scholar 

  39. R. Neal and G. Hinton. A view of EM that justifies incremental, sparse and other variants. In M Jordan, editor, Learning in Graphical Models, pages 355–371, 1998.

    Google Scholar 

  40. S.-K. Ng and G. McLachlan. On some variants of the EM Algorithm for fitting finite mixture models. Australian Journal of Statistics, 32:143–161, 2003.

    Google Scholar 

  41. S.-K. Ng and G. McLachlan. On the choice of the number of blocks with the incremental EM algorithm for the fitting of normal mixtures. Statistics and Computing, 13:45–55, 2003.

    Article  MathSciNet  Google Scholar 

  42. K. Nigam, A. Mccallum, S. Thrun and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39:103–134, 2000.

    Article  Google Scholar 

  43. A. Owen and S. Tribble. A Quasi-Monte Carlo Metropolis algorithm. Proceedings of the National Academy of Sciences, 102:8844–8849, 2005.

    Article  CAS  MathSciNet  ADS  Google Scholar 

  44. B. Polyak and A. Juditsky. Acceleration of stochastic approximation by aver aging. SIAM Journal of Control and Optimization, 30:838–855, 1992.

    Article  MathSciNet  Google Scholar 

  45. F. Quintana, J. Liu and G. delPino. Monte Carlo EM with importance reweighting and its applications in random effects models. Computational Statistics and Data Analysis, 29:429–444, 1999.

    Article  Google Scholar 

  46. J. Ramsay and B. Silverman. Functional Data Analysis. Springer-Verlag, New York, 1997.

    MATH  Google Scholar 

  47. H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22:400–407, 1951.

    Article  MathSciNet  Google Scholar 

  48. D. Rubin. EM and beyond. Psychometrika, 56:241–254, 1991.

    Article  MathSciNet  Google Scholar 

  49. B. Thiesson, C. Meek and D. Heckerman. Accelerating EM for large databases. Machine Learning, 45:279–299, 2001.

    Article  Google Scholar 

  50. Y. Tu, M. Ball and W. Jank. Estimating flight departure delay distributions: A statistical approach with long-term trend and short-term pattern. Technical report, University of Maryland, 2005.

    Google Scholar 

  51. X. Wang and F. Hickernell. Randomized Halton sequences. Mathematical and Computer Modelling, 32:887–899, 2000.

    Article  MathSciNet  Google Scholar 

  52. G. Wei and M. Tanner. A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association, 85:699–704, 1990.

    Article  Google Scholar 

  53. C. Wu. On the convergence properties of the EM algorithm. The Annals of Statistics, 11:95–103, 1983.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Robert H. Smith School of Business, Department of Decision and Information Technologies, University of Maryland, College Park, MD, 20742

    Wolfgang Jank

Authors
  1. Wolfgang Jank
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Maryland, USA

    Francis B. Alt , Michael C. Fu  & Bruce L. Golden ,  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Jank, W. (2006). The EM Algorithm, Its Randomized Implementation and Global Optimization: Some Challenges and Opportunities for Operations Research. In: Alt, F.B., Fu, M.C., Golden, B.L. (eds) Perspectives in Operations Research. Operations Research/Computer Science Interfaces Series, vol 36. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-39934-8_21

Download citation

Publish with us

Policies and ethics

Search

Navigation