Optimization pp 221-244 | Cite as

# The EM Algorithm

## Abstract

Maximum likelihood is the dominant form of estimation in applied statistics. Because closed-form solutions to likelihood equations are the exception rather than the rule, numerical methods for finding maximum likelihood estimates are of paramount importance. In this chapter we study maximum likelihood estimation by the EM algorithm a special case of the MM algorithm. At the heart of every EM algorithm is some notion of missing data. Data can be missing in the ordinary sense of a failure to record certain observations on certain cases. Data can also be missing in a theoretical sense. We can think of the E (expectation) step of the algorithm as filling in the missing data. This action replaces the loglikelihood of the observed data by a minorizing function. This surrogate function is then maximized in the M step. Because the surrogate function is usually much simpler than the likelihood, we can often solve the M step analytically. The price we pay for this simplification is that the EM algorithm is iterative. Reconstructing the missing data is bound to be slightly wrong if the parameters do not already equal their maximum likelihood estimates.

## Keywords

Random Vector Success Probability Conditional Expectation Conditional Density Multivariate Normal Distribution## References

- 8.Baum LE (1972) An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes. Inequalities 3:1–8Google Scholar
- 65.Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm (with discussion). J Roy Stat Soc B 39:1–38MathSciNetMATHGoogle Scholar
- 71.Devijver PA (1985) Baum’s forward-backward algorithm revisited. Pattern Recogn Lett 3:369–373MATHCrossRefGoogle Scholar
- 73.Dobson AJ (1990) An introduction to generalized linear models. Chapman & Hall, LondonMATHGoogle Scholar
- 76.Duan J-C, Simonato J-G (1993) Multiplicity of solutions in maximum likelihood factor analysis. J Stat Comput Simul 47:37–47CrossRefGoogle Scholar
- 78.Durbin R, Eddy S, Krogh A, Mitchison G (1998) Biological sequence analysis: probabilistic models of proteins and nucleic acids. Cambridge University Press, CambridgeMATHCrossRefGoogle Scholar
- 95.Flury B, Zoppè A (2000) Exercises in EM. Am Stat 54:207–209Google Scholar
- 112.Green PJ (1990) On use of the EM algorithm for penalized likelihood estimation. J Roy Stat Soc B 52:443–452MATHGoogle Scholar
- 165.Lange K (2002) Mathematical and statistical methods for genetic analysis, 2nd edn. Springer, New YorkMATHCrossRefGoogle Scholar
- 166.Lange K (2010) Numerical analysis for statisticians, 2nd edn. Springer, New YorkMATHCrossRefGoogle Scholar
- 179.Little RJA, Rubin DB (1987) Statistical analysis with missing data. Wiley, HobokenMATHGoogle Scholar
- 191.McLachlan GJ, Krishnan T (2008) The EM algorithm and extensions, 2nd edn. Wiley, HobokenMATHCrossRefGoogle Scholar
- 192.McLachlan GJ, Peel D (2000) Finite mixture models. Wiley, HobokenMATHCrossRefGoogle Scholar
- 216.Rabiner L (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77:257–285CrossRefGoogle Scholar
- 218.Rao CR (1973) Linear statistical inference and its applications, 2nd edn. Wiley, HobokenMATHCrossRefGoogle Scholar
- 251.Tanner MA (1993) Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions, 2nd edn. Springer, New YorkMATHGoogle Scholar
- 259.Titterington DM, Smith AFM, Makov UE (1985) Statistical analysis of finite mixture distributions. Wiley, HobokenMATHGoogle Scholar
- 270.Weeks DE, Lange K (1989) Trials, tribulations, and triumphs of the EM algorithm in pedigree analysis. IMA J Math Appl Med Biol 6:209–232MathSciNetMATHCrossRefGoogle Scholar
- 281.Zhou H, Lange K (2009) On the bumpy road to the dominant mode. Scand J Stat 37:612–631MathSciNetCrossRefGoogle Scholar