The EM Algorithm

Abstract

In the previous chapters, we examined various methods that are applied directly to the likelihood or to the posterior density. In this and the following chapters, we examine the data augmentation algorithms, including the EM algorithm and the data augmentation algorithm. All of these data augmentation algorithms share a common approach to problems: rather than performing a complicated maximization or simulation, one augments the observed data with “stuff” (latent data) that simplifies the calculation and subsequently performs a series of simple maximizations or simulations. This “stuff” can be the “missing” data or parameter values. The principle of data augmentation can then be stated as follows: Augment the observed data Y with latent data Z so that the augmented posterior distribution p(θ | Y, Z) is “simple.” Make use of this simplicity in maximizing/marginalizing/calculating/sampling the observed posterior p (θ | Y).