Skip to main content
SpringerLink
Log in

EM Algorithms

  • Reference work entry
Handbook of Mathematical Methods in Imaging

Maximum Likelihood Estimation

Expectation-Maximization algorithms, or em algorithms for short, are iterative algorithms designed to solve maximum likelihood estimation problems. The general setting is that one observes a random sample \({Y }_{1},{Y }_{2},\ldots,{Y }_{n}\) of a random variable Y whose probability density function (pdf) \(f(\,\cdot \,\vert \,{x}_{o})\) with respect to some (known) dominating measure is known up to an unknown “parameter” \({x}_{o}\). The goal is to estimate \({x}_{o}\) and, one might add, to do it well. In this chapter that means to solve the maximum likelihood problem

$$\mathrm{maximize}\ \quad \prod \limits_{i=1}^{n}\,f({Y }_{ i}^{}\,\vert \,x)\quad \mathrm{over}\quad x,$$
(8.1)

and to solve it by means of em algorithms. The solution, assuming it exists and is unique, is called the maximum likelihood estimator of \({x}_{o}\). Here, the estimator is typically denoted by \(\widehat{x}\).

The notion of em algorithms was coined by [27], who unified various...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 679.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References and Further Reading

  1. Aronszajn N, Smith KT (1961) Theory of Bessel potentials. I. Ann Inst Fourier (Grenoble) 11:385–475, www.numdam.org

    Google Scholar 

  2. Atkinson KE (1969) The numerical solution of integral equations on the half line. SIAM J Numer Anal 6:375–397

    Article  MathSciNet  MATH  Google Scholar 

  3. Bardsley JM, Luttman A (2009) Total variation-penalized Poisson likelihood estimation for ill-posed problems. Adv Comput Math 31:35–39

    Article  MathSciNet  MATH  Google Scholar 

  4. Bertero M, Bocacci P, Desiderá G, Vicidomini G (2009) Image de-blurring with Poisson data: from cells to galaxies. Inverse Probl 25(123006):26

    Google Scholar 

  5. Browne J, De Pierro AR (1996) A row-action alternative to the EM algorithm for maximizing likelihoods in emission tomography. IEEE Trans Med Imag 15:687–699

    Article  Google Scholar 

  6. Brune C, Sawatzky A, Burger M (2009) Bregman-EM-TV methods with application to optical nanoscopy, scale space and variational methods in computer vision, Lecture Notes in Computer Science 5567. Springer, Berlin, pp 235–246

    Google Scholar 

  7. Byrne CL (1993) Iterative image reconstruction algorithms based on cross-entropy minimization. IEEE Trans Image Process 2:96–103

    Article  Google Scholar 

  8. Byrne CL (1996) Block-iterative methods for image reconstruction from projections. IEEE Trans Image Process 5:792–794

    Article  Google Scholar 

  9. Byrne CL (1998) Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative methods. IEEE Trans Image Process 7:792–794

    Article  MathSciNet  Google Scholar 

  10. Byrne CL (2001) Likelihood maximization for list-mode emission tomographic image reconstruction. IEEE Trans Med Imag 20:1084–1092

    Article  Google Scholar 

  11. Byrne CL (2005) Choosing parameters in block-iterative or ordered subset reconstruction algorithms. IEEE Trans Image Process 14:321–327

    Article  MathSciNet  Google Scholar 

  12. Byrne CL (2005) Signal processing: a mathematical approach. AK Peters, Wellesley

    MATH  Google Scholar 

  13. Byrne CL (2008) Applied iterative methods. AK Peters, Wellesley

    MATH  Google Scholar 

  14. Byrne CL, Fiddy MA (1988) Images as power spectra; reconstruction as a Wiener filter approximation. Inverse Probl 4:399–409

    Article  MathSciNet  MATH  Google Scholar 

  15. Cao Yu, Eggermont PPB, Terebey S (1999) Cross Burg entropy maximization and its application to ringing suppression in image reconstruction. IEEE Trans Image Process 8:286–292

    Article  Google Scholar 

  16. Censor Y, Eggermont PPB, Gordon D (1983) Strong under relaxation in Kaczmarz’s method for inconsistent systems. Numer Math 41:83–92

    Article  MathSciNet  MATH  Google Scholar 

  17. Censor Y, Lent AH (1987) Optimization of “log x” entropy over linear equality constraints. SIAM J Control Optim 25:921–933

    Article  MathSciNet  MATH  Google Scholar 

  18. Censor Y, Segman J (1987) On block-iterative entropy maximization. J Inform Optim Sci 8: 275–291

    MathSciNet  MATH  Google Scholar 

  19. Censor Y, Zenios SA (1992) Proximal minimization algorithm with D-functions. J Optim Theory Appl 73:451–464

    Article  MathSciNet  MATH  Google Scholar 

  20. Cover TM (1984) An algorithm for maximizing expected log investment return. IEEE Trans Inform Theory 30:369–373

    Article  MathSciNet  MATH  Google Scholar 

  21. Crowther RA, DeRosier DJ, Klug A (1971) The reconstruction of three-dimensional structure from projections and its application to electron microscopy. Proc R Soc Lond A Math Phys Sci 317(3):19–340

    Google Scholar 

  22. Csiszár I (1975) I-divergence geometry of probability distributions and minimization problems. Ann Probab 3:146–158

    Article  MATH  Google Scholar 

  23. Csiszár I, Tusnády G (1984) Information geometry and alternating minimization procedures. Stat Decisions 1(Supplement 1):205–237

    Google Scholar 

  24. Daley DJ, Vere-Jones D (2003) An introduction to the theory of point processes. Springer, New York

    MATH  Google Scholar 

  25. Darroch JN, Ratcliff D (1972) Generalized iterative scaling for log-linear models. Ann Math Stat 43:1470–1480

    Article  MathSciNet  MATH  Google Scholar 

  26. Daube-Witherspoon ME, Muehllehner G (1986) An iterative space reconstruction algorithm suitable for volume ECT. IEEE Trans Med Imag 5: 61–66

    Article  Google Scholar 

  27. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 37:1–38

    MathSciNet  Google Scholar 

  28. De Pierro AR (1987) On the convergence of the iterative image space reconstruction algorithm for volume ECT. IEEE Trans Med Imag 6: 174–175

    Article  Google Scholar 

  29. De Pierro AR (1995) A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography. IEEE Trans Med Imag 14:132–137

    Article  Google Scholar 

  30. De Pierro A, Yamaguchi M (2001) Fast EM-like methods for maximum a posteriori estimates in emission tomography. Trans Med Imag 20: 280–288

    Article  Google Scholar 

  31. Dey N, Blanc-Ferraud L, Zimmer Ch, Roux P, Kam Z, Olivo-Martin J-Ch, Zerubia J (2006) Richardson-Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution. Microsc Res Tech 69:260–266

    Article  Google Scholar 

  32. Duijster A, Scheunders P, De Backer S (2009) Wavelet-based EM algorithm for multispectral-image restoration. IEEE Trans Geoscience Remote Sensing 47:3892–3898

    Article  Google Scholar 

  33. Eggermont PPB (1990) Multiplicative iterative algorithms for convex programming. Linear Algebra Appl 130:25–42

    Article  MathSciNet  MATH  Google Scholar 

  34. Eggermont PPB (1999) Nonlinear smoothing and the EM algorithm for positive integral equations of the first kind. Appl Math Optimiz 39: 75–91

    Article  MathSciNet  MATH  Google Scholar 

  35. Eggermont PPB, Herman GT, Lent AH (1981) Iterative algorithms for large partitioned linear systems with applications to image reconstruction. Linear Algebra Appl 40:37–67

    Article  MathSciNet  MATH  Google Scholar 

  36. Eggermont PPB, LaRiccia VN (1995) Smoothed maximum likelihood density estimation for inverse problems. Ann Stat 23:199–220

    Article  MathSciNet  MATH  Google Scholar 

  37. Eggermont PPB, LaRiccia VN (1997) Maximum penalized likelihood estimation and smoothed EM algorithms for positive integral equations of the first kind. Numer Funct Anal Optimiz 17:737–754

    Article  MathSciNet  Google Scholar 

  38. Eggermont PPB, LaRiccia VN (1998) On EM-like algorithms for minimum distance estimation. Manuscript, University of Delaware

    Google Scholar 

  39. Eggermont PPB, LaRiccia VN (2001) Maximum penalized likelihood estimation, I: Density estimation. Springer, New York

    Google Scholar 

  40. Elfving T (1980) On some methods for entropy maximization and matrix scaling. Linear Algebra Appl 34:321–339

    Article  MathSciNet  MATH  Google Scholar 

  41. Fessler JA, Ficaro EP, Clinthorne NH, Lange K (1997) Grouped coordinate ascent algorithms for penalized log-likelihood transmission image reconstruction. IEEE Trans Med Imag 16:166–175

    Article  Google Scholar 

  42. Fessler JA, Hero AO (1995) Penalized maximum-likelihood image reconstruction using space-alternating generalized EM algorithms. IEEE Trans Image Process 4:1417–1429

    Article  Google Scholar 

  43. Figueiredo MAT, Nowak RD (2003) An EM algorithm for wavelet-based image restoration. IEEE Trans Image Process 12:906–916

    Article  MathSciNet  Google Scholar 

  44. Frank J (2006) Three-dimensional electron microscopy of macromolecular assemblies, 2nd edn. Oxford University Press, New York

    Book  Google Scholar 

  45. Geman S, Geman D (1984) Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6:721–741

    Article  MATH  Google Scholar 

  46. Geman S, McClure DE (1985) Bayesian image analysis, an application to single photon emission tomography, Statistical Computing Section. Proc Am Stat Assoc 12–18

    Google Scholar 

  47. Good IJ (1971) A nonparametric roughness penalty for probability densities. Nature 229: 29–30

    Google Scholar 

  48. Gordon R, Bender R, Herman GT (1970) Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J Theor Biol 29:471–482

    Article  Google Scholar 

  49. Green PJ (1990) Bayesian reconstructions from emission tomography data using a modified EM algorithm. IEEE Trans Med Imag 9:84–93

    Article  Google Scholar 

  50. Guillaume M, Melon P, Réfrégier P (1998) Maximum-likelihood estimation of an astronomical image from a sequence at low photon levels. J Opt Soc Am A 15:2841–2848

    Article  Google Scholar 

  51. Haltmeier M, Leitão A, Resmerita E (2009) On regularization methods of EM-Kaczmarz type. Inverse Probl 25(075008):17

    Google Scholar 

  52. Hanke M (1991) Accelerated Landweber iterations for the solution of ill-posed problems. Numer Math 60:341–373

    Article  MathSciNet  MATH  Google Scholar 

  53. Hartley HO (1958) Maximum likelihood estimation from incomplete data. Biometrics 14: 174–194

    Article  MATH  Google Scholar 

  54. Hebert T, Leahy R (1989) A generalized EM algorithm for 3-D Bayesian reconstruction from Poisson data using Gibbs priors. IEEE Trans Med Imag 8:194–202

    Article  Google Scholar 

  55. Herman GT (2009) Fundamentals of computerized tomography: image reconstruction from projections. Springer, New York

    Google Scholar 

  56. Herman GT, Meyer LB (1993) Algebraic reconstruction techniques can be made computationally efficient. IEEE Trans Med Imag 12: 600–609

    Article  Google Scholar 

  57. Holte S, Schmidlin P, Lindén A, Rosenqvist G, Eriksson L (1990) Iterative image reconstruction for positron emission tomography: a study of convergence and quantitation problems. IEEE Trans Nuclear Sci 37:629–635

    Article  Google Scholar 

  58. Horváth I, Bagoly Z, Balász LG, de Ugarte Postigo A, Veres P, Mészáros A (2010) Detailed classification of Swift’s Gamma-ray bursts. J Astrophys 713:552–557

    Article  Google Scholar 

  59. Hudson HM, Larkin RS (1994) Accelerated image reconstruction using ordered subsets of projection data. IEEE Trans Med Imag 13:601–609

    Article  Google Scholar 

  60. Kamphuis C, Beekman FJ, Viergever MA (1996) Evaluation of OS-EM vs. EM-ML for 1D, 2D and fully 3D SPECT reconstruction. IEEE Trans Nucl Sci 43:2018–2024

    Article  Google Scholar 

  61. Kondor A (1983) Method of convergent weights – an iterative procedure for solving Fredholm’s integral equations of the first kind. Nucl Instrum Methods 216:177–181

    Article  Google Scholar 

  62. Lange K (1990) Convergence of EM image reconstruction algorithms with Gibbs smoothing. IEEE Trans Med Imag 9:439–446

    Article  Google Scholar 

  63. Lange K, Bahn M, Little R (1987) A theoretical study of some maximum likelihood algorithms for emission and transmission tomography. IEEE Trans Med Imag 6:106–114

    Article  Google Scholar 

  64. Lange K, Carson R (1984) EM reconstruction algorithms for emission and transmission tomography. J Comput Assisted Tomography 8:306–316

    Google Scholar 

  65. Latham GA (1995) Existence of EMS solutions and a priori estimates. SIAM J Matrix Anal Appl 16:943–953

    Article  MathSciNet  MATH  Google Scholar 

  66. Levitan E, Chan M, Herman GT (1995) Image-modeling Gibbs priors. Graph Models Image Process 57:117–130

    Article  Google Scholar 

  67. Lewitt RM, Muehllehner G (1986) Accelerated iterative reconstruction in PET and TOFPET. IEEE Trans Med Imag 5:16–22

    Article  Google Scholar 

  68. Liu C, Rubin H (1994) The ECME algorithm: a simple extension of EM and ECM with faster monotone convergence. Biometrika 81:633–648

    Article  MathSciNet  MATH  Google Scholar 

  69. Llacer J, Veklerov E (1989) Feasible images and practical stopping rules for iterative algorithms in emission tomography. IEEE Trans Med Imag 8:186–193

    Article  Google Scholar 

  70. Lucy LB (1974) An iterative technique for the rectification of observed distributions. Astronomical J 79:745–754

    Article  Google Scholar 

  71. McLachlan GJ, Krishnan T (2008) The EM algorithm and its extensions. Wiley, Hoboken

    Book  Google Scholar 

  72. Meidunas E (2001) Re-scaled block iterative expectation maximization maximum likelihood (RBI-EMML) abundance estimation and sub-pixel material identification in hyperspectral imagery. MS thesis, Department of Electrical Engineering, University of Massachusetts Lowell

    Google Scholar 

  73. Miller MI, Roysam B (1991) Bayesian image reconstruction for emission tomography incorporating Good’s roughness prior on massively parallel processors. Proc Natl Acad Sci USA 88:3223–3227

    Article  Google Scholar 

  74. Mülthei HN, Schorr B (1987) On an iterative method for a class of integral equations of the first kind. Math Meth Appl Sci 9:137–168

    Article  MATH  Google Scholar 

  75. Mülthei HN, Schorr B (1989) On properties of the iterative maximum likelihood reconstruction method. Math Meth Appl Sci 11:331–342

    Article  MATH  Google Scholar 

  76. Nielsen SF (2006) The stochastic EM algorithm: estimation and asymptotic results. Bernoulli 6:457–489

    Article  Google Scholar 

  77. Parra L, Barrett H (1998) List-mode likelihood: EM algorithm and image quality estimation demonstrated on 2-D PET. IEEE Trans Med Imag 17:228–235

    Article  Google Scholar 

  78. Penczek P, Zhu J, Schroeder R, Frank J (1997) Three-dimensional reconstruction with contrast transfer function compensation. Scanning Microscopy 11:147–154

    Google Scholar 

  79. Redner RA, Walker HF (1984) Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev 26:195–239

    Article  MathSciNet  MATH  Google Scholar 

  80. Resmerita E, Engl HW, Iusem AN (2007) The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis. Inverse Probl 23:2575–2588

    Article  MathSciNet  MATH  Google Scholar 

  81. Richardson WH (1972) Bayesian based iterative method of image restoration. J Opt Soc Am 62:55–59

    Article  Google Scholar 

  82. Rockmore A, Macovski A (1976) A maximum likelihood approach to emission image reconstruction from projections. IEEE Trans Nucl Sci 23:1428–1432

    Article  Google Scholar 

  83. Scheres SHW, Valle M, Núñez R, Sorzano COS, Marabini R, Herman GT, Carazo J-M (2005) Maximum-likelihood multi-reference refinement for electron microscopy images. J Mol Biol 348:139–149

    Article  Google Scholar 

  84. Scheres SHW, Gao HX, Valle M, Herman GT, Eggermont PPB, Frank J, Carazo J-M (2007a) Disentangling conformational states of macromolecules in 3D-EM through likelihood optimization. Nat Methods 4:27–29

    Article  Google Scholar 

  85. Scheres SHW, Núñez-Ramírez R, Gómez-Llorente Y, San Martín C, Eggermont PPB, Carazo J-M (2007b) Modeling experimental image formation for likelihood-based classification of electron microscopy. Structure 15:1167–1177

    Article  Google Scholar 

  86. Schmidlin P (1972) Iterative separation of tomographic scintigrams. Nuklearmedizin 11:1–16

    Google Scholar 

  87. Setzer S, Steidl G, Teuber T (2010) Deblurring Poissonian images by split Bregman techniques. J Vis Commun Image Repr 21:193–199

    Article  Google Scholar 

  88. Shepp LA, Vardi Y (1982) Maximum likelihood reconstruction in emission tomography. IEEE Trans Med Imag 1:113–122

    Article  Google Scholar 

  89. Sigworth FJ (1998) A maximum-likelihood approach to single-particle image refinement. J Struct Biol 122:328–339

    Article  Google Scholar 

  90. Silverman BW, Jones MC, Wilson JD, Nychka DW (1990) A smoothed EM algorithm approach to indirect estimation problems, with particular reference to stereology and emission tomography (with discussion). J R Stat Soc B 52:271–324

    MathSciNet  MATH  Google Scholar 

  91. Sun Y, Walker JG (2008) Maximum likelihood data inversion for photon correlation spectroscopy. Meas Sci Technol 19(115302):8

    Google Scholar 

  92. Tanaka E, Kudo H (2010) Optimal relaxation parameters of DRAMA (dynamic RAMLA) aiming at one-pass image reconstruction for 3D-PET. Phys Med Biol 55:2917–2939

    Article  Google Scholar 

  93. Tarasko MZ (1969) On a method for solution of the linear system with stochastic matrices (in Russian), Report Physics and Energetics Institute, Obninsk PEI-156

    Google Scholar 

  94. Trummer MR (1984) A note on the ART of relaxation. Computing 33:349–352

    Article  MathSciNet  MATH  Google Scholar 

  95. van der Sluis A, van der Vorst HA (1990) SIRT- and CG-type methods for the iterative solution of sparse linear least-squares problems. Linear algebra in image reconstruction from projections. Linear Algebra Appl 130: 257–303

    Article  MathSciNet  MATH  Google Scholar 

  96. Vardi Y, Shepp LA, Kaufman L (1985) A statistical model for positron emission tomography (with discussion). J Am Stat Assoc 80:8–38

    Article  MathSciNet  MATH  Google Scholar 

  97. Wernick M, Aarsvold J (2004) Emission tomography: the fundamentals of PET and SPECT. Elsevier Academic Press, San Diego

    Google Scholar 

  98. Wu CFJ (1983) On the convergence properties of the EM algorithm. Ann Stat 11:95–103

    Article  MATH  Google Scholar 

  99. Yu S, Latham GA, Anderssen RS (1994) Stabilizing properties of maximum penalized likelihood estimation for additive Poisson regression. Inverse Probl 10:1199–1209

    Article  MathSciNet  MATH  Google Scholar 

  100. Yuan Jianhua, Yu Jun (2007) Median-prior tomography reconstruction combined with nonlinear anisotropic diffusion filtering. J Opt Soc Am A 24: 1026–1033

    Google Scholar 

Download references

Authors
  1. Charles Byrne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul P. B. Eggermont
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Computational Science Center, University of Vienna, Nordbergstrasse 15, Vienna, Austria

    Otmar Scherzer

  2. RICAM, Austrian Academy of Sciences, Linz, Austria

    Otmar Scherzer

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer Science+Business Media, LLC

About this entry

Cite this entry

Byrne, C., Eggermont, P.P.B. (2011). EM Algorithms. In: Scherzer, O. (eds) Handbook of Mathematical Methods in Imaging. Springer, New York, NY. https://doi.org/10.1007/978-0-387-92920-0_8

Download citation

Publish with us

Policies and ethics

Search

Navigation