EM Algorithms

  • Charles Byrne
  • Paul P. B. Eggermont

Maximum Likelihood Estimation

Expectation-Maximization algorithms, or em algorithms for short, are iterative algorithms designed to solve maximum likelihood estimation problems. The general setting is that one observes a random sample \({Y }_{1},{Y }_{2},\ldots,{Y }_{n}\)


Conditional Expectation Probability Vector Monotonicity Property Leibler Divergence Algebraic Reconstruction Technique 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References and Further Reading

  1. 1.
    Aronszajn N, Smith KT (1961) Theory of Bessel potentials. I. Ann Inst Fourier (Grenoble) 11:385–475, www.numdam.orgGoogle Scholar
  2. 2.
    Atkinson KE (1969) The numerical solution of integral equations on the half line. SIAM J Numer Anal 6:375–397MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Bardsley JM, Luttman A (2009) Total variation-penalized Poisson likelihood estimation for ill-posed problems. Adv Comput Math 31:35–39MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Bertero M, Bocacci P, Desiderá G, Vicidomini G (2009) Image de-blurring with Poisson data: from cells to galaxies. Inverse Probl 25(123006):26Google Scholar
  5. 5.
    Browne J, De Pierro AR (1996) A row-action alternative to the EM algorithm for maximizing likelihoods in emission tomography. IEEE Trans Med Imag 15:687–699CrossRefGoogle Scholar
  6. 6.
    Brune C, Sawatzky A, Burger M (2009) Bregman-EM-TV methods with application to optical nanoscopy, scale space and variational methods in computer vision, Lecture Notes in Computer Science 5567. Springer, Berlin, pp 235–246Google Scholar
  7. 7.
    Byrne CL (1993) Iterative image reconstruction algorithms based on cross-entropy minimization. IEEE Trans Image Process 2:96–103CrossRefGoogle Scholar
  8. 8.
    Byrne CL (1996) Block-iterative methods for image reconstruction from projections. IEEE Trans Image Process 5:792–794CrossRefGoogle Scholar
  9. 9.
    Byrne CL (1998) Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative methods. IEEE Trans Image Process 7:792–794MathSciNetCrossRefGoogle Scholar
  10. 10.
    Byrne CL (2001) Likelihood maximization for list-mode emission tomographic image reconstruction. IEEE Trans Med Imag 20:1084–1092CrossRefGoogle Scholar
  11. 11.
    Byrne CL (2005) Choosing parameters in block-iterative or ordered subset reconstruction algorithms. IEEE Trans Image Process 14:321–327MathSciNetCrossRefGoogle Scholar
  12. 12.
    Byrne CL (2005) Signal processing: a mathematical approach. AK Peters, WellesleyMATHGoogle Scholar
  13. 13.
    Byrne CL (2008) Applied iterative methods. AK Peters, WellesleyMATHGoogle Scholar
  14. 14.
    Byrne CL, Fiddy MA (1988) Images as power spectra; reconstruction as a Wiener filter approximation. Inverse Probl 4:399–409MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Cao Yu, Eggermont PPB, Terebey S (1999) Cross Burg entropy maximization and its application to ringing suppression in image reconstruction. IEEE Trans Image Process 8:286–292CrossRefGoogle Scholar
  16. 16.
    Censor Y, Eggermont PPB, Gordon D (1983) Strong under relaxation in Kaczmarz’s method for inconsistent systems. Numer Math 41:83–92MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Censor Y, Lent AH (1987) Optimization of “log x” entropy over linear equality constraints. SIAM J Control Optim 25:921–933MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    Censor Y, Segman J (1987) On block-iterative entropy maximization. J Inform Optim Sci 8: 275–291MathSciNetMATHGoogle Scholar
  19. 19.
    Censor Y, Zenios SA (1992) Proximal minimization algorithm with D-functions. J Optim Theory Appl 73:451–464MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Cover TM (1984) An algorithm for maximizing expected log investment return. IEEE Trans Inform Theory 30:369–373MathSciNetMATHCrossRefGoogle Scholar
  21. 21.
    Crowther RA, DeRosier DJ, Klug A (1971) The reconstruction of three-dimensional structure from projections and its application to electron microscopy. Proc R Soc Lond A Math Phys Sci 317(3):19–340Google Scholar
  22. 22.
    Csiszár I (1975) I-divergence geometry of probability distributions and minimization problems. Ann Probab 3:146–158MATHCrossRefGoogle Scholar
  23. 23.
    Csiszár I, Tusnády G (1984) Information geometry and alternating minimization procedures. Stat Decisions 1(Supplement 1):205–237Google Scholar
  24. 24.
    Daley DJ, Vere-Jones D (2003) An introduction to the theory of point processes. Springer, New YorkMATHGoogle Scholar
  25. 25.
    Darroch JN, Ratcliff D (1972) Generalized iterative scaling for log-linear models. Ann Math Stat 43:1470–1480MathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    Daube-Witherspoon ME, Muehllehner G (1986) An iterative space reconstruction algorithm suitable for volume ECT. IEEE Trans Med Imag 5: 61–66CrossRefGoogle Scholar
  27. 27.
    Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 37:1–38MathSciNetGoogle Scholar
  28. 28.
    De Pierro AR (1987) On the convergence of the iterative image space reconstruction algorithm for volume ECT. IEEE Trans Med Imag 6: 174–175CrossRefGoogle Scholar
  29. 29.
    De Pierro AR (1995) A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography. IEEE Trans Med Imag 14:132–137CrossRefGoogle Scholar
  30. 30.
    De Pierro A, Yamaguchi M (2001) Fast EM-like methods for maximum a posteriori estimates in emission tomography. Trans Med Imag 20: 280–288CrossRefGoogle Scholar
  31. 31.
    Dey N, Blanc-Ferraud L, Zimmer Ch, Roux P, Kam Z, Olivo-Martin J-Ch, Zerubia J (2006) Richardson-Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution. Microsc Res Tech 69:260–266CrossRefGoogle Scholar
  32. 32.
    Duijster A, Scheunders P, De Backer S (2009) Wavelet-based EM algorithm for multispectral-image restoration. IEEE Trans Geoscience Remote Sensing 47:3892–3898CrossRefGoogle Scholar
  33. 33.
    Eggermont PPB (1990) Multiplicative iterative algorithms for convex programming. Linear Algebra Appl 130:25–42MathSciNetMATHCrossRefGoogle Scholar
  34. 34.
    Eggermont PPB (1999) Nonlinear smoothing and the EM algorithm for positive integral equations of the first kind. Appl Math Optimiz 39: 75–91MathSciNetMATHCrossRefGoogle Scholar
  35. 35.
    Eggermont PPB, Herman GT, Lent AH (1981) Iterative algorithms for large partitioned linear systems with applications to image reconstruction. Linear Algebra Appl 40:37–67MathSciNetMATHCrossRefGoogle Scholar
  36. 36.
    Eggermont PPB, LaRiccia VN (1995) Smoothed maximum likelihood density estimation for inverse problems. Ann Stat 23:199–220MathSciNetMATHCrossRefGoogle Scholar
  37. 37.
    Eggermont PPB, LaRiccia VN (1997) Maximum penalized likelihood estimation and smoothed EM algorithms for positive integral equations of the first kind. Numer Funct Anal Optimiz 17:737–754MathSciNetCrossRefGoogle Scholar
  38. 38.
    Eggermont PPB, LaRiccia VN (1998) On EM-like algorithms for minimum distance estimation. Manuscript, University of DelawareGoogle Scholar
  39. 39.
    Eggermont PPB, LaRiccia VN (2001) Maximum penalized likelihood estimation, I: Density estimation. Springer, New YorkGoogle Scholar
  40. 40.
    Elfving T (1980) On some methods for entropy maximization and matrix scaling. Linear Algebra Appl 34:321–339MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    Fessler JA, Ficaro EP, Clinthorne NH, Lange K (1997) Grouped coordinate ascent algorithms for penalized log-likelihood transmission image reconstruction. IEEE Trans Med Imag 16:166–175CrossRefGoogle Scholar
  42. 42.
    Fessler JA, Hero AO (1995) Penalized maximum-likelihood image reconstruction using space-alternating generalized EM algorithms. IEEE Trans Image Process 4:1417–1429CrossRefGoogle Scholar
  43. 43.
    Figueiredo MAT, Nowak RD (2003) An EM algorithm for wavelet-based image restoration. IEEE Trans Image Process 12:906–916MathSciNetCrossRefGoogle Scholar
  44. 44.
    Frank J (2006) Three-dimensional electron microscopy of macromolecular assemblies, 2nd edn. Oxford University Press, New YorkCrossRefGoogle Scholar
  45. 45.
    Geman S, Geman D (1984) Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6:721–741MATHCrossRefGoogle Scholar
  46. 46.
    Geman S, McClure DE (1985) Bayesian image analysis, an application to single photon emission tomography, Statistical Computing Section. Proc Am Stat Assoc 12–18Google Scholar
  47. 47.
    Good IJ (1971) A nonparametric roughness penalty for probability densities. Nature 229: 29–30Google Scholar
  48. 48.
    Gordon R, Bender R, Herman GT (1970) Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J Theor Biol 29:471–482CrossRefGoogle Scholar
  49. 49.
    Green PJ (1990) Bayesian reconstructions from emission tomography data using a modified EM algorithm. IEEE Trans Med Imag 9:84–93CrossRefGoogle Scholar
  50. 50.
    Guillaume M, Melon P, Réfrégier P (1998) Maximum-likelihood estimation of an astronomical image from a sequence at low photon levels. J Opt Soc Am A 15:2841–2848CrossRefGoogle Scholar
  51. 51.
    Haltmeier M, Leitão A, Resmerita E (2009) On regularization methods of EM-Kaczmarz type. Inverse Probl 25(075008):17Google Scholar
  52. 52.
    Hanke M (1991) Accelerated Landweber iterations for the solution of ill-posed problems. Numer Math 60:341–373MathSciNetMATHCrossRefGoogle Scholar
  53. 53.
    Hartley HO (1958) Maximum likelihood estimation from incomplete data. Biometrics 14: 174–194MATHCrossRefGoogle Scholar
  54. 54.
    Hebert T, Leahy R (1989) A generalized EM algorithm for 3-D Bayesian reconstruction from Poisson data using Gibbs priors. IEEE Trans Med Imag 8:194–202CrossRefGoogle Scholar
  55. 55.
    Herman GT (2009) Fundamentals of computerized tomography: image reconstruction from projections. Springer, New YorkGoogle Scholar
  56. 56.
    Herman GT, Meyer LB (1993) Algebraic reconstruction techniques can be made computationally efficient. IEEE Trans Med Imag 12: 600–609CrossRefGoogle Scholar
  57. 57.
    Holte S, Schmidlin P, Lindén A, Rosenqvist G, Eriksson L (1990) Iterative image reconstruction for positron emission tomography: a study of convergence and quantitation problems. IEEE Trans Nuclear Sci 37:629–635CrossRefGoogle Scholar
  58. 58.
    Horváth I, Bagoly Z, Balász LG, de Ugarte Postigo A, Veres P, Mészáros A (2010) Detailed classification of Swift’s Gamma-ray bursts. J Astrophys 713:552–557CrossRefGoogle Scholar
  59. 59.
    Hudson HM, Larkin RS (1994) Accelerated image reconstruction using ordered subsets of projection data. IEEE Trans Med Imag 13:601–609CrossRefGoogle Scholar
  60. 60.
    Kamphuis C, Beekman FJ, Viergever MA (1996) Evaluation of OS-EM vs. EM-ML for 1D, 2D and fully 3D SPECT reconstruction. IEEE Trans Nucl Sci 43:2018–2024CrossRefGoogle Scholar
  61. 61.
    Kondor A (1983) Method of convergent weights – an iterative procedure for solving Fredholm’s integral equations of the first kind. Nucl Instrum Methods 216:177–181CrossRefGoogle Scholar
  62. 62.
    Lange K (1990) Convergence of EM image reconstruction algorithms with Gibbs smoothing. IEEE Trans Med Imag 9:439–446CrossRefGoogle Scholar
  63. 63.
    Lange K, Bahn M, Little R (1987) A theoretical study of some maximum likelihood algorithms for emission and transmission tomography. IEEE Trans Med Imag 6:106–114CrossRefGoogle Scholar
  64. 64.
    Lange K, Carson R (1984) EM reconstruction algorithms for emission and transmission tomography. J Comput Assisted Tomography 8:306–316Google Scholar
  65. 65.
    Latham GA (1995) Existence of EMS solutions and a priori estimates. SIAM J Matrix Anal Appl 16:943–953MathSciNetMATHCrossRefGoogle Scholar
  66. 66.
    Levitan E, Chan M, Herman GT (1995) Image-modeling Gibbs priors. Graph Models Image Process 57:117–130CrossRefGoogle Scholar
  67. 67.
    Lewitt RM, Muehllehner G (1986) Accelerated iterative reconstruction in PET and TOFPET. IEEE Trans Med Imag 5:16–22CrossRefGoogle Scholar
  68. 68.
    Liu C, Rubin H (1994) The ECME algorithm: a simple extension of EM and ECM with faster monotone convergence. Biometrika 81:633–648MathSciNetMATHCrossRefGoogle Scholar
  69. 69.
    Llacer J, Veklerov E (1989) Feasible images and practical stopping rules for iterative algorithms in emission tomography. IEEE Trans Med Imag 8:186–193CrossRefGoogle Scholar
  70. 70.
    Lucy LB (1974) An iterative technique for the rectification of observed distributions. Astronomical J 79:745–754CrossRefGoogle Scholar
  71. 71.
    McLachlan GJ, Krishnan T (2008) The EM algorithm and its extensions. Wiley, HobokenCrossRefGoogle Scholar
  72. 72.
    Meidunas E (2001) Re-scaled block iterative expectation maximization maximum likelihood (RBI-EMML) abundance estimation and sub-pixel material identification in hyperspectral imagery. MS thesis, Department of Electrical Engineering, University of Massachusetts LowellGoogle Scholar
  73. 73.
    Miller MI, Roysam B (1991) Bayesian image reconstruction for emission tomography incorporating Good’s roughness prior on massively parallel processors. Proc Natl Acad Sci USA 88:3223–3227CrossRefGoogle Scholar
  74. 74.
    Mülthei HN, Schorr B (1987) On an iterative method for a class of integral equations of the first kind. Math Meth Appl Sci 9:137–168MATHCrossRefGoogle Scholar
  75. 75.
    Mülthei HN, Schorr B (1989) On properties of the iterative maximum likelihood reconstruction method. Math Meth Appl Sci 11:331–342MATHCrossRefGoogle Scholar
  76. 76.
    Nielsen SF (2006) The stochastic EM algorithm: estimation and asymptotic results. Bernoulli 6:457–489CrossRefGoogle Scholar
  77. 77.
    Parra L, Barrett H (1998) List-mode likelihood: EM algorithm and image quality estimation demonstrated on 2-D PET. IEEE Trans Med Imag 17:228–235CrossRefGoogle Scholar
  78. 78.
    Penczek P, Zhu J, Schroeder R, Frank J (1997) Three-dimensional reconstruction with contrast transfer function compensation. Scanning Microscopy 11:147–154Google Scholar
  79. 79.
    Redner RA, Walker HF (1984) Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev 26:195–239MathSciNetMATHCrossRefGoogle Scholar
  80. 80.
    Resmerita E, Engl HW, Iusem AN (2007) The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis. Inverse Probl 23:2575–2588MathSciNetMATHCrossRefGoogle Scholar
  81. 81.
    Richardson WH (1972) Bayesian based iterative method of image restoration. J Opt Soc Am 62:55–59CrossRefGoogle Scholar
  82. 82.
    Rockmore A, Macovski A (1976) A maximum likelihood approach to emission image reconstruction from projections. IEEE Trans Nucl Sci 23:1428–1432CrossRefGoogle Scholar
  83. 83.
    Scheres SHW, Valle M, Núñez R, Sorzano COS, Marabini R, Herman GT, Carazo J-M (2005) Maximum-likelihood multi-reference refinement for electron microscopy images. J Mol Biol 348:139–149CrossRefGoogle Scholar
  84. 84.
    Scheres SHW, Gao HX, Valle M, Herman GT, Eggermont PPB, Frank J, Carazo J-M (2007a) Disentangling conformational states of macromolecules in 3D-EM through likelihood optimization. Nat Methods 4:27–29CrossRefGoogle Scholar
  85. 85.
    Scheres SHW, Núñez-Ramírez R, Gómez-Llorente Y, San Martín C, Eggermont PPB, Carazo J-M (2007b) Modeling experimental image formation for likelihood-based classification of electron microscopy. Structure 15:1167–1177CrossRefGoogle Scholar
  86. 86.
    Schmidlin P (1972) Iterative separation of tomographic scintigrams. Nuklearmedizin 11:1–16Google Scholar
  87. 87.
    Setzer S, Steidl G, Teuber T (2010) Deblurring Poissonian images by split Bregman techniques. J Vis Commun Image Repr 21:193–199CrossRefGoogle Scholar
  88. 88.
    Shepp LA, Vardi Y (1982) Maximum likelihood reconstruction in emission tomography. IEEE Trans Med Imag 1:113–122CrossRefGoogle Scholar
  89. 89.
    Sigworth FJ (1998) A maximum-likelihood approach to single-particle image refinement. J Struct Biol 122:328–339CrossRefGoogle Scholar
  90. 90.
    Silverman BW, Jones MC, Wilson JD, Nychka DW (1990) A smoothed EM algorithm approach to indirect estimation problems, with particular reference to stereology and emission tomography (with discussion). J R Stat Soc B 52:271–324MathSciNetMATHGoogle Scholar
  91. 91.
    Sun Y, Walker JG (2008) Maximum likelihood data inversion for photon correlation spectroscopy. Meas Sci Technol 19(115302):8Google Scholar
  92. 92.
    Tanaka E, Kudo H (2010) Optimal relaxation parameters of DRAMA (dynamic RAMLA) aiming at one-pass image reconstruction for 3D-PET. Phys Med Biol 55:2917–2939CrossRefGoogle Scholar
  93. 93.
    Tarasko MZ (1969) On a method for solution of the linear system with stochastic matrices (in Russian), Report Physics and Energetics Institute, Obninsk PEI-156Google Scholar
  94. 94.
    Trummer MR (1984) A note on the ART of relaxation. Computing 33:349–352MathSciNetMATHCrossRefGoogle Scholar
  95. 95.
    van der Sluis A, van der Vorst HA (1990) SIRT- and CG-type methods for the iterative solution of sparse linear least-squares problems. Linear algebra in image reconstruction from projections. Linear Algebra Appl 130: 257–303MathSciNetMATHCrossRefGoogle Scholar
  96. 96.
    Vardi Y, Shepp LA, Kaufman L (1985) A statistical model for positron emission tomography (with discussion). J Am Stat Assoc 80:8–38MathSciNetMATHCrossRefGoogle Scholar
  97. 97.
    Wernick M, Aarsvold J (2004) Emission tomography: the fundamentals of PET and SPECT. Elsevier Academic Press, San DiegoGoogle Scholar
  98. 98.
    Wu CFJ (1983) On the convergence properties of the EM algorithm. Ann Stat 11:95–103MATHCrossRefGoogle Scholar
  99. 99.
    Yu S, Latham GA, Anderssen RS (1994) Stabilizing properties of maximum penalized likelihood estimation for additive Poisson regression. Inverse Probl 10:1199–1209MathSciNetMATHCrossRefGoogle Scholar
  100. 100.
    Yuan Jianhua, Yu Jun (2007) Median-prior tomography reconstruction combined with nonlinear anisotropic diffusion filtering. J Opt Soc Am A 24: 1026–1033Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Charles Byrne
  • Paul P. B. Eggermont

There are no affiliations available

Personalised recommendations