Skip to main content
SpringerLink
Log in

A Double-Loop Algorithm to Minimize the Bethe Free Energy

  • Conference paper
  • First Online:
Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2001)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 2134))

Abstract

Recent work (Yedidia, Freeman, Weiss [22]) has shown that stable points of belief propagation (BP) algorithms [12] for graphs with loops correspond to extrema of the Bethe free energy [3]. These BP algorithms have been used to obtain good solutions to problems for which alternative algorithms fail to work [4], [5], [10] [11]. In this paper we introduce a discrete iterative algorithm which we prove is guaranteed to converge to a minimum of the Bethe free energy. We call this the double-loop algorithm because it contains an inner and an outer loop. The algorithm is developed by decomposing the free energy into a convex part and a concave part, see [25], and extends a class of mean field theory algorithms developed by [7],[8] and, in particular, [13]. Moreover, the double-loop algorithm is formally very similar to BP which may help understand when BP converges. In related work [24] we extend this work to the more general Kikuchi approximation [3] which includes the Bethe free energy as a special case. It is anticipated that these double-loop algorithms will be useful for solving optimization problems in computer vision and other applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. S. Amari. “Differential Geometry of curved exponential families — Curvature and information loss. Annals of Statistics, vol. 10, no. 2, pp 357–385. 1982.

    Article  MATH  MathSciNet  Google Scholar 

  2. M. Arbib (Ed.) The Handbook of Brain Theory and Neural Networks. A Bradford Book. The MIT Press. 1995.

    Google Scholar 

  3. C. Domb and M.S. Green (Eds). Phase Transitions and Critical Phenomena. Vol. 2. Academic Press. London. 1972.

    Google Scholar 

  4. W.T. Freeman and E.C. Pasztor. “Learning low level vision”. In Proc. International Conference of Computer Vision. ICCV’99. pp 1182–1189. 1999.

    Google Scholar 

  5. B. Frey. Graphical Models for Pattern Classification, Data Compression and Channel Coding. MIT Press. 1998.

    Google Scholar 

  6. S. Kirkpatrick, C. Gelatt (Jr.), and M. Vecchi. “Optimization by Simulated Annealing”. Science, 220:671–680. 1983.

    Article  MathSciNet  Google Scholar 

  7. J. Kosowsky and A.L. Yuille. “The Invisible Hand Algorithm: Solving the Assignment Problem with Statistical Physics”. Neural Networks., Vol. 7, No. 3, pp 477–490. 1994.

    Article  MATH  Google Scholar 

  8. J. Kosowsky. Flows Suspending Iterative Algorithms. PhD Thesis. Division of Applied Sciences. Harvard University. Cambridge, Massachusetts. 1995.

    Google Scholar 

  9. C.M. Marcus and R.M. Westervelt. “Dynamics of iterated-map neural networks”. Physical Review A, 40, pp 501–504. 1989.

    Article  MathSciNet  Google Scholar 

  10. R.J. McEliece, D.J.C. Mackay, and J.F. Cheng. “Turbo decoding as an instance of Pearl’s belief propagation algorithm”. IEEE Journal on Selected Areas in Communication. 16(2), pp 140–152. 1998.

    Article  Google Scholar 

  11. K.P. Murphy, Y. Weiss, and M.I. Jordan. “Loopy belief propagation for approximate inference: an empirical study”. In Proceedings of Uncertainty in AI. 1999.

    Google Scholar 

  12. J. Pearl. Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann. 1988.

    Google Scholar 

  13. A. Rangarajan, S. Gold, and E. Mjolsness. “A Novel Optimizing Network Architecture with Applications”. Neural Computation, 8(5), pp 1041–1060. 1996.

    Article  Google Scholar 

  14. A. Rangarajan, A.L. Yuille, S. Gold. and E. Mjolsness.” A Convergence Proof for the Softassign Quadratic assignment Problem”. In Proceedings of NIPS’96. Snowmass. Colorado. 1996.

    Google Scholar 

  15. B.D. Ripley. “Pattern Recognition and Neural Networks”. Cambridge University Press. 1996.

    Google Scholar 

  16. L. K. Saul, T. Jaakkola, and M. I. Jordan. “Mean Field Theory for Sigmoid Belief Networks”. Journal of Artificial Intelligence Research, 4, 61–76, 1996.

    Google Scholar 

  17. R. Sinkhorn. “A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices”. Ann. Math. Statist.. 35, pp 876–879. 1964.

    Article  MathSciNet  MATH  Google Scholar 

  18. G. Strang. Introduction to Applied Mathematics. Wellesley-Cambridge Press. Wellesley, Massachusetts. 1986.

    MATH  Google Scholar 

  19. F.R. Waugh and R.M. Westervelt. “Analog neural networks with local competition: I. Dynamics and stability”. Physical Review E, 47(6), pp 4524–4536. 1993.

    Article  MathSciNet  Google Scholar 

  20. Y. Weiss. “Correctness of local probability propagation in graphical models with loops”. Neural Computation 12 (1–41) 2000.

    Article  MATH  Google Scholar 

  21. Y. Weiss. “Comparing the mean field method and belief propagation for approximate inference in MRFs”. To appear in Advanced Mean Field Methods. Saad and Opper (Eds). MIT Press. 2001.

    Google Scholar 

  22. J.S. Yedidia, W.T. Freeman, Y. Weiss. “Bethe free energy, Kikuchi approximations and belief propagation algorithms”. To appear in NIPS’2000. 2000.

    Google Scholar 

  23. A.L. Yuille and J.J. Kosowsky. “Statistical Physics Algorithms that Converge”. Neural Computation. 6, pp 341–356. 1994.

    Article  Google Scholar 

  24. A.L. Yuille. “A Double-Loop Algorithm to Minimize the Bethe and Kikuchi Free Energies”. Submitted to Neural Computation. 2001.

    Google Scholar 

  25. A.L. Yuille and A. Rangarajan. “The Concave-Convex Procedure (CCCP)”. Submitted to Neural Computation. 2001.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Smith Kettlewell Eye Research Institute, San Francisco, CA, 94965, USA

    Alan Yuille

Authors
  1. Alan Yuille
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Instituto de Telecomunicacoes and Instituto Superior Tecnico, 1049-001, Lisboa, Portugal

    Mário Figueiredo

  2. INRIA, 2004 Route des Lucioles, 06902 Sophia-Antipolis cedex, France

    Josiane Zerubia

  3. Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, 48824, USA

    Anil K. Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2001 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yuille, A. (2001). A Double-Loop Algorithm to Minimize the Bethe Free Energy. In: Figueiredo, M., Zerubia, J., Jain, A.K. (eds) Energy Minimization Methods in Computer Vision and Pattern Recognition. EMMCVPR 2001. Lecture Notes in Computer Science, vol 2134. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44745-8_1

Download citation

  • DOI: https://doi.org/10.1007/3-540-44745-8_1

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-42523-6

  • Online ISBN: 978-3-540-44745-0

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation