Nonlinear Spectral Image Fusion

  • Martin BenningEmail author
  • Michael Möller
  • Raz Z. Nossek
  • Martin Burger
  • Daniel Cremers
  • Guy Gilboa
  • Carola-Bibiane Schönlieb
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10302)


In this paper we demonstrate that the framework of nonlinear spectral decompositions based on total variation (TV) regularization is very well suited for image fusion as well as more general image manipulation tasks. The well-localized and edge-preserving spectral TV decomposition allows to select frequencies of a certain image to transfer particular features, such as wrinkles in a face, from one image to another. We illustrate the effectiveness of the proposed approach in several numerical experiments, including a comparison to the competing techniques of Poisson image editing, linear osmosis, wavelet fusion and Laplacian pyramid fusion. We conclude that the proposed spectral TV image decomposition framework is a valuable tool for semi- and fully-automatic image editing and fusion.


Nonlinear spectral decomposition Total variation regularization Image fusion Image composition Multiscale methods 



MBe and CBS acknowledge support from EPSRC grant EP/M00483X/1 and the Leverhulme Trust project Breaking the non-convexity barrier. MBe further acknowledges support from the Leverhulme Trust early career fellowship “Learning from mistakes: a supervised feedback-loop for imaging applications” and the Newton Trust. MM acknowledges support from the German Research Foundation (DFG) as part of the research training group GRK 1564 Imaging New Modalities. RN and GG acknowledge support by the Israel Science Foundation (grant 718/15). MBu acknowledges support by ERC via Grant EU FP 7 - ERC Consolidator Grant 615216 LifeInverse. DC acknowledges support from ERC Consolidator Grant 3D Reloaded. CBS further acknowledges support from EPSRC centre EP/N014588/1, the Cantab Capital Institute for the Mathematics of Information, and from CHiPS (Horizon 2020 RISE project grant).

Supplementary material (16.6 mb)
Supplementary material 1 (zip 16947 KB)


  1. 1.
    Amolins, K., Zhang, Y., Dare, P.: Wavelet based image fusion techniques an introduction, review and comparison. ISPRS J. Photogramm. Remote Sens. 62, 249–263 (2007)CrossRefGoogle Scholar
  2. 2.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: CVPR Proceedings, pp. 1859–1866 (2014)Google Scholar
  3. 3.
    Benning, M., Burger, M.: Ground states and singular vectors of convex variational regularization methods. Methods Appl. Anal. 20(4), 295–334 (2013)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  5. 5.
    Burger, M., Eckardt, L., Gilboa, G., Moeller, M.: Spectral representations of one-homogeneous functionals. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 16–27. Springer, Cham (2015). doi: 10.1007/978-3-319-18461-6_2 Google Scholar
  6. 6.
    Burger, M., Frick, K., Osher, S.J., Scherzer, O.: Inverse total variation flow. Multiscale Model. Simul. 6(2), 366–395 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Burger, M., Gilboa, G., Moeller, M., Eckardt, L., Cremers, D.: Spectral decompositions using one-homogeneous functionals (2015, submitted).
  8. 8.
    Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Comm. Math. Sci. 4(1), 179–212 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Burt, P.J., Adelson, E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. (TOG) 2(4), 217–236 (1983)CrossRefGoogle Scholar
  10. 10.
    Condat, L., Mosaddegh, S.: Joint demosaicking and denoising by total variation minimization. In: 19th IEEE International Conference on Image Processing (ICIP), pp. 2781–2784 (2012)Google Scholar
  11. 11.
    Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone, detail manipulation. In: ACM SIGGRAPH 2008 Papers, SIGGRAPH 2008, pp. 67:1–67:10. ACM, New York (2008)Google Scholar
  12. 12.
    Fattal, R., Agrawala, M., Rusinkiewicz, S.: Multiscale shape and detail enhancement from multi-light image collections. In: ACM SIGGRAPH 2007 Papers, SIGGRAPH 2007. ACM, New York (2007)Google Scholar
  13. 13.
    Gilboa, G.: A spectral approach to total variation. In: Kuijper, A., Bredies, K., Pock, T., Bischof, H. (eds.) SSVM 2013. LNCS, vol. 7893, pp. 36–47. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38267-3_4 CrossRefGoogle Scholar
  14. 14.
    Gilboa, G.: A total variation spectral framework for scale and texture analysis. SIAM J. Imaging Sci. 7(4), 1937–1961 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Gilboa, G., Moeller, M., Burger, M.: Nonlinear spectral analysis via one-homogeneous functionals: overview and future prospects. J. Math. Imaging Vis. 56(2), 300–319 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Goldstein, T., Esser, E., Baraniuk, R.: Adaptive primal-dual hybrid gradient methods for saddle-point problems. arXiv Preprint (2013)Google Scholar
  17. 17.
    Hagenburg, K., Breuß, M., Weickert, J., Vogel, O.: Novel schemes for hyperbolic PDEs using osmosis filters from visual computing. In: Bruckstein, A.M., Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 532–543. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-24785-9_45 CrossRefGoogle Scholar
  18. 18.
    Weickert, J., Hagenburg, K., Breuß, M., Vogel, O.: Linear osmosis models for visual computing. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 26–39. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40395-8_3 CrossRefGoogle Scholar
  19. 19.
    Kass, M., Solomon, J.: Smoothed local histogram filters. In: ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010, pp. 100:1–100:10. ACM. New York (2010)Google Scholar
  20. 20.
    Li, S., Yang, B.: Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recogn. Lett. 29(9), 1295–1301 (2008)CrossRefGoogle Scholar
  21. 21.
    Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multi-label segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1234–1247 (2013)CrossRefGoogle Scholar
  22. 22.
    Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation based image restoration. SIAM J. Multiscale Model. Simul. 4, 460–489 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 Papers, SIGGRAPH 2003, pp. 313–318. ACM (2003)Google Scholar
  24. 24.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: ICCV, pp. 1762–1769 (2011)Google Scholar
  25. 25.
    Stathaki, T.: Image Fusion: Algorithms and Applications. Academic Press, Cambridge (2008)Google Scholar
  26. 26.
    Su, Z., Lu, X., Artusi, A.: A novel image decomposition approach and its applications. Vis. Comput. 29, 1011–1023 (2013)CrossRefGoogle Scholar
  27. 27.
    Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. In: ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia 2009, pp. 147:1–147:9. ACM, New York (2009)Google Scholar
  28. 28.
    Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., Theobalt, C.: Real-time expression transfer for facial reenactment. ACM Trans. Graph. (TOG), 34(6) (2015)Google Scholar
  29. 29.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  30. 30.
    Zeeuw, P.M.: Wavelets and Image Fusion. CWI, Amsterdam (1998)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Martin Benning
    • 1
    Email author
  • Michael Möller
    • 2
  • Raz Z. Nossek
    • 3
  • Martin Burger
    • 4
  • Daniel Cremers
    • 5
  • Guy Gilboa
    • 3
  • Carola-Bibiane Schönlieb
    • 1
  1. 1.University of CambridgeCambridgeUK
  2. 2.Universität SiegenSiegenGermany
  3. 3.Technion IITHaifaIsrael
  4. 4.Westfälische Wilhelms-UniversitätMünsterGermany
  5. 5.Technische Universität MünchenGarchingGermany

Personalised recommendations