Abstract
In this paper we demonstrate that the framework of nonlinear spectral decompositions based on total variation (TV) regularization is very well suited for image fusion as well as more general image manipulation tasks. The well-localized and edge-preserving spectral TV decomposition allows to select frequencies of a certain image to transfer particular features, such as wrinkles in a face, from one image to another. We illustrate the effectiveness of the proposed approach in several numerical experiments, including a comparison to the competing techniques of Poisson image editing, linear osmosis, wavelet fusion and Laplacian pyramid fusion. We conclude that the proposed spectral TV image decomposition framework is a valuable tool for semi- and fully-automatic image editing and fusion.
M. Benning, M. Möller and R.Z. Nossek—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Amolins, K., Zhang, Y., Dare, P.: Wavelet based image fusion techniques an introduction, review and comparison. ISPRS J. Photogramm. Remote Sens. 62, 249–263 (2007)
Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: CVPR Proceedings, pp. 1859–1866 (2014)
Benning, M., Burger, M.: Ground states and singular vectors of convex variational regularization methods. Methods Appl. Anal. 20(4), 295–334 (2013)
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)
Burger, M., Eckardt, L., Gilboa, G., Moeller, M.: Spectral representations of one-homogeneous functionals. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 16–27. Springer, Cham (2015). doi:10.1007/978-3-319-18461-6_2
Burger, M., Frick, K., Osher, S.J., Scherzer, O.: Inverse total variation flow. Multiscale Model. Simul. 6(2), 366–395 (2007)
Burger, M., Gilboa, G., Moeller, M., Eckardt, L., Cremers, D.: Spectral decompositions using one-homogeneous functionals (2015, submitted). http://arxiv.org/pdf/1601.02912v1.pdf
Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Comm. Math. Sci. 4(1), 179–212 (2006)
Burt, P.J., Adelson, E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. (TOG) 2(4), 217–236 (1983)
Condat, L., Mosaddegh, S.: Joint demosaicking and denoising by total variation minimization. In: 19th IEEE International Conference on Image Processing (ICIP), pp. 2781–2784 (2012)
Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone, detail manipulation. In: ACM SIGGRAPH 2008 Papers, SIGGRAPH 2008, pp. 67:1–67:10. ACM, New York (2008)
Fattal, R., Agrawala, M., Rusinkiewicz, S.: Multiscale shape and detail enhancement from multi-light image collections. In: ACM SIGGRAPH 2007 Papers, SIGGRAPH 2007. ACM, New York (2007)
Gilboa, G.: A spectral approach to total variation. In: Kuijper, A., Bredies, K., Pock, T., Bischof, H. (eds.) SSVM 2013. LNCS, vol. 7893, pp. 36–47. Springer, Heidelberg (2013). doi:10.1007/978-3-642-38267-3_4
Gilboa, G.: A total variation spectral framework for scale and texture analysis. SIAM J. Imaging Sci. 7(4), 1937–1961 (2014)
Gilboa, G., Moeller, M., Burger, M.: Nonlinear spectral analysis via one-homogeneous functionals: overview and future prospects. J. Math. Imaging Vis. 56(2), 300–319 (2016)
Goldstein, T., Esser, E., Baraniuk, R.: Adaptive primal-dual hybrid gradient methods for saddle-point problems. arXiv Preprint (2013)
Hagenburg, K., Breuß, M., Weickert, J., Vogel, O.: Novel schemes for hyperbolic PDEs using osmosis filters from visual computing. In: Bruckstein, A.M., Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 532–543. Springer, Heidelberg (2012). doi:10.1007/978-3-642-24785-9_45
Weickert, J., Hagenburg, K., Breuß, M., Vogel, O.: Linear osmosis models for visual computing. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 26–39. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40395-8_3
Kass, M., Solomon, J.: Smoothed local histogram filters. In: ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010, pp. 100:1–100:10. ACM. New York (2010)
Li, S., Yang, B.: Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recogn. Lett. 29(9), 1295–1301 (2008)
Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multi-label segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1234–1247 (2013)
Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation based image restoration. SIAM J. Multiscale Model. Simul. 4, 460–489 (2005)
Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 Papers, SIGGRAPH 2003, pp. 313–318. ACM (2003)
Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: ICCV, pp. 1762–1769 (2011)
Stathaki, T.: Image Fusion: Algorithms and Applications. Academic Press, Cambridge (2008)
Su, Z., Lu, X., Artusi, A.: A novel image decomposition approach and its applications. Vis. Comput. 29, 1011–1023 (2013)
Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. In: ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia 2009, pp. 147:1–147:9. ACM, New York (2009)
Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., Theobalt, C.: Real-time expression transfer for facial reenactment. ACM Trans. Graph. (TOG), 34(6) (2015)
Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)
Zeeuw, P.M.: Wavelets and Image Fusion. CWI, Amsterdam (1998)
Acknowledgements
MBe and CBS acknowledge support from EPSRC grant EP/M00483X/1 and the Leverhulme Trust project Breaking the non-convexity barrier. MBe further acknowledges support from the Leverhulme Trust early career fellowship “Learning from mistakes: a supervised feedback-loop for imaging applications” and the Newton Trust. MM acknowledges support from the German Research Foundation (DFG) as part of the research training group GRK 1564 Imaging New Modalities. RN and GG acknowledge support by the Israel Science Foundation (grant 718/15). MBu acknowledges support by ERC via Grant EU FP 7 - ERC Consolidator Grant 615216 LifeInverse. DC acknowledges support from ERC Consolidator Grant 3D Reloaded. CBS further acknowledges support from EPSRC centre EP/N014588/1, the Cantab Capital Institute for the Mathematics of Information, and from CHiPS (Horizon 2020 RISE project grant).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Benning, M. et al. (2017). Nonlinear Spectral Image Fusion. In: Lauze, F., Dong, Y., Dahl, A. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2017. Lecture Notes in Computer Science(), vol 10302. Springer, Cham. https://doi.org/10.1007/978-3-319-58771-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-58771-4_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-58770-7
Online ISBN: 978-3-319-58771-4
eBook Packages: Computer ScienceComputer Science (R0)