Interactive image recoloring by combining global and local optimization
- 286 Downloads
We propose a novel interactive image recoloring method by combining global and local optimization. Our approach assumes that each pixel is a linear transform of its neighbors which can be in spatial or feature space. Corresponding, a new framework for combining global and local energy optimization is designed and derived. By taking advantage of global and local color propagation, our approach requires only a few user scribbles to produce the high-quality results. We show various experimental results and comparisons on image recoloring. Compared with the state-of-the-art methods, our approach produces higher-quality results with only a small amount of user interaction than those only consider local propagation or global propagation approaches.
KeywordsImage recoloring Global optimization Local optimization Color propagation
Image recoloring is the process of modification and adjustment of color appearance in images [2, 7, 18, 24, 25]. In recent years, image recoloring is one of the most popular photo editing tasks. The techniques of image recoloring can be broadly classified as example-based and scribble-based techniques [1, 19]. Although the use of a reference image can save considerable labor, the quality of the result depends heavily on the choice of the reference image for the example-based techniques. In particular, this techniques may produce incorrect results when the illumination condition of a reference image differs from that of the target image. In comparison, the scribble-based techniques can achieve a satisfactory result with a few user scribbles to indicate the desired color. Therefore, we focus on the scribble-based techniques in this paper.
The scribble-based image recoloring techniques are performed by propagating user-provided color scribbles to similar neighborhood pixels. According to the distribution of neighbors, the scribble-based image recoloring techniques can be divided into local image recoloring methods and global image recoloring methods. Local image recoloring methods provide the user with good local control, but such approaches perform poorly when recolorizing relatively far from the provided color constraints. Therefore, these approaches need to incorporate considerable user interaction to achieve a satisfactory result. In comparison, global image recoloring methods can propagate color cues in global manner which reduces the amount of necessary user inputs, but these methods lack local or direct selection control. They will suffer from mixing the two colors of scribbles when the similar neighborhood pixels are scribbled with two different colors.
Previous local and global approaches have their respective limitations to produce correct results. Therefore, in order to reduce the amount of necessary user inputs while producing high-quality results, we exploit a novel interactive image recoloring method by combining global and local optimization. This simple combination generates surprisingly good results as demonstrated by various experimental results.
Our main contribution is to combine global and local optimization to related pixels considering the user’s interactive scribbles. Our approach assumes that each pixel is a linear transform of its neighbors which can be in spatial or feature space. The distribution of neighbors is related to the user’s interactive scribbles in our approach. To our knowledge, this technique has seldom been reported in previous researches.
The remainder of this paper is organized as follows. Section 2 gives a brief survey of related work. In Section 3, we introduce the image recoloring framework of our algorithm including the energy optimization issues and implementation details. Applications and results are discussed in Section 4. Conclusions and future work conclude the paper.
2 Related work
The image recoloring techniques are related to image editing such as colorization, matting and tonal editing, etc. According to the distribution of neighbors, existing work on scribble-based recoloring techniques can be roughly divided into local image recoloring methods and global image recoloring methods.
The classical local image recoloring methods are proposed by Levin et al. . They formulate the image recoloring method as a constraint quadratic optimization problem with the assumption that neighbors in spatial space with similar color should keep similar after image recoloring. This method can achieve a satisfactory result. However, it requires considerable user interaction. Yatziv et al.  and Criminisi et al.  both produce high-quality recoloring results by chrominance blending using geodesic distance. Fattal et al.  propose a family of second-generation wavelets constructed using a robust data-prediction lifting scheme and apply it to image recoloring. Bhat et al.  present an optimization framework for exploring gradient domain solutions for image recoloring. Farbman et al.  propose the use of diffusion distances to replace these Euclidean distances for calculating the affinity among pixels in image editing. Approaches based on spatial-intensity continuity require a large number of user inputs for disjoint regions with a similar texture. In order to propagate color in highly textured and fragmented regions, Qu et al.  and Sheng et al. [22, 23] both employ texture continuity to recolor pattern-intensive manga and natural images, respectively.
Recently, global image recoloring methods are also studied in the literature. Global image recoloring methods can propagate color cues relatively far from the provided color constraints, which is complementary to the local methods. An and Pellacini  develop a general and robust framework to image editing by efficiently approximating the all-pairs affinity matrix. Chen et al.  and Lee et al.  define a new Laplacian based on the nonlocal principle for image matting. Musialski et al.  and Chen et al.  propose a novel edit propagation algorithm for interactive image and video manipulations. These approaches use the locally linear embedding (LLE) to represent each pixel as a linear combination of its neighbors in a feature space.
Recently, Chen et al.  propose a novel alpha matting method with local and nonlocal smoothing priors. In our previous research , we propose a novel scribble-based colorization method which uses the neighbors in feature space combining color and spatial coordinate. However the distribution of neighbors restricted to the weight of image coordinate. On the one hand, usually, it is implicit that which is more important, local optimization or global optimization? Therefore it is nontrivial to set an appropriate range parameter which restricts the spatial range of searching nearest neighbors. On the other hand, the same strategy of the neighbors selection is used to the whole image region. This method cannot achieve global color propagation and provide the user with good local control at the same time.In contrast to all these techniques, we exploit a novel interactive image recoloring method by combining global and local optimization. The distribution of neighbors is related to the user’s interactive scribbles in our approach. The neighbors in spatial space yield the local optimization to color propagation, while the neighbors in nonlocal space yield the global optimization to color propagation. Because our approach considers the global and local color propagation for different image region which is related to user interaction, the accurate local control can achieve by our local linear model, while our global model can propagate color cues in global manner. With only a small amount of user interaction, our approach can produce higher-quality results than those only consider local propagation or global propagation approaches.
3.1 Local optimization
3.2 Global optimization
It is notable the (11) and (16) are similar in form. However, the main distinction between the global optimization and local optimization is the distribution of neighbors. The user can utilize brush tools to explicitly select whether to use local optimization or global optimization for different image regions. Therefore, for each region, the strategy of the neighbors selection is explicit. The local optimization solver (11) and global optimization solver (16) are used in each region separately. Both of the (11) and (16) are a large sparse system. Finally, we use the Gaussian-Seidel method iteratively to optimize these equations and get the optimal solutions in our implementation.
4 Applications and results
MSE (mean squared error) of the recolored results in Fig. 7
We have presented a novel interactive image recoloring framework only requiring a small number of user interaction. Experiments have demonstrated that our interactive image recoloring approach can produce higher-quality results than those only consider local propagation or global propagation with only using a small amount of user interaction. As our method can be performed in parallel, we plan to further accelerate the algorithm by employing the GPU implementation of KNN and PCG (Preconditioned Conjugate Gradient)  to solve the linear system, achieving the near real-time processing in the future.
This work was supported by the National Natural Science Foundation of China (Grant No. 61100146), the Zhejiang Provincial Natural Science Foundation of China (Grant Nos. LQ14F020006, LQ12F02010, LY12F02015 and LY12F02014), and the Science and Technology Plan Program of Wenzhou, China (Grant Nos. G20130017 and No. S20100053).
- 2.Beigpour S, van de Weijer J (2011) Object recoloring based on intrinsic image estimation. In: Proceedings of the 2011 International Conference on Computer Vision, ICCV ’11, pp 327–334, Washington, DC, USA. IEEE Computer SocietyGoogle Scholar
- 5.Chen X, Zou D, Zhao Q, Tan P (2012) Manifold preserving edit propagation. ACM Trans Graph 31(6):132:1–132:7Google Scholar
- 6.Chen X, Zou D, Zhou SZ, Zhao Q, Tan P (2013) Image matting with local and nonlocal smooth priors. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’13, pp 1902–1907, Washington, DC, USA. IEEE Computer SocietyGoogle Scholar
- 11.He K, Sun J, Tang X (2010) Guided image filtering. In: Proceedings of the 11th European Conference on Computer Vision: Part I, ECCV’10, pp 1–14. Springer-Verlag, Berlin, HeidelbergGoogle Scholar
- 13.Huang H, Li X, Zhao H, Nie G, Zhongyi H, Xiao L (2014) Manifold-preserving image colorization with nonlocal estimation. Multimedia Tools and Applications:1–14Google Scholar
- 15.Lee P, Wu Y (2011) Nonlocal matting. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’11, pp 2193–2200, Washington, DC, USA. IEEE Computer SocietyGoogle Scholar
- 21.Olonetsky I, Avidan S (2012) Treecann - k-d tree coherence approximate nearest neighbor algorithm. In: Proceedings of the 12th European Conference on Computer Vision - Volume Part IV, ECCV’12, pp 602–615. Springer-Verlag, Berlin, HeidelbergGoogle Scholar
- 25.Wang B, Yizhou Y, Wong T-T, Chen C, Xu Y-Q (2010) Data-driven image color theme enhancement. ACM Trans Graph 29(6):146:1–146:10Google Scholar