Advertisement

Interactive Boundary Prediction for Object Selection

  • Hoang LeEmail author
  • Long Mai
  • Brian Price
  • Scott Cohen
  • Hailin Jin
  • Feng Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11218)

Abstract

Interactive image segmentation is critical for many image editing tasks. While recent advanced methods on interactive segmentation focus on the region-based paradigm, more traditional boundary-based methods such as Intelligent Scissor are still popular in practice as they allow users to have active control of the object boundaries. Existing methods for boundary-based segmentation solely rely on low-level image features, such as edges for boundary extraction, which limits their ability to adapt to high-level image content and user intention. In this paper, we introduce an interaction-aware method for boundary-based image segmentation. Instead of relying on pre-defined low-level image features, our method adaptively predicts object boundaries according to image content and user interactions. Therein, we develop a fully convolutional encoder-decoder network that takes both the image and user interactions (e.g. clicks on boundary points) as input and predicts semantically meaningful boundaries that match user intentions. Our method explicitly models the dependency of boundary extraction results on image content and user interactions. Experiments on two public interactive segmentation benchmarks show that our method significantly improves the boundary quality of segmentation results compared to state-of-the-art methods while requiring fewer user interactions.

Notes

Acknowledgments

This work was partially done when the first author was an intern at Adobe Research. Figure 2 uses images from Flickr user Liz West and Laura Wolf, Fig. 3 uses an image from Flickr user Mathias Appel, and Fig. 8 uses an image from Flickr user GlobalHort Image Library/ Imagetheque under a Creative Commons license.

References

  1. 1.
    Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283 (2016)Google Scholar
  2. 2.
    Adams, R., Bischof, L.: Seeded region growing. IEEE Trans. Pattern Anal. Mach. Intell. 16(6), 641–647 (1994)CrossRefGoogle Scholar
  3. 3.
    Bai, X., Sapiro, G.: A geodesic framework for fast interactive image and video segmentation and matting. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  4. 4.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  5. 5.
    Boykov, Y., Funka-Lea, G.: Graph cuts and efficient N-D image segmentation. Int. J. Comput. Vis. 70(2), 109–131 (2006)CrossRefGoogle Scholar
  6. 6.
    Brinkmann, R.: The Art and Science of Digital Compositing, 2nd edn. Morgan Kaufmann Publishers Inc., San Francisco (2008)Google Scholar
  7. 7.
    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI 8(6), 679–698 (1986)CrossRefGoogle Scholar
  8. 8.
    Castrejon, L., Kundu, K., Urtasun, R., Fidler, S.: Annotating object instances with a polygon-RNN. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4485–4493 (2017)Google Scholar
  9. 9.
    Chen, D., Mirebeau, J.M., Cohen, L.D.: A new Finsler minimal path model with curvature penalization for image segmentation and closed contour detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 355–363 (2016)Google Scholar
  10. 10.
    Chen, D., Mirebeau, J.M., Cohen, L.D.: Global minimum for a finsler elastica minimal path approach. Int. J. Comput. Vis. 122(3), 458–483 (2017)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Cohen, L.: Minimal paths and fast marching methods for image analysis. In: Paragios, N., Chen, Y., Faugeras, O. (eds.) Handbook of Mathematical Models in Computer Vision, pp. 97–111. Springer, Boston (2006).  https://doi.org/10.1007/0-387-28831-7_6CrossRefGoogle Scholar
  12. 12.
    Cohen, L.D., Kimmel, R.: Global minimum for active contour models: a minimal path approach. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 666–673 (1996)Google Scholar
  13. 13.
    Criminisi, A., Sharp, T., Blake, A.: GeoS: Geodesic Image Segmentation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 99–112. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88682-2_9CrossRefGoogle Scholar
  14. 14.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org
  15. 15.
    Gulshan, V., Rother, C., Criminisi, A., Blake, A., Zisserman, A.: Geodesic star convexity for interactive image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3129–3136 (2010)Google Scholar
  16. 16.
    He, J., Kim, C.S., Kuo, C.C.J.: Interactive image segmentation techniques. Interactive Segmentation Techniques. SpringerBriefs in Electrical and Computer Engineering, pp. 17–62. Springer, Singapore (2014).  https://doi.org/10.1007/978-981-4451-60-4_3CrossRefzbMATHGoogle Scholar
  17. 17.
    Jung, M., Peyré, G., Cohen, L.D.: Non-local active contours. In: Bruckstein, A.M., ter Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 255–266. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-24785-9_22CrossRefGoogle Scholar
  18. 18.
    Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988)CrossRefGoogle Scholar
  19. 19.
    Kim, T.H., Lee, K.M., Lee, S.U.: Generative image segmentation using random walks with restart. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 264–275. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88690-7_20CrossRefGoogle Scholar
  20. 20.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  21. 21.
    Lempitsky, V., Kohli, P., Rother, C., Sharp, T.: Image segmentation with a bounding box prior. In: IEEE International Conference on Computer Vision, pp. 277–284 (2009)Google Scholar
  22. 22.
    Li, Y., Sun, J., Tang, C.K., Shum, H.Y.: Lazy snapping. ACM Trans. Graph. 23(3), 303–308 (2004)CrossRefGoogle Scholar
  23. 23.
    Liu, Y., Cheng, M.M., Hu, X., Wang, K., Bai, X.: Richer convolutional features for edge detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5872–5881 (2017)Google Scholar
  24. 24.
    McGuinness, K., O’connor, N.E.: A comparative evaluation of interactive segmentation algorithms. Pattern Recognit. 43(2), 434–444 (2010)CrossRefGoogle Scholar
  25. 25.
    McIntyre, C.: Visual Alchemy: The Fine Art of Digital Montage. Taylor & Francis, New York (2014)CrossRefGoogle Scholar
  26. 26.
    Mille, J., Bougleux, S., Cohen, L.D.: Combination of piecewise-geodesic paths for interactive segmentation. Int. J. Comput. Vis. 112(1), 1–22 (2015)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Mirebeau, J.M.: Fast-marching methods for curvature penalized shortest paths. J. Math. Imaging Vis. 60(6), 784–815 (2017)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Mortensen, E.N., Barrett, W.A.: Intelligent scissors for image composition. In: Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1995, pp. 191–198. ACM, New York (1995)Google Scholar
  29. 29.
    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 724–732 (2016)Google Scholar
  30. 30.
    Rother, C., Kolmogorov, V., Blake, A.: GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)CrossRefGoogle Scholar
  31. 31.
    Santner, J., Pock, T., Bischof, H.: Interactive multi-label segmentation. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6492, pp. 397–410. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-19315-6_31CrossRefGoogle Scholar
  32. 32.
    Ulen, J., Strandmark, P., Kahl, F.: Shortest paths with higher-order regularization. IEEE Trans. Pattern Anal. Mach. Intell. 37(12), 2588–2600 (2015)CrossRefGoogle Scholar
  33. 33.
    Whalley, R.: Photoshop Layers: Professional Strength Image Editing: Lenscraft Photography (2015)Google Scholar
  34. 34.
    Wu, J., Zhao, Y., Zhu, J., Luo, S., Tu, Z.: MILCut: a sweeping line multiple instance learning paradigm for interactive image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 256–263 (2014)Google Scholar
  35. 35.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)Google Scholar
  36. 36.
    Xu, N., Price, B., Cohen, S., Yang, J., Huang, T.S.: Deep interactive object selection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 373–381 (2016)Google Scholar
  37. 37.
    Xu, N., Price, B.L., Cohen, S., Yang, J., Huang, T.S.: Deep grabcut for object selection. In: British Machine Vision Conference (2017)Google Scholar
  38. 38.
    Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.H.: Object contour detection with a fully convolutional encoder-decoder network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 193–202 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Hoang Le
    • 1
    Email author
  • Long Mai
    • 2
  • Brian Price
    • 2
  • Scott Cohen
    • 2
  • Hailin Jin
    • 2
  • Feng Liu
    • 1
  1. 1.Portland State UniversityPortlandUSA
  2. 2.Adobe ResearchSan JoseUSA

Personalised recommendations